Target behavior identification method and device, storage medium, and terminal

Abstract

A method and device for target behavior recognition, a storage medium, and a terminal, wherein the target behavior recognition method comprises: obtaining a video to be recognized; performing target detection on each frame of the image to be recognized to obtain a moving target and a lane line in each frame of the image to be recognized; mapping the moving target and the lane line recognized in each frame of the image to be recognized in the same mapping image according to pixel coordinates; determining the contact line between the moving target and the road surface as a horizontal reference line in each mapping image, and determining the first pixel coordinate of the moving target on the horizontal reference line, as well as the second pixel coordinate and the third pixel coordinate of the two lane lines on the horizontal reference line; calculating the relative position relationship between the first pixel coordinate and the second pixel coordinate and the third pixel coordinate in each mapping image to obtain the behavior time series data of the moving target in the video to be recognized. The technical solution of the present invention can realize the action recognition of moving targets in high-speed and long-distance scenes.

Description

Technical Field

[0001] The present invention relates to the field of image processing technology, and in particular to a target behavior recognition method and device, a storage medium, and a terminal. Background Art

[0002] The main technology for lane intrusion prediction is video action recognition, or more precisely, video action recognition in high-speed and long-distance scenarios. Video action recognition refers to identifying the actions of a target (person) in a video from a continuous video (i.e., a sequence of image frames). Videos contain two types of features: temporal features and spatial features. Temporal features are features of pixel changes corresponding to images in different frames (at different times). Spatial features are pixel features in a frame of an image. Extracting temporal and spatial features requires calculating the relationship between pixels within a frame and between different frames.

[0003] For video action recognition, there are currently two mainstream methods: 3D convolutional neural network and two-stream method. The 3D spatiotemporal convolution method uses 3D convolution kernels to capture temporal and spatial features simultaneously. Compared with 2D convolution, 3D convolution kernels add one dimension. 3D convolution kernels can calculate the pixel relationship of the same area in several consecutive frames, so they can extract temporal and spatial features at the same time, thereby realizing video action recognition. The two-stream method includes an optical flow network branch and an image network branch, and uses optical flow images to represent temporal features. Optical flow is the instantaneous speed of pixel movement of a spatial moving object on the observation imaging plane. The optical flow method is a method that uses the change of pixels in the time domain in an image sequence and the correlation between adjacent frames to find the corresponding relationship between the previous frame and the current frame, thereby calculating the motion information of the object between adjacent frames. The two-stream method uses 2D convolution kernels to extract temporal features in the optical flow network branch and spatial features in the image network branch, and then fuses the two to extract spatiotemporal features, thereby classifying video actions.

[0004] However, 3D convolution and dual-stream methods are not suitable for high-speed and long-distance scenarios. High-speed and long-distance scenarios are when the camera moves at high speed and the distance between the action target and the camera is greater than a preset distance, such as 100 meters. In this case, the irregular visual changes of the moving target in the image frame and the violent shaking of the image may cause the changes in the image frame to fail to truly reflect the actual movement of the target. Whether it is the 3D convolution method or the optical flow method, considering that they extract the change features of the corresponding pixels in the image frame, it is obviously impossible to identify the entire behavior based only on the absolute change of the pixels in the target area in the image. In addition, due to the long distance, the action target only occupies a small part of the pixel space in the video image frame plane. Both the 3D convolution and dual-stream methods will extract the pixel features of the entire image, including the background, and the moving target features account for a very small proportion. Therefore, these methods are difficult to capture the action features of such a small target, and it is even more difficult to accurately predict its behavioral intentions. Summary of the invention

[0005] The technical problem solved by the present invention is how to realize the motion recognition of moving targets in high-speed and long-distance scenes.

[0006] To solve the above technical problems, an embodiment of the present invention provides a target behavior recognition method, which includes: obtaining a video to be recognized, wherein the video to be recognized includes multiple frames of images to be recognized; performing target detection on each frame of the image to be recognized to respectively obtain a moving target and a lane line in each frame of the image to be recognized; mapping the moving target and the lane line recognized in each frame of the image to be recognized in the same mapping image according to pixel coordinates, and each frame of the image to be recognized has a corresponding mapping image; determining in each mapping image the contact line between the moving target and the road surface as a horizontal reference line, and determining a first pixel coordinate of the moving target on the horizontal reference line, as well as a second pixel coordinate and a third pixel coordinate of two lines of the lane line on the horizontal reference line; calculating the relative position relationship between the first pixel coordinate and the second pixel coordinate and the third pixel coordinate in each mapping image to obtain behavior time series data of the moving target in the video to be recognized, wherein the behavior time series data includes multiple relative position relationships arranged in order of time corresponding to multiple frames of the image to be recognized.

[0007] Optionally, the calculation of the relative position relationship between the first pixel coordinate and the second pixel coordinate and the third pixel coordinate in each mapping image includes: determining the fourth pixel coordinate of the center point of the lane line on the horizontal baseline based on the second pixel coordinate and the third pixel coordinate in each mapping image; and calculating the difference between the first pixel coordinate and the fourth pixel coordinate in each mapping image as the relative position relationship.

[0008] Optionally, the calculation of the relative position relationship between the first pixel coordinate and the second pixel coordinate and the third pixel coordinate in each mapping image includes: determining the width of the lane line and the fourth pixel coordinate of the center point of the lane line on the horizontal baseline based on the second pixel coordinate and the third pixel coordinate in each mapping image; calculating the difference between the first pixel coordinate and the fourth pixel coordinate in each mapping image, and normalizing it using the width of the lane line as the relative position relationship.

[0009] Optionally, performing target detection on each frame of the image to be identified includes: inputting each frame of the image to be identified into a trained motion target detection model and a lane line detection model respectively, the motion target detection model outputting the pixel coordinates of the identified motion target, and the lane line detection model outputting the pixel coordinates of the lane line identified.

[0010] Optionally, the moving target is represented by pixel coordinates of an envelope box, and the lane line is represented by pixel coordinates in the image to be identified.

[0011] Optionally, after determining the first pixel coordinates of the moving target on the horizontal baseline, and the second pixel coordinates and third pixel coordinates of the two lines of the lane line on the horizontal baseline, the method also includes: performing Kalman filtering on the first pixel coordinates, the second pixel coordinates and the third pixel coordinates corresponding to each frame of the image to be identified.

[0012] Optionally, the target behavior recognition method further includes: classifying the intrusion behavior of the moving target into the lane line according to the relative position relationship in the behavior time series data.

[0013] To solve the above technical problems, an embodiment of the present invention further discloses a target behavior recognition device, which includes: an acquisition module, used to acquire a video to be recognized, wherein the video to be recognized includes multiple frames of images to be recognized; a target detection module, used to perform target detection on each frame of the image to be recognized, so as to obtain the moving target and lane line in each frame of the image to be recognized respectively; a mapping module, used to map the moving target and lane line recognized in each frame of the image to be recognized into the same mapping image according to pixel coordinates, and each frame of the image to be recognized has a corresponding mapping image; a pixel coordinate calculation module, used to determine in each mapping image that the contact line between the moving target and the road surface is a horizontal reference line, and determine the first pixel coordinate of the moving target on the horizontal reference line, and the second pixel coordinate and the third pixel coordinate of the two lines of the lane line on the horizontal reference line; a relative position relationship calculation module, used to calculate the relative position relationship between the first pixel coordinate and the second pixel coordinate and the third pixel coordinate in each mapping image, so as to obtain the behavior time series data of the moving target in the video to be recognized, wherein the behavior time series data includes a plurality of relative position relationships arranged in the order of moments corresponding to the multiple frames of the image to be recognized.

[0014] The embodiment of the present invention further discloses a storage medium on which a computer program is stored. When the computer program is executed by a processor, the steps of the target behavior recognition method are executed.

[0015] An embodiment of the present invention further discloses a terminal, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and the processor executes the steps of the target behavior recognition method when running the computer program.

[0016] Compared with the prior art, the technical solution of the embodiment of the present invention has the following beneficial effects:

[0017] The technical solution of the present invention can capture the position characteristics of small targets by mapping the moving targets and lane lines in each frame of the image to be identified into the same mapping image, and calculating the relative position relationship between the moving targets and lane lines on the horizontal reference line in the mapping image, thereby solving the problem of difficulty in capturing the movements of small targets in long-distance scenes, and laying the foundation for accurate prediction of target behavior. In addition, the technical solution of the present invention can reduce the difficulty of identification, improve the identification accuracy and system robustness by converting the target behavior identification into behavior time series data for identification.

[0018] Furthermore, the width of the lane line and the fourth pixel coordinate of the center point of the lane line on the horizontal reference line are determined according to the second pixel coordinate and the third pixel coordinate in each mapped image; the difference between the first pixel coordinate and the fourth pixel coordinate in each mapped image is calculated, and the lane line width is used for normalization as the relative position relationship. Since the video shot in high-speed long-distance scenes has problems such as scale change and image jitter, the position relationship between the moving target and the lane line in each frame of the image can only represent the position relationship in the current frame, and the behavior time series data based on this cannot accurately reflect the movement behavior of the target. The technical solution of the present invention uses the width of the lane line to normalize the relative position relationship, which can accurately describe the trajectory of the moving target and solve the problems caused by image jitter and scale change in high-speed scenes. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] Figure 1 is a flow chart of a target behavior recognition method according to an embodiment of the present invention;

[0020] Figure 2 is a schematic diagram of a specific application scenario of an embodiment of the present invention;

[0021] Figure 3 is a schematic diagram of another specific application scenario of an embodiment of the present invention;

[0022] Figure 4 yes Figure 1 A flowchart of a specific implementation of step S105 is shown;

[0023] Figure 5 It is a specific schematic diagram of a relative position relationship of an embodiment of the present invention;

[0024] Figure 6 is a specific schematic diagram of another relative position relationship of an embodiment of the present invention;

[0025] Figure 7 It is a structural schematic diagram of a target behavior recognition device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0026] As described in the background technology, 3D convolution and dual-stream methods are not suitable for high-speed and long-distance scenes. High-speed and long-distance scenes, that is, the camera moves at high speed, and the distance between the action target and the camera is greater than a preset distance, such as 100 meters. In this case, the irregular visual changes of the moving target in the image frame and the violent shaking of the image may cause the changes in the image frame to fail to truly reflect the actual movement of the target. Whether it is the 3D convolution method or the optical flow method, considering that they extract the change features of the corresponding pixels in the image frame, it is obviously impossible to identify the entire behavior based only on the absolute change of the pixels in the target area in the image. In addition, due to the long distance, the action target only occupies a small part of the pixel space in the plane of the video image frame. Both 3D convolution and dual-stream methods extract pixel features of the entire image, including the background, and the moving target features account for a very small proportion. Therefore, these methods are difficult to capture the action features of such a small target, and it is even more difficult to accurately predict its behavioral intentions.

[0027] The technical solution of the present invention can capture the position characteristics of small targets by mapping the moving targets and lane lines in each frame of the image to be identified into the same mapping image, and calculating the relative position relationship between the moving targets and lane lines on the horizontal reference line in the mapping image, thereby solving the problem of difficulty in capturing the movements of small targets in long-distance scenes, and laying the foundation for accurate prediction of target behavior. In addition, the technical solution of the present invention can reduce the difficulty of identification, improve the identification accuracy and system robustness by converting the target behavior identification into behavior time series data for identification.

[0028] Furthermore, due to the problems of scale change and image jitter in videos shot in high-speed and long-distance scenes, the positional relationship between the moving target and the lane line in each frame can only represent the positional relationship in the current frame. The behavior time series data based on this cannot accurately reflect the movement behavior of the target. The technical solution of the present invention uses the width of the lane line to normalize the relative position relationship, which can accurately describe the trajectory of the moving target and solve the problems caused by image jitter and scale change in high-speed scenes.

[0029] The "moving target" in the embodiment of the present invention may refer to an object that is smaller than a vehicle, such as a pedestrian, a person riding a non-motor vehicle, or other animals. Further, the moving target in the embodiment of the present invention refers to a pedestrian or a person riding a non-motor vehicle who intrudes into the lane of a motor vehicle.

[0030] The "lane line" referred to in the embodiment of the present invention may refer to a motor vehicle lane line, which usually appears in pairs. The two lines of the lane line can limit the range of motor vehicle driving, that is, the prohibited area for non-motor vehicles.

[0031] The "high-speed long-distance scene" referred to in the embodiments of the present invention may refer to a scene in which the speed of the camera collecting the video is greater than a preset speed, such as 60 km / h; the distance between the camera and the moving target being photographed is greater than a preset distance, such as 100 meters. In a non-limiting example, the camera may be set in a vehicle, the vehicle speed is greater than 60 km / h, and the distance between the vehicle and the moving target is greater than 100 meters.

[0032] The “target behavior” referred to in the embodiment of the present invention may be a lane intrusion behavior, that is, a behavior in which a moving target enters the range defined by two lane lines.

[0033] In order to make the above-mentioned objects, features and advantages of the present invention more obvious and easy to understand, specific embodiments of the present invention are described in detail below with reference to the accompanying drawings.

[0034] Figure 1 It is a flow chart of a target behavior recognition method according to an embodiment of the present invention.

[0035] The target behavior recognition method of the embodiment of the present invention can be used on the terminal device side, for example, it can be a vehicle-mounted device, that is, it can be executed by the vehicle-mounted device. Figure 1 The individual steps of the method are shown.

[0036] Specifically, Figure 1 The target behavior recognition method may include the following steps:

[0037] Step S101: obtaining a video to be identified, wherein the video to be identified includes multiple frames of images to be identified;

[0038] Step S102: performing target detection on each frame of the image to be identified, so as to obtain the moving target and lane line in each frame of the image to be identified;

[0039] Step S103: mapping the moving objects and lane lines identified in each frame of the image to be identified into the same mapping image according to pixel coordinates, so that each frame of the image to be identified has a corresponding mapping image;

[0040] Step S104: determining in each mapping image the contact line between the moving object and the road surface as a horizontal reference line, and determining a first pixel coordinate of the moving object on the horizontal reference line, and a second pixel coordinate and a third pixel coordinate of two lines of the lane line on the horizontal reference line;

[0041] Step S105: Calculate the relative position relationship between the first pixel coordinates and the second pixel coordinates and the third pixel coordinates in each mapped image to obtain the behavior time series data of the moving target in the video to be identified, wherein the behavior time series data includes multiple relative position relationships arranged in the order of moments corresponding to multiple frames of images to be identified.

[0042] It should be pointed out that the serial numbers of the steps in this embodiment do not limit the execution order of the steps.

[0043] In the specific implementation of step S101, the vehicle-mounted device can obtain the video to be identified from the vehicle-mounted camera, which can be set inside the vehicle-mounted device or can be externally coupled to the vehicle-mounted device. Specifically, the vehicle-mounted camera can be a monocular wide-angle front-view camera that can collect video in real time and transmit it to the vehicle-mounted device for target behavior recognition.

[0044] In the specific implementation of step S102, the vehicle-mounted device performs target detection on each frame of the image to be identified in the video to be identified, that is, the identification of the moving target and the lane line is performed independently. The moving target and the lane line in each frame of the image to be identified are represented by their pixel coordinates in the image to be identified. More specifically, the moving target is represented by the pixel coordinates of the envelope box, and the lane line is represented by the coordinates of its pixels in the image to be identified.

[0045] In a non-limiting example, the envelope of a moving object is defined as the coordinates in the image pixel UV coordinate system, including its upper left coordinate (u l,t, v l,t ) and the lower right coordinate (u r,b ,v r,b ).

[0046] In a non-limiting example, when identifying lane lines, it can be determined by a grayscale image. The smaller the grayscale value of a pixel, the greater the probability that the pixel is a lane line, so the coordinates of the left and right lane lines of the lane line in the image coordinate system can be obtained in the grayscale image.

[0047] In a non-limiting embodiment, Figure 1 The shown step S102 may include the following steps: inputting each frame of the image to be identified into a trained motion target detection model and a lane line detection model respectively, the motion target detection model outputting the pixel coordinates of the identified motion target, and the lane line detection model outputting the pixel coordinates of the lane line identified.

[0048] In this embodiment, the moving target detection model and the lane line detection model can be constructed and pre-trained using a deep learning algorithm, and can be used for moving target detection and lane line detection, respectively. Specifically, the moving target detection model can be trained using annotated video data, where the video data includes annotated moving targets; the lane line detection model can be trained using annotated video data, where the video data includes annotated lane lines.

[0049] It should be noted that the specific method of using deep learning to build and train a motion target detection model and a lane line detection model can be referred to the prior art, and the embodiments of the present invention will not be repeated here.

[0050] In the specific implementation of step S103, since the moving target and the lane line are detected separately, they need to be mapped in the same image. Compared with the original image to be identified, the proportion of the moving target and the lane line in the mapped image becomes larger, which lays a foundation for the accuracy of the behavior prediction of the moving target.

[0051] In the specific implementation of step S104, it can be reasonably assumed that the lower edge line of the moving target envelope box is the contact line between the moving target and the road surface, that is, the horizontal reference line of the current frame image.

[0052] Please refer to Figure 2 , and extend it in a straight line along the contact line. This line is the horizontal reference line L that passes through the moving target T and is perpendicular to the current lane lines V1 and V2. By constructing a one-dimensional coordinate system on the horizontal reference line L, the position of the moving target T and the lane lines V1 and V2 can be clearly represented. The one-dimensional coordinate system can be the u coordinate system in the image coordinate system. The interval [u l,t ,u r,b ] represents the area occupied by the moving target, point O represents the center of the moving target, and its coordinate value is (u l,t ,u r,b ) / 2, where u l,t and u r,b The upper left and lower right coordinates of the envelope box of the moving target T in the u coordinate system. So far, the pixel positions of the moving target and lane line in the unified one-dimensional coordinate system can be determined.

[0053] Furthermore, please refer to Figure 3 , Figure 3 The horizontal axis represents the pixel coordinates on the horizontal reference line, and the vertical axis represents the probability p that the pixel is a lane line. When determining the second pixel coordinates and the third pixel coordinates of the two lane lines on the horizontal reference line, each peak represents a lane line. Figure 3 The peaks L1 and L2 are the pixel coordinates of the left and right lane lines, respectively. Generally speaking, the distance between the two lane lines, that is, the distance between point L1 and point L2, should be 2-4 times the area of ​​the moving target O (that is, the width of the envelope box on the horizontal reference line).

[0054] Specifically, some pixel points of the lane line may not be detected due to occlusion, etc. In this case, a quadratic polynomial can be used to fit the pixel coordinates L1 and L2 of the lane line to ensure that the lane line is a continuous and complete curve.

[0055] In the specific implementation of step S105, the relative position relationship between the first pixel coordinate and the second pixel coordinate and the third pixel coordinate may be calculated to characterize the relative position relationship between the moving object and the lane line.

[0056] Specifically, the relative position relationship between the first pixel coordinate and the second pixel coordinate and the third pixel coordinate can be expressed by the difference between the first pixel coordinate and the pixel coordinate of the midpoint of the lane line on the horizontal reference line, and the midpoint position of the lane line on the horizontal reference line is the sum of the second pixel coordinate and the third pixel coordinate divided by 2. In other words, the relative position relationship can be directly expressed by the second pixel coordinate, the third pixel coordinate and the midpoint position of the lane line on the horizontal reference line.

[0057] Furthermore, since the video to be identified includes multiple frames of images to be identified, a relative position relationship can be determined according to each frame of the images to be identified, and thus a behavior time series data is finally obtained, and the behavior time series data includes multiple relative position relationships arranged in the time sequence of the multiple frames of images to be identified. Specifically, the behavior time series data may include the difference between the first pixel coordinates at multiple moments and the pixel coordinates of the midpoint of the lane line on the horizontal reference line, or the second pixel coordinates, the third pixel coordinates and the midpoint position of the lane line on the horizontal reference line at multiple moments.

[0058] In a non-limiting embodiment, Figure 1 The shown step S105 may include the following steps: determining the fourth pixel coordinates of the center point of the lane line on the horizontal baseline based on the second pixel coordinates and the third pixel coordinates in each mapping image; calculating the difference between the first pixel coordinates and the fourth pixel coordinates in each mapping image as the relative position relationship.

[0059] In this embodiment, the fourth pixel coordinate L of the center point of the lane line on the horizontal reference line is first calculated. c =(L1+L2) / 2. Then calculate the difference P between the first pixel coordinate and the fourth pixel coordinate r =(OL c ), where L c represents the fourth pixel coordinate of the center point of the current lane, L1 and L2 represent the pixel coordinates of the left and right lane lines of the current lane respectively, O represents the pixel coordinate of a moving target, such as a pedestrian or a cyclist, P r Indicates the position relationship of the moving target relative to the center point.

[0060] In another non-limiting embodiment, see Figure 4 , Figure 1 The step S105 shown may include the following steps:

[0061] Step S401: determining the width of the lane line and the fourth pixel coordinate of the center point of the lane line on the horizontal reference line according to the second pixel coordinate and the third pixel coordinate in each mapping image;

[0062] Step S402: Calculate the difference between the first pixel coordinate and the fourth pixel coordinate in each mapping image, and normalize it using the width of the lane line to serve as the relative position relationship.

[0063] Different from the above-mentioned embodiment, the embodiment of the present invention performs normalization processing on the difference between the first pixel coordinate and the fourth pixel coordinate.

[0064] Specifically, in high-speed long-distance scenes, the camera will have obvious problems such as scale change and image jitter as the vehicle moves at high speed. At this time, the position of the moving target and the lane line can only represent the position relationship in the current frame, and the position time series of the moving target based on this cannot accurately reflect the movement behavior of the target. In order to accurately describe the movement of the target, this embodiment uses a normalized relative positioning method to solve the scale change and image jitter problems.

[0065] Where w = |L1-L2|, L c =(L1+L2) / 2, P r =(OL c ) / w, w represents the pixel width of the current lane, L c represents the fourth pixel coordinate of the center point of the current lane, L1 and L2 represent the pixel coordinates of the left and right lane lines of the current lane respectively, O represents the pixel coordinate of a moving target, such as a pedestrian or a cyclist, and P r Indicates the position relationship of the moving target relative to the center point.

[0066] Please refer to Figure 5 and Figure 6 , Figure 5 The middle lines l1 and l2 are obtained by fitting the second pixel coordinates and the third pixel coordinates in the behavior time series data, respectively representing the two lane lines, and the line o is obtained by fitting the first pixel coordinates in the behavior time series data, representing the moving target. Figure 5 It can be seen that due to the camera jitter and scale changes in high-speed scenes, the width of the lane line is also changing, which will lead to inaccurate judgment of the intrusion behavior of moving targets.

[0067] Figure 6It shows the relative relationship between line o, line l1, and line l2 after normalization processing. Among them, the abscissa represents time, and the ordinate represents the normalized value of the difference between the first pixel coordinate and the fourth pixel coordinate. A negative value indicates that the moving target is on the left side of the center point of the lane line, and a positive value indicates that the moving target is on the right side of the center point of the lane line. Combining Figure 5 and Figure 6 it can be seen that after normalization processing, the width of the lane is stable, solving the problems of image scale change and image jitter in the images captured by the camera in the high-speed scenario.

[0068] In a non-limiting embodiment, Figure 1 the shown step S104 may include: performing Kalman filtering on the first pixel coordinate, the second pixel coordinate, and the third pixel coordinate corresponding to each frame of the image to be recognized.

[0069] Specifically, when using traditional or deep learning-based visual target detection methods to perform real-time tracking and detection of moving targets and lane lines, false detections and missed detections will inevitably occur. To reduce the adverse effects brought by false detections and missed detections, this embodiment performs filtering processing on the calculated moving targets and lane lines. For high-speed application scenarios, the number of moving targets is small, for example, within 5. The multi-object tracking algorithm Sort with very good real-time performance can be applied to track the moving targets, so as to achieve the purpose of target alignment. After alignment, the ratio r = n / N of the number of frames n in which each moving target appears to the total number of frames N of the image is statistically calculated. If r < T (T is a preset threshold), it is considered that the target is a false detection and the target is discarded. The size of the preset threshold T can be determined with reference to the precision and recall rate of the target detection model. After obtaining the aligned sequence of moving targets, the false detections and missed detections generated by the target detection model are regarded as outliers and random noise, and the Kalman filter is used to eliminate the outliers and smooth the noise. Then the accurate continuous time series data of the moving targets and lane lines in the one-dimensional coordinate system can be obtained.

[0070] In a non-limiting embodiment, Figure 1 after the shown step S105, the following steps may further be included: classifying the intrusion behavior of the moving target into the lane line according to the relative position relationship in the behavior time series data.

[0071] In specific implementation, for the lane intrusion behavior, its categories can be intrusion and non-intrusion. The category of intrusion can be further subdivided into left intrusion and right intrusion.

[0072] In a specific application scenario, combining Figure 6 , target o enters the range defined by lane lines l1 and l2, and the category of the intrusion behavior is intrusion. Further, target o intrudes from lane line l1, and lane line l1 is on the right side of the lane, so the category of the intrusion behavior is right intrusion.

[0073] It is understandable that the specific algorithm for classifying intrusion behaviors may adopt any feasible algorithm in the prior art, and the category of intrusion behaviors may also be adaptively set according to actual application scenarios, which is not limited in the embodiment of the present invention.

[0074] Please refer to Figure 7 The embodiment of the present invention also discloses a target behavior recognition device. The target behavior recognition device 70 may include:

[0075] An acquisition module 701 is used to acquire a video to be identified, where the video to be identified includes multiple frames of images to be identified;

[0076] The target detection module 702 is used to perform target detection on each frame of the image to be identified, so as to obtain the moving target and lane line in each frame of the image to be identified;

[0077] A mapping module 703 is used to map the moving objects and lane lines identified in each frame of the image to be identified into the same mapping image according to pixel coordinates, and each frame of the image to be identified has a corresponding mapping image;

[0078] A pixel coordinate calculation module 704 is used to determine, in each mapped image, a contact line between the moving object and the road surface as a horizontal reference line, and to determine a first pixel coordinate of the moving object on the horizontal reference line, and a second pixel coordinate and a third pixel coordinate of two lines of the lane line on the horizontal reference line;

[0079] The relative position relationship calculation module 705 is used to calculate the relative position relationship between the first pixel coordinate and the second pixel coordinate and the third pixel coordinate in each mapped image to obtain the behavior time series data of the moving target in the video to be identified, and the behavior time series data includes multiple relative position relationships arranged in the order of moments corresponding to multiple frames of images to be identified.

[0080] The embodiment of the present invention can capture the positional features of small targets by mapping the moving targets and lane lines in each frame of the image to be identified into the same mapping image, and calculate the relative position relationship between the moving targets and lane lines on the horizontal reference line in the mapping image, thereby solving the problem of difficulty in capturing the movements of small targets in long-distance scenes, and laying the foundation for accurate prediction of target behavior. In addition, the embodiment of the present invention can reduce the difficulty of identification, improve identification accuracy and system robustness by converting target behavior identification into behavior time series data for identification.

[0081] For more information about the working principle and working method of the target behavior recognition device 70, please refer to Figures 1 to 6 The relevant description in will not be repeated here.

[0082] The embodiment of the present invention further discloses a storage medium, which is a computer-readable storage medium and stores a computer program. When the computer program is run, it can execute Figure 1 The steps of the target behavior recognition method shown in . The storage medium may include ROM, RAM, magnetic disk or optical disk, etc. The storage medium may also include non-volatile memory (non-volatile) or non-transitory memory, etc.

[0083] The embodiment of the present invention further discloses a terminal, which may include a memory and a processor, wherein the memory stores a computer program that can be run on the processor. When the processor runs the computer program, it can execute Figure 1 The steps of the target behavior identification method shown in . The user equipment includes but is not limited to terminal equipment such as mobile phones, computers, and tablet computers.

[0084] It should be understood that the above-mentioned processor can be a general-purpose processor, a digital signal processor (digital signal processor, DSP), an application specific integrated circuit (application specific integrated circuit, ASIC), a field programmable gate array (field programmable gate array, FPGA) or other programmable logic devices, discrete gates or transistor logic devices, discrete hardware components, and can also be a system on chip (system on chip, SoC), a central processor unit (central processor unit, CPU), a network processor (network processor, NP), a digital signal processing circuit (digital signal processor, DSP), a microcontroller unit (micro controller unit, MCU), a programmable logic device (programmable logic device, PLD) or other integrated chips. The methods, steps and logic block diagrams disclosed in the embodiments of the present application can be implemented or executed. The general-purpose processor can be a microprocessor or the processor can also be any conventional processor, etc. The steps of the method disclosed in the embodiments of the present application can be directly embodied as a hardware decoding processor for execution, or a combination of hardware and software modules in the decoding processor for execution. The software module can be located in a storage medium mature in the art such as a random access memory, a flash memory, a read-only memory, a programmable read-only memory, or an electrically erasable programmable memory, a register, etc. The storage medium is located in the memory, and the processor reads the information in the memory and completes the steps of the above method in combination with its hardware.

[0085] It should also be understood that the memory mentioned in the embodiments of the present invention may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memories. Among them, the non-volatile memory may be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may be a random access memory (RAM), which is used as an external cache. By way of example and not limitation, many forms of RAM are available, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double data rate synchronous DRAM (DDR SDRAM), enhanced SDRAM (ESDRAM), synchlink DRAM (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory of the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0086] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA or other programmable logic device, discrete gate or transistor logic device, discrete hardware component, the memory (storage module) is integrated into the processor. It should be noted that the memory described herein is intended to include but is not limited to these and any other suitable types of memory.

[0087] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0088] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0089] If the functions are implemented in the form of software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can be essentially or partly embodied in the form of a software product that contributes to the prior art. The computer software product is stored in a storage medium and includes several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the methods described in the various embodiments of the present application. The aforementioned storage media include: various media that can store program codes, such as USB flash drives, mobile hard disks, read-only memories (ROM), random access memories (RAM), magnetic disks or optical disks.

[0090] Although the present invention is disclosed as above, the present invention is not limited thereto. Any person skilled in the art can make various changes and modifications without departing from the spirit and scope of the present invention. Therefore, the protection scope of the present invention shall be subject to the scope defined by the claims.

Claims

1. A target behavior recognition method, characterized in that: include: Acquire a video to be identified, wherein the video to be identified includes multiple frames of images to be identified; Performing target detection on each frame of the image to be identified to obtain a moving target and a lane line in each frame of the image to be identified, wherein the moving target is represented by the pixel coordinates of an envelope box, and the lane line is represented by the pixel coordinates in the image to be identified; The moving objects and lane lines identified in each frame of the image to be identified are mapped into the same mapping image according to pixel coordinates, and each frame of the image to be identified has a corresponding mapping image; Determine in each mapped image the contact line between the moving object and the road surface as a horizontal reference line, and determine a first pixel coordinate of the moving object on the horizontal reference line, and a second pixel coordinate and a third pixel coordinate of two lines of the lane line on the horizontal reference line; Calculating the relative position relationship between the first pixel coordinate and the second pixel coordinate and the third pixel coordinate in each mapped image to obtain the behavior time series data of the moving target in the video to be identified, wherein the behavior time series data includes a plurality of relative position relationships arranged in the order of time corresponding to the plurality of frames of images to be identified; Among them, the relative position relationship between the first pixel coordinate and the second pixel coordinate and the third pixel coordinate is expressed by the difference between the first pixel coordinate and the pixel coordinate of the midpoint of the lane line on the horizontal baseline, and the midpoint position of the lane line on the horizontal baseline is the sum of the second pixel coordinate and the third pixel coordinate divided by 2, or the relative position relationship is directly expressed by the second pixel coordinate, the third pixel coordinate and the midpoint position of the lane line on the horizontal baseline.

2. The target behavior recognition method according to claim 1, characterized in that: The calculating the relative position relationship between the first pixel coordinate, the second pixel coordinate and the third pixel coordinate in each mapped image comprises: Determine a fourth pixel coordinate of the center point of the lane line on the horizontal reference line according to the second pixel coordinate and the third pixel coordinate in each mapped image; The difference between the first pixel coordinate and the fourth pixel coordinate in each mapped image is calculated as the relative position relationship.

3. The target behavior recognition method according to claim 1, characterized in that: The calculating the relative position relationship between the first pixel coordinate, the second pixel coordinate and the third pixel coordinate in each mapped image comprises: Determine the width of the lane line and the fourth pixel coordinates of the center point of the lane line on the horizontal baseline according to the second pixel coordinates and the third pixel coordinates in each mapped image; calculate the difference between the first pixel coordinates and the fourth pixel coordinates in each mapped image, and normalize it using the width of the lane line as the relative position relationship.

4. The target behavior recognition method according to claim 1, characterized in that: The target detection of each frame of the image to be identified includes: Each frame of the image to be identified is input into the trained moving target detection model and lane line detection model respectively. The moving target detection model outputs the pixel coordinates of the identified moving target and the lane line detection model outputs the pixel coordinates of the identified lane line.

5. The target behavior recognition method according to claim 4, characterized in that: The moving target is represented by the pixel coordinates of the envelope box, and the lane line is represented by the pixel coordinates in the image to be identified.

6. The target behavior recognition method according to claim 1, characterized in that: After determining the first pixel coordinate of the moving target on the horizontal reference line, and the second pixel coordinate and the third pixel coordinate of the two lane lines on the horizontal reference line, the method further includes: The first pixel coordinates, the second pixel coordinates and the third pixel coordinates corresponding to each frame of the image to be recognized are subjected to Kalman filtering.

7. The target behavior recognition method according to claim 1, characterized in that: Also includes: The intrusion behavior of the moving object into the lane line is classified according to the relative position relationship in the behavior time series data.

8. A target behavior recognition device, characterized in that: include: An acquisition module, used for acquiring a video to be identified, wherein the video to be identified includes multiple frames of images to be identified; The target detection module is used to perform target detection on each frame of the image to be identified, so as to obtain the moving target and lane line in each frame of the image to be identified, wherein the moving target is represented by the pixel coordinates of the envelope box, and the lane line is represented by the pixel coordinates in the image to be identified; A mapping module is used to map the moving targets and lane lines identified in each frame of the image to be identified into the same mapping image according to pixel coordinates, and each frame of the image to be identified has a corresponding mapping image; a pixel coordinate calculation module, for determining in each mapped image the contact line between the moving target and the road surface as a horizontal reference line, and determining a first pixel coordinate of the moving target on the horizontal reference line, and a second pixel coordinate and a third pixel coordinate of two lines of the lane line on the horizontal reference line; A relative position relationship calculation module, used for calculating the relative position relationship between the first pixel coordinate and the second pixel coordinate and the third pixel coordinate in each mapped image, so as to obtain the behavior time series data of the moving target in the video to be identified, wherein the behavior time series data includes a plurality of relative position relationships arranged in the order of time corresponding to the plurality of frames of images to be identified; Among them, the relative position relationship between the first pixel coordinate and the second pixel coordinate and the third pixel coordinate is expressed by the difference between the first pixel coordinate and the pixel coordinate of the midpoint of the lane line on the horizontal baseline, and the midpoint position of the lane line on the horizontal baseline is the sum of the second pixel coordinate and the third pixel coordinate divided by 2, or the relative position relationship is directly expressed by the second pixel coordinate, the third pixel coordinate and the midpoint position of the lane line on the horizontal baseline.

9. A storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the target behavior recognition method according to any one of claims 1 to 7 are executed.

10. A terminal comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, characterized in that: When the processor runs the computer program, the steps of the target behavior recognition method according to any one of claims 1 to 7 are performed.

Images