Target motion recognition method, model training method, device and electronic equipment

Abstract

The present disclosure provides a target motion recognition method, a model training method, a device, and an electronic device, which relate to the fields of image processing technology, and in particular to the fields of computer vision and deep learning technology. The specific implementation scheme of the target motion recognition method is as follows: obtaining a target video frame, and determining an optical flow feature map corresponding to the target video frame based on an optical flow algorithm; determining a target object in the target video frame, and obtaining a first area where the target object is located in the target video frame, and obtaining a second area where the target object is located in the optical flow feature map; splicing the first area and the second area to obtain a spliced image; and identifying the spliced image to determine whether the target object has moved.

Description

Technical Field

[0001] The present disclosure relates to the field of image processing technology, in particular to the field of computer vision and deep learning technology, and specifically to a target motion recognition method, a model training method, a device, and an electronic device. Background Art

[0002] Multi-target motion recognition involves identifying the presence of target types within a video image that continuously changes over time, and determining whether each target is moving or experiencing any changes in motion. In security monitoring scenarios, such as railway and construction site safety monitoring, it is often necessary to identify the operating status of multiple targets and then use judgment logic to determine whether there are any dangers within the monitored scene to ensure safety. Summary of the Invention

[0003] The present disclosure provides a target motion recognition method, a model training method, a device, and an electronic device.

[0004] According to a first aspect of the present disclosure, a method for identifying target movement is provided, comprising:

[0005] Obtain a target video frame, and determine an optical flow feature map corresponding to the target video frame based on an optical flow algorithm;

[0006] Determine a target object in the target video frame, obtain a first region in the target video frame where the target object is located, and obtain a second region in the optical flow feature map where the target object is located;

[0007] splicing the first area and the second area to obtain a spliced image;

[0008] The spliced image is identified to determine whether the target object moves.

[0009] According to a second aspect of the present disclosure, a model training method is provided, comprising:

[0010] Acquire training data; wherein the training data includes a first image and a second image, the first image is obtained by splicing a first original image and an optical flow feature map corresponding to the first original image, and the first original image includes a moving object; the second image is obtained by splicing a second original image and an optical flow feature map corresponding to the second original image, and the second original image includes a stationary object;

[0011] The training data is input into a motion recognition model, and the motion recognition model is trained based on the input training data.

[0012] According to a third aspect of the present disclosure, there is provided a target movement identification device, comprising:

[0013] A determination module is used to obtain a target video frame and determine an optical flow feature map corresponding to the target video frame based on an optical flow algorithm;

[0014] a first acquisition module, configured to determine a target object in the target video frame, and acquire a first region in the target video frame where the target object is located, and acquire a second region in the optical flow feature map where the target object is located;

[0015] a stitching module, configured to stitch the first area and the second area together to obtain a stitched image;

[0016] The recognition module is used to recognize the spliced image to determine whether the target object moves.

[0017] According to a fourth aspect of the present disclosure, a model training device is provided, comprising:

[0018] a second acquisition module, configured to acquire training data; wherein the training data includes a first image and a second image, the first image being obtained by concatenating a first original image and an optical flow feature map corresponding to the first original image, and the first original image including a moving object; and the second image being obtained by concatenating a second original image and an optical flow feature map corresponding to the second original image, and the second original image including a stationary object;

[0019] The training module is used to input the training data into the motion recognition model and train the motion recognition model based on the input training data.

[0020] According to a fifth aspect of the present disclosure, there is provided an electronic device, including:

[0021] at least one processor; and

[0022] a memory communicatively connected to the at least one processor; wherein,

[0023] The memory stores instructions that can be executed by the at least one processor. The instructions are executed by the at least one processor to enable the at least one processor to perform the method according to the first aspect or the second aspect.

[0024] According to a sixth aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute the method as described in the first aspect or the second aspect.

[0025] According to a seventh aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to the first aspect or the second aspect.

[0026] In the embodiment of the present disclosure, the stitched image can fuse the change information of the target object in time and space respectively, and determine whether the target object has moved by identifying the stitched image, and then identify whether the target object in the stitched image after time and space fusion has moved. This recognition method not only focuses on the changes of the target object in time, but also focuses on the changes of the target object in space, thereby effectively improving the accuracy of the target object movement recognition.

[0027] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] The accompanying drawings are provided to facilitate a better understanding of the present invention and do not constitute a limitation of the present disclosure.

[0029] Figure 1 is a flow chart of a target movement identification method provided by an embodiment of the present disclosure;

[0030] Figure 2 is a flow chart of another target movement identification method provided by an embodiment of the present disclosure;

[0031] Figure 3 is a structural diagram of a target movement identification device provided by an embodiment of the present disclosure;

[0032] Figure 4 is a flowchart of a model training method provided by an embodiment of the present disclosure;

[0033] Figure 5 is a structural diagram of a model training device provided by an embodiment of the present disclosure;

[0034] Figure 6 It is a block diagram of an electronic device used to implement the target movement recognition method or model training method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0036] Please refer to Figure 1 , Figure 1 is a flow chart of a target movement identification method provided by an embodiment of the present disclosure, such as Figure 1 As shown, the method includes the following steps:

[0037] Step S101: Acquire a target video frame, and determine an optical flow feature map corresponding to the target video frame based on an optical flow algorithm.

[0038] It should be noted that the target motion recognition method provided in the embodiments of the present disclosure can be applied to electronic devices such as computers, mobile phones, and tablet computers. The target video frame can be a video frame captured by the electronic device based on its camera, or can also be a video frame obtained by other means, such as downloading from the Internet or sending from other devices.

[0039] Optionally, the target video frame may be any frame other than the first frame in the video stream acquired by the electronic device. In this step, after determining the target video frame, the electronic device determines the optical flow feature map corresponding to the target video frame based on the optical flow algorithm. Exemplarily, the dense optical flow field of the target video frame may be calculated by the optical flow algorithm based on the video stream image of the target video frame and the previous frame of the target video frame, thereby obtaining the displacement information of each pixel in the target video frame, and obtaining the optical flow feature map corresponding to the target video frame based on the displacement information.

[0040] It can be understood that an optical flow field refers to a two-dimensional instantaneous velocity field formed by all pixels in an image. In the disclosed embodiments, an optical flow algorithm is used to obtain a dense optical flow field of the target video frame, thereby obtaining a two-dimensional instantaneous velocity field formed by all pixels in the target video frame. In other words, the displacement information of each pixel in the target video frame can be obtained. Based on this displacement information, an optical flow feature map corresponding to the target video frame is determined. The optical flow feature map can also represent the displacement information of each pixel in the target video frame.

[0041] Optionally, the optical flow algorithm used in the embodiments of the present disclosure is not limited to traditional methods or deep learning methods. Traditional methods include opencv's calcOpticalFlowFarneback and deep learning methods such as FlowNet, which are not described in detail in this disclosure.

[0042] Step S102: determine the target object in the target video frame, obtain a first area where the target object is located in the target video frame, and obtain a second area where the target object is located in the optical flow feature map.

[0043] Optionally, the electronic device may use a corresponding target detection model to determine the target object in the target video frame. For example, the target detection model is not limited to YOLO, FasterRcnn, etc., and this disclosure does not make specific limitations on this.

[0044] It should be noted that the target detection model can be used to detect target objects in video frames to identify target objects in video frames. For example, the target object can be an object that can move in a video frame, such as a person, an animal, a vehicle, etc. It can be understood that the target detection model can be trained based on a large amount of training data. For example, the training data can be video frames including movable objects such as people and vehicles, and video frames including stationary objects that cannot move such as buildings, road signs, roads, etc. The target detection model is trained based on these training data, so that the trained target detection model can identify objects that can move in video frames, that is, the target objects.

[0045] In the disclosed embodiment, after acquiring the target video frame, the electronic device may identify the target video frame based on a pre-trained target detection model, so as to identify the target object in the target video frame through the target detection model, that is, the object that can move in the target video frame. It should be noted that the target detection model is only used to identify the target object that can move in the target video frame, and does not mean that the target object is in a moving state. For example, the target detection model identifies that the target object in the target video frame includes a vehicle. The vehicle may be stationary, such as parked on the side of the road, or the vehicle may be driving on the road. As to whether the vehicle is moving, it needs to be determined through subsequent steps. The target detection model cannot identify whether the target object is moving.

[0046] Optionally, in the process of identifying the target video frame, the target detection model can determine the target object by outputting the position area coordinates of the target object in the target video frame, and then can determine the area where the target object is located based on the position area coordinates, that is, can obtain the first area where the target object is located in the target video frame.

[0047] It is understood that the optical flow feature map is obtained by using an optical flow algorithm for the target video frame, and has the same size and coordinate range as the target video frame. Furthermore, after determining the coordinates of the target object's location region in the target video frame, the second region corresponding to the coordinates of the location region in the optical flow feature map can be determined, i.e., the location of the target object in the optical flow feature map.

[0048] In the embodiment of the present disclosure, the target video frame may be cropped based on the position region coordinates, that is, the region corresponding to the position region coordinates in the target video frame is cropped, that is, the first region is obtained by cropping; and the optical flow feature map may be cropped based on the position region coordinates, that is, the second region is obtained by cropping. It is understood that both the first region and the second region include the target object.

[0049] Step S103: stitching the first area and the second area to obtain a stitched image.

[0050] Optionally, after cropping the first and second regions in the above manner, the first and second regions, i.e., two images including the target object, can be stitched together by channel to obtain a stitched image. It is understood that the stitched image also includes the target object.

[0051] Step S104: Identify the spliced image to determine whether the target object moves.

[0052] In an embodiment of the present disclosure, the cropped first area and the second area are spliced together to obtain a spliced image, which includes a target object. The spliced image is further identified based on a motion recognition model to determine whether the target object in the spliced image moves.

[0053] It is understandable that the target object is identified based on a pre-trained target detection model, which is used to identify objects that can move in an image, that is, the identified target object itself is able to move. In the embodiment of the present disclosure, after the first area including the target object and the second area are spliced to obtain a spliced image, the spliced image is identified. For example, the spliced image can be identified based on a pre-trained motion recognition model to determine whether the target object in the spliced image is moving. In this way, it is possible to determine whether the target object in the target video frame is in a moving state.

[0054] In the embodiment of the present disclosure, after acquiring the target video frame, the electronic device determines the optical flow feature map corresponding to the target video frame based on the optical flow algorithm, so that the temporal change information of the target video frame can be obtained. Further, the target object in the target video frame is determined, and a first area where the target object is located in the target video frame and a second area where the target object is located in the optical flow feature map are obtained. The first area is obtained based on the target video frame, and the image of the first area also represents the spatial change information of the target object. The second area is obtained based on the optical flow feature map, and the image of the second area also represents the temporal change information of the target object. The first area and the second area are spliced to obtain a spliced image, which is a fusion of the temporal and spatial change information of the target object. By identifying the spliced image, it is determined whether the target object has moved, and then whether the target object in the spliced image after spatiotemporal fusion has moved. This recognition method not only focuses on the temporal change of the target object, but also focuses on the spatial change of the target object, thereby effectively improving the accuracy of the target object's motion recognition.

[0055] The target motion recognition method provided by the embodiment of the present disclosure has a stitched image that effectively integrates temporal changes and spatial features, which can effectively avoid misidentification caused by camera shaking. In addition, by determining the target object in the target video frame, cropping the first area where the target object is located in the target video frame and the second area where the target object is located in the optical flow feature map, only the stitched image obtained by stitching the first area and the second area is recognized, which effectively narrows the recognition range and avoids interference from other moving objects in the target video frame, thereby effectively improving the recognition accuracy of whether the target object is moving.

[0056] It should be noted that, in the process of determining the target object in the target video frame, the identified target object may be one or more. For example, if the target video frame includes multiple movable objects, such as multiple vehicles, one vehicle may be determined as one target object, that is, multiple target objects can be identified. Furthermore, the position area coordinates corresponding to each target object may be determined separately, and the target video frame and the optical flow feature map may be cropped based on the position area coordinates to obtain the first area and the second area corresponding to each target object, and then the first area and the second area may be spliced to obtain a spliced image after each target object is cropped and spliced. Each spliced image is identified separately based on the motion recognition model to determine whether the target object in the spliced image is moving. In this way, motion recognition of multiple target objects in the target video frame can be achieved, thereby making the motion recognition of multiple targets faster. The multi-target motion recognition achieved based on the spatiotemporal fusion method provided in the present disclosure also makes its recognition accuracy higher.

[0057] Optionally, step S104 may include:

[0058] The spliced image is identified based on a motion recognition model to determine whether the target object is moving.

[0059] It should be noted that the motion recognition model may be pre-trained based on a large amount of training data, so that the trained motion recognition model can recognize whether the target object in the image is moving.

[0060] The training data of the motion recognition model includes a first image and a second image, wherein the first image is obtained by splicing a first original image and an optical flow feature map corresponding to the first original image, and the first original image includes a moving object, and the second image is obtained by splicing a second original image and an optical flow feature map corresponding to the second original image, and the second original image includes a stationary object. In other words, the first image and the second image are both spliced images obtained by splicing original images and corresponding optical flow feature maps, so that the training data of the motion recognition model is also spliced images, and the first image includes a moving object, and the second image includes a stationary object. Then, the motion recognition model is trained by the first image and the second image, so that the trained motion recognition model can recognize the spliced image and can recognize whether the target object in the spliced image is in a moving state or a stationary state.

[0061] In the embodiment of the present disclosure, the first image and the second image are also spliced images that fuse temporal change information and spatial change information. Therefore, the motion recognition model trained based on the first image and the second image can realize the motion recognition of the target object in the spliced image after temporal and spatial fusion, thereby effectively improving the accuracy of motion recognition of the target object.

[0062] Optionally, in the embodiment of the present disclosure, step S101 may further include:

[0063] Acquire a target video frame, and determine an optical flow field of the target video frame based on an optical flow algorithm;

[0064] An optical flow feature map is determined according to the movement information of each pixel in the optical flow field.

[0065] In the embodiment of the present disclosure, after acquiring the target video frame, the electronic device can calculate the dense optical flow field of the target video frame based on the target video frame and the video stream image of the previous frame of the target video frame based on the optical flow algorithm, thereby obtaining the movement information of each pixel point in the optical flow field, that is, the displacement information. Furthermore, based on the movement information of each pixel point in the optical flow field, the optical flow feature map corresponding to the target video frame is determined. The optical flow feature map can also characterize the movement information of each pixel in the target video frame, thereby obtaining the characteristic expression of the target video frame based on pixel displacement, thereby obtaining the expression of the target object in the target video frame in terms of time change, thereby providing a basis for subsequent spatiotemporal fusion and helping to improve the accuracy of target movement recognition.

[0066] Optionally, determining the optical flow feature map according to the movement information of each pixel in the optical flow field includes:

[0067] Acquire first movement information of each pixel point in the optical flow field in a first direction and second movement information in a second direction, where the first direction is perpendicular to the second direction;

[0068] determining a displacement of a target pixel point according to first movement information of the target pixel point and second movement information of the target pixel point, where the target pixel point is any pixel point in the optical flow field;

[0069] The displacement of each pixel in the optical flow field is normalized, and an optical flow feature map is obtained based on the normalized displacement of each pixel.

[0070] The first direction and the second direction may be the x-axis direction and the y-axis direction of a rectangular coordinate system in the optical flow field, respectively. Optionally, first movement information in the x-axis direction and second movement information in the y-axis direction are obtained for each pixel in the optical flow field, and the displacement of the pixel is calculated based on the first movement information and the second movement information.

[0071] For example, taking a target pixel in an optical flow field as an example, first movement information of the target pixel in the x-axis direction and second movement information of the target pixel in the y-axis direction are obtained. The displacement of the target pixel can be calculated based on the following formula:

[0072]

[0073] Wherein, Flow_x is the first movement information of the target pixel point, Flow_y is the second movement information of the target pixel point, and Flow is the displacement of the target pixel point.

[0074] Furthermore, based on the above method, the displacement of each pixel in the optical flow field can be obtained.

[0075] Furthermore, the displacement of each pixel in the optical flow field is normalized, and an optical flow feature map is determined based on the normalized displacement of each pixel. For example, normalization can be achieved by dividing each pixel in the optical flow field by a preset value, or by dividing each pixel in the optical flow field by the sum of all pixels.

[0076] In the embodiment of the present disclosure, the displacement of the corresponding pixel is calculated based on the movement information of each pixel in the optical flow field in the x-axis direction and the y-axis direction, thereby obtaining the displacement of each pixel in the optical flow field. In this way, the characteristic expression method of the target video frame based on the pixel displacement can be obtained, thereby realizing the expression of the target object in the target video frame in terms of time change, so that the characteristic data of the moving area is obviously different from the static area, thereby better realizing the judgment of whether the target object is moving. Furthermore, by normalizing the displacement of each pixel, the characteristic value of each pixel in the optical flow field can be limited to a certain range, so that each pixel is at the same order of magnitude, thereby better realizing the comparison of the characteristic values of the pixel points, so as to eliminate the adverse effects of the data with relatively singular characteristic values of the pixel points in the optical flow field, thereby also helping to improve the accuracy of the judgment of whether the target object is moving.

[0077] Optionally, in one embodiment, the normalizing the displacement of each pixel in the optical flow field includes:

[0078] Obtaining a maximum displacement and a minimum displacement among the displacements of all pixels in the optical flow field;

[0079] Based on the maximum displacement and the minimum displacement, the displacement of each pixel in the optical flow field is normalized.

[0080] In this embodiment, the normalization process can be implemented based on the maximum displacement and the minimum displacement of all pixels in the optical flow field. For example, the normalization process of the pixels can be obtained based on the following formula:

[0081]

[0082] Among them, Flow ′ is the displacement of the normalized pixel, Flow is the displacement of the pixel before normalization, max(Flow) is the maximum displacement of all pixels in the optical flow field, and min(Flow) is the minimum displacement of all pixels in the optical flow field. Based on the above formula, all pixels in the optical flow field are normalized to obtain the normalized displacement of each pixel.

[0083] In this embodiment, by obtaining the maximum displacement and the minimum displacement of all pixel points in the optical flow field, the displacement of each pixel point in the optical flow field is normalized based on the maximum displacement and the minimum displacement, so that the characteristic value of each pixel point in the optical flow field can be limited to a certain range, thereby better realizing the comparison of the characteristic values of the pixel points, so that the characteristic values of the moving area can be clearly distinguished from the static area, which can help to improve the accuracy of judging whether the target object is moving.

[0084] Optionally, in another implementation, the normalizing the displacement of each pixel in the optical flow field includes:

[0085] Acquire a target area where a target pixel point in the optical flow field is located, where the pixel points corresponding to the target object include the target pixel point;

[0086] Obtaining the maximum displacement and the minimum displacement among the displacements of all pixels in the target area;

[0087] Based on the maximum displacement and the minimum displacement, the displacement of each pixel in the optical flow field is normalized.

[0088] In this embodiment, the target pixel is the pixel corresponding to the target object, and the target area where the target pixel is located can also be the area where the target object is located. The maximum displacement and the minimum displacement of all the pixel points in the target area are obtained, and the displacement of each pixel point in the optical flow field is normalized based on the maximum displacement and the minimum displacement in the target area. Optionally, the normalization of the pixel points can be obtained based on the following formula:

[0089]

[0090] Among them, Flow ′ is the displacement of the pixel after normalization, Flow is the displacement of the pixel before normalization, max(Flow) is the maximum displacement of the pixel points in the target area, and min(Flow) is the minimum displacement of the pixel points in the target area.

[0091] In this embodiment, the pixels in the optical flow field are normalized by the maximum displacement and the minimum displacement in the target area where the target object is located, so that the characteristic values of the normalized pixels can be better in the same order of magnitude as the characteristic values of the pixels corresponding to the target object, thereby better realizing the comparison of the characteristic values of the pixels, thereby helping to improve the accuracy of judging whether the target object has moved based on the pixel displacement.

[0092] Please refer to Figure 2 , Figure 2 is a flow chart of another target movement identification method provided by an embodiment of the present disclosure, such as Figure 2 As shown, the method includes the following steps:

[0093] Step S201: Obtain video stream;

[0094] Step S202: extracting a video frame from the video stream, that is, a target video frame;

[0095] Step S203: Calculate the optical flow field of the video frame, such as a dense optical flow field;

[0096] Step S204: calculating the displacement of each pixel in the optical flow field;

[0097] Step S205: generating an optical flow feature map based on the displacement of each pixel in the optical flow field;

[0098] Step S206: detecting a target to be identified, so as to identify a target object in the video frame;

[0099] Step S207: based on the target area of the identified target object in the optical flow feature map, cropping the target area of the optical flow feature map;

[0100] Step S208: cropping the target area of the original video frame based on the identified target object;

[0101] Step S209: fusing (i.e., stitching) the target area cropped from the optical flow feature map with the target area cropped from the original video frame to obtain a fused image.

[0102] Step S210: performing target motion recognition on the fused image to identify whether the target object is moving.

[0103] In the disclosed embodiment, the fused image effectively integrates temporal changes and spatial features, can effectively avoid misidentification caused by camera shaking, and effectively improves the accuracy of identifying whether the target object is moving.

[0104] It should be noted that the specific implementation process of the above steps in the embodiment of the present disclosure and the related concepts involved can be referred to Figure 1 To avoid repetition, the description in the embodiments will not be repeated here.

[0105] Please refer to Figure 3 , Figure 3 is a structural diagram of a target movement identification device provided by an embodiment of the present disclosure, such as Figure 3 As shown, the target movement identification device 300 includes:

[0106] The determination module 301 is configured to obtain a target video frame and determine an optical flow feature map corresponding to the target video frame based on an optical flow algorithm;

[0107] A first acquisition module 302 is configured to determine a target object in the target video frame, and acquire a first region in the target video frame where the target object is located, and acquire a second region in the optical flow feature map where the target object is located;

[0108] A stitching module 303 is configured to stitch the first area and the second area to obtain a stitched image;

[0109] The recognition module 304 is configured to recognize the spliced image to determine whether the target object is moving.

[0110] Optionally, the determining module 301 includes:

[0111] A first determining unit is configured to obtain a target video frame and determine an optical flow field of the target video frame based on an optical flow algorithm;

[0112] The second determining unit is configured to determine an optical flow feature map according to the movement information of each pixel in the optical flow field.

[0113] Optionally, the second determining unit is further configured to:

[0114] Acquire first movement information of each pixel point in the optical flow field in a first direction and second movement information in a second direction, where the first direction is perpendicular to the second direction;

[0115] determining a displacement of a target pixel point according to first movement information of the target pixel point and second movement information of the target pixel point, where the target pixel point is any pixel point in the optical flow field;

[0116] The displacement of each pixel in the optical flow field is normalized, and an optical flow feature map is obtained based on the normalized displacement of each pixel.

[0117] Optionally, the second determining unit is further configured to:

[0118] Obtaining a maximum displacement and a minimum displacement among the displacements of all pixels in the optical flow field;

[0119] Based on the maximum displacement and the minimum displacement, the displacement of each pixel in the optical flow field is normalized.

[0120] Optionally, the second determining unit is further configured to:

[0121] Acquire a target area where a target pixel point in the optical flow field is located, where the pixel points corresponding to the target object include the target pixel point;

[0122] Obtaining the maximum displacement and the minimum displacement among the displacements of all pixels in the target area;

[0123] Based on the maximum displacement and the minimum displacement, the displacement of each pixel in the optical flow field is normalized.

[0124] Optionally, the identification module 304 is further configured to:

[0125] Identify the spliced image based on a motion recognition model to determine whether the target object is moving;

[0126] In which, the training data of the motion recognition model includes a first image and a second image, the first image is obtained by splicing the first original image and the optical flow feature map corresponding to the first original image, and the first original image includes a moving object, and the second image is obtained by splicing the second original image and the optical flow feature map corresponding to the second original image, and the second original image includes a stationary object.

[0127] In the embodiment of the present disclosure, the stitched image can fuse the change information of the target object in time and space respectively, and determine whether the target object has moved by identifying the stitched image, and then identify whether the target object in the stitched image after time and space fusion has moved. This recognition method not only focuses on the changes of the target object in time, but also focuses on the changes of the target object in space, thereby effectively improving the accuracy of the target object movement recognition.

[0128] It should be noted that the target motion identification device provided in the embodiment of the present disclosure can implement the entire process of the above-mentioned target motion identification method and can achieve the same technical effect. To avoid repetition, it will not be described here.

[0129] Please refer to Figure 4 , Figure 4 This is a flow chart of a model training method provided by an embodiment of the present disclosure. Figure 4 As shown, the model training method includes the following steps:

[0130] Step S401: Obtain training data.

[0131] The training data includes a first image and a second image, the first image is obtained by splicing a first original image and an optical flow feature map corresponding to the first original image, and the first original image includes a moving object, and the second image is obtained by splicing a second original image and an optical flow feature map corresponding to the second original image, and the second original image includes a stationary object.

[0132] It should be noted that the first original image and the second original image are different images. The first original image includes moving objects, such as moving vehicles, people, etc.; the first original image is processed based on the optical flow algorithm to obtain a corresponding optical flow feature map, and the optical flow feature map is spliced with the first original image to obtain a first image; or, the area corresponding to the moving object in the optical flow feature map is spliced with the area corresponding to the moving object in the first original image to obtain the first image. Among them, the area corresponding to the moving object A in the optical flow feature map is spliced with the area corresponding to the moving object A in the first original image to obtain a first image, and the area corresponding to the moving object B in the optical flow feature map is spliced with the area corresponding to the moving object B in the first original image to obtain another first image, that is, the areas where different moving objects are located in the optical flow feature map are spliced with the areas corresponding to the moving objects in the first original image, and thus multiple first images including moving objects can be obtained, which effectively expands the sample richness of the training data.

[0133] The second original image includes stationary objects, such as buildings, road signs, and street lights. Based on similar processing, the second original image is processed by an optical flow algorithm to obtain a corresponding optical flow feature map, and the optical flow feature map is spliced with the second original image to obtain a second image. Alternatively, the area corresponding to the stationary object in the optical flow feature map is spliced with the area corresponding to the stationary object in the second original image to obtain a second image. The area corresponding to stationary object A in the optical flow feature map is spliced with the area corresponding to stationary object A in the second original image to obtain a second image, and the area corresponding to stationary object B in the optical flow feature map is spliced with the area corresponding to stationary object B in the second original image to obtain another second image. That is, the areas where different stationary objects are located in the optical flow feature map are spliced with the areas corresponding to the stationary objects in the second original image, thereby obtaining multiple second images including stationary objects, effectively expanding the sample richness of the training data.

[0134] It should be noted that the specific process of obtaining the optical flow feature map corresponding to the first original image or the second original image based on the optical flow algorithm can be the steps of obtaining the optical flow feature map by referring to the optical flow algorithm in the above embodiment, and the present disclosed embodiment will not go into details.

[0135] Step S402: Input the training data into a motion recognition model, and train the motion recognition model based on the input training data.

[0136] The trained motion recognition model is used to recognize a target object in an input image to determine whether the target object is moving.

[0137] In an embodiment of the present disclosure, the training data input to the motion recognition model includes a first image and a second image, wherein the first image includes a moving object and the second image includes a stationary object. The motion recognition model is trained by comparing feature values between the first image and the second image, so that the trained motion recognition model can recognize the target object in the input image to determine whether the target object in the image is moving. Optionally, the motion recognition model can adopt a convolutional neural network (CNN) classification model, including but not limited to Resnet, Densenet, etc., which is not specifically limited in the present disclosure.

[0138] It should be noted that the image used to input the mobile recognition model for recognition can also be a spliced image obtained by splicing the original image and the corresponding optical flow feature map. The acquisition of the optical flow feature map and the acquisition of the spliced image can refer to the description in the above embodiment, and the embodiments of this disclosure will not be described in detail.

[0139] In an embodiment of the present disclosure, the training data used to train the motion recognition model includes a first image and a second image, the first image is obtained by splicing a first original image and an optical flow feature map corresponding to the first original image, and the first original image includes a moving object, and the second image is obtained by splicing a second original image and an optical flow feature map corresponding to the second original image, and the second original image includes a stationary object. In this way, the trained motion recognition model can identify whether the object in the input image is moving, thereby effectively improving the efficiency of image recognition, and the training data is a spliced image obtained by splicing the optical flow feature map and the original image, so that the trained motion recognition model can identify whether the target object in the spliced image after time-space fusion is moving. This recognition method not only focuses on the changes of the target object in time, but also focuses on the changes of the target object in space, thereby effectively improving the recognition accuracy of the motion recognition model corresponding to whether the target object is moving.

[0140] Please refer to Figure 5 , Figure 5 This is a structural diagram of a model training device provided by an embodiment of the present disclosure. Figure 5 As shown, the model training device 500 includes:

[0141] A second acquisition module 501 is configured to acquire training data, wherein the training data includes a first image and a second image, wherein the first image is obtained by concatenating a first original image and an optical flow feature map corresponding to the first original image, and the first original image includes a moving object; and the second image is obtained by concatenating a second original image and an optical flow feature map corresponding to the second original image, and the second original image includes a stationary object.

[0142] The training module 502 is configured to input the training data into a motion recognition model and train the motion recognition model based on the input training data.

[0143] The trained motion recognition model is used to recognize a target object in an input image to determine whether the target object is moving.

[0144] It should be noted that the model training device 500 provided in the embodiment of the present disclosure can implement the entire process of the above-mentioned model training method and achieve the same technical effect. To avoid repetition, it will not be described here.

[0145] In the technical solutions disclosed herein, the acquisition, storage, and application of user personal information involved comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0146] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0147] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0148] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. Various programs and data required for the operation of the device 600 can also be stored in the RAM 603. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0149] Various components in device 600 are connected to I / O interface 605, including an input unit 606, such as a keyboard, mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a magnetic disk, optical disk, etc.; and a communication unit 609, such as a network card, modem, wireless communication transceiver, etc. The communication unit 609 allows device 600 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0150] The computing unit 601 can be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units that run machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as the target movement recognition method or the model training method. For example, in some embodiments, the target movement recognition method or the model training method can be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program can be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the target movement recognition method or the model training method described above can be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the above-mentioned target movement recognition method or model training method in any other appropriate manner (for example, by means of firmware).

[0151] Various embodiments of the systems and techniques described herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-chip systems (SOCs), programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0152] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0153] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0154] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0155] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0156] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises through computer programs running on the respective computers and having a client-server relationship with each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0157] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not limited herein.

[0158] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for identifying target movement, comprising: Obtain a target video frame, and determine an optical flow feature map corresponding to the target video frame based on an optical flow algorithm; Determine a target object in the target video frame, obtain a first region in the target video frame where the target object is located, and obtain a second region in the optical flow feature map where the target object is located; splicing the first area and the second area to obtain a spliced image; Identifying the stitched image to determine whether the target object moves; The identifying the spliced image to determine whether the target object has moved includes: Identify the spliced image based on a motion recognition model to determine whether the target object is moving; In which, the training data of the motion recognition model includes a first image and a second image, the first image is obtained by splicing the first original image and the optical flow feature map corresponding to the first original image, and the first original image includes a moving object, and the second image is obtained by splicing the second original image and the optical flow feature map corresponding to the second original image, and the second original image includes a stationary object.

2. The method according to claim 1, wherein The acquiring of the target video frame and determining the optical flow feature map corresponding to the target video frame based on the optical flow algorithm includes: Acquire a target video frame, and determine an optical flow field of the target video frame based on an optical flow algorithm; An optical flow feature map is determined according to the movement information of each pixel in the optical flow field.

3. The method according to claim 2, wherein: The determining of the optical flow feature map according to the movement information of each pixel in the optical flow field includes: Acquire first movement information of each pixel point in the optical flow field in a first direction and second movement information in a second direction, where the first direction is perpendicular to the second direction; determining a displacement of a target pixel point according to first movement information of the target pixel point and second movement information of the target pixel point, where the target pixel point is any pixel point in the optical flow field; The displacement of each pixel in the optical flow field is normalized, and an optical flow feature map is obtained based on the normalized displacement of each pixel.

4. The method according to claim 3, wherein: The normalizing the displacement of each pixel in the optical flow field includes: Obtaining a maximum displacement and a minimum displacement among the displacements of all pixels in the optical flow field; Based on the maximum displacement and the minimum displacement, the displacement of each pixel in the optical flow field is normalized.

5. The method according to claim 3, wherein: The normalizing the displacement of each pixel in the optical flow field includes: Acquire a target area where a target pixel point in the optical flow field is located, where the pixel points corresponding to the target object include the target pixel point; Obtaining the maximum displacement and the minimum displacement among the displacements of all pixels in the target area; Based on the maximum displacement and the minimum displacement, the displacement of each pixel in the optical flow field is normalized.

6. A model training method comprising: Acquire training data; wherein the training data includes a first image and a second image, the first image is obtained by splicing a first original image and an optical flow feature map corresponding to the first original image, and the first original image includes a moving object; the second image is obtained by splicing a second original image and an optical flow feature map corresponding to the second original image, and the second original image includes a stationary object; Inputting the training data into a motion recognition model, and training the motion recognition model based on the input training data; The trained motion recognition model is used to execute the target motion recognition method according to any one of claims 1 to 5.

7. A target movement identification device comprising: A determination module is used to obtain a target video frame and determine an optical flow feature map corresponding to the target video frame based on an optical flow algorithm; a first acquisition module, configured to determine a target object in the target video frame, and acquire a first region in the target video frame where the target object is located, and acquire a second region in the optical flow feature map where the target object is located; a stitching module, configured to stitch the first area and the second area together to obtain a stitched image; an identification module, configured to identify the spliced image to determine whether the target object has moved; The identification module is further configured to: Identify the spliced image based on a motion recognition model to determine whether the target object is moving; In which, the training data of the motion recognition model includes a first image and a second image, the first image is obtained by splicing the first original image and the optical flow feature map corresponding to the first original image, and the first original image includes a moving object, and the second image is obtained by splicing the second original image and the optical flow feature map corresponding to the second original image, and the second original image includes a stationary object.

8. The device according to claim 7, wherein The determination module includes: A first determining unit is configured to obtain a target video frame and determine an optical flow field of the target video frame based on an optical flow algorithm; The second determining unit is configured to determine an optical flow feature map according to the movement information of each pixel in the optical flow field.

9. The device according to claim 8, wherein The second determining unit is further configured to: Acquire first movement information of each pixel point in the optical flow field in a first direction and second movement information in a second direction, where the first direction is perpendicular to the second direction; determining a displacement of a target pixel point according to first movement information of the target pixel point and second movement information of the target pixel point, where the target pixel point is any pixel point in the optical flow field; The displacement of each pixel in the optical flow field is normalized, and an optical flow feature map is obtained based on the normalized displacement of each pixel.

10. The device according to claim 9, wherein The second determining unit is further configured to: Obtaining a maximum displacement and a minimum displacement among the displacements of all pixels in the optical flow field; Based on the maximum displacement and the minimum displacement, the displacement of each pixel in the optical flow field is normalized.

11. The device according to claim 9, wherein The second determining unit is further configured to: Acquire a target area where a target pixel point in the optical flow field is located, where the pixel points corresponding to the target object include the target pixel point; Obtaining the maximum displacement and the minimum displacement among the displacements of all pixels in the target area; Based on the maximum displacement and the minimum displacement, the displacement of each pixel in the optical flow field is normalized.

12. A model training device comprising: a second acquisition module, configured to acquire training data; wherein the training data includes a first image and a second image, the first image being obtained by concatenating a first original image and an optical flow feature map corresponding to the first original image, and the first original image including a moving object; and the second image being obtained by concatenating a second original image and an optical flow feature map corresponding to the second original image, and the second original image including a stationary object; a training module, configured to input the training data into a motion recognition model and train the motion recognition model based on the input training data; The trained motion recognition model is used to execute the target motion recognition method according to any one of claims 1 to 5.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images