Image processing method, recognition model and electronic device

Abstract

The embodiments of the present application provide an image processing method, a recognition model and an electronic device. The method includes: acquiring multiple frames of images; determining a first spatiotemporal feature of the multiple frames of images and a first inter-frame continuity feature of adjacent frames of the multiple frames of images; wherein the first inter-frame continuity feature reflects the degree of continuity in the motion of the target object in adjacent frames of images; adjusting the first spatiotemporal feature according to the first inter-frame continuity feature; and using the adjusted first spatiotemporal feature to identify the motion behavior of the target object appearing in at least part of the multiple frames of images. The technical solution provided in the embodiments of the present application uses the inter-frame continuity feature of adjacent frames to adjust the spatiotemporal features of multiple frames of images, which can weaken the influence of frames with poor inter-frame continuity and strengthen the correlation between frames with strong inter-frame continuity, and is suitable for behavior recognition in videos with rapid camera movement or transitions.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to an image processing method, a recognition model and an electronic device. Background Art

[0002] With the development of computer technology and 5G technology, video-related technologies have also been widely used, such as social video applications, video surveillance applications, human-computer interaction applications and other application scenarios.

[0003] In some application scenarios, it is necessary to perform feature recognition on a target object in a video. Specifically, after obtaining multiple continuous frames of images from the video, the behavior of the target object is identified from the multiple frames. In the recognition process based on the prior art, it is necessary not only to ensure that the brightness of each frame of the video is relatively constant, but also to ensure that the inter-frame continuity of the multiple frames of video images is relatively good, in order to accurately identify the behavioral characteristics of the target object. However, in actual applications, due to the rapid movement of the target or the rapid movement of the video acquisition angle, the continuity between video frames is poor, which affects the accurate recognition of the behavioral characteristics of the target object in the video frame. Therefore, there is a need for a solution that can be applied to target object behavior recognition under various video conditions. Summary of the invention

[0004] In order to solve or improve the problems existing in the prior art, each embodiment of the present application provides an image processing method, a recognition model and an electronic device.

[0005] In a first aspect, in one embodiment of the present application, an image processing method is provided. The method comprises:

[0006] Acquire multiple frames of images;

[0007] Determining a first spatiotemporal feature of the multiple frames of images and a first inter-frame continuity feature of adjacent frame images in the multiple frames of images; wherein the first inter-frame continuity feature reflects the degree of coherence of motion coherence of the target object in adjacent frame images;

[0008] Adjusting the first spatiotemporal feature according to the first inter-frame continuity feature;

[0009] The movement behavior of the target object appearing in at least part of the multiple frames of images is identified by using the adjusted first spatiotemporal feature.

[0010] In a second aspect, in one embodiment of the present application, another image processing method is provided. The method comprises:

[0011] Acquire multiple frames of images;

[0012] Determining inter-frame continuity features of adjacent frame images in the plurality of frame images, wherein the inter-frame continuity features reflect the degree of coherence of adjacent frame images in terms of motion coherence of the target object;

[0013] Determining the influence of the image features of each frame of the multiple frames in the motion behavior recognition process according to the inter-frame continuity features;

[0014] Based on the image features of the multiple frames and the influence degree corresponding to the image features of each frame in the multiple frames, the movement behavior of the target object appearing in at least part of the frames of the multiple frames is identified.

[0015] In a third aspect, in one embodiment of the present application, a recognition model is provided. The model includes:

[0016] At least one first calculation layer, used to determine the first spatiotemporal features of the multiple frames of images; the first spatiotemporal features include a first time feature and a first space feature;

[0017] at least one second calculation layer, used to determine, based on the first spatial feature, a first inter-frame continuity feature of adjacent frame images in the multiple frame images, wherein the first inter-frame continuity feature reflects the degree of coherence of adjacent frame images in terms of motion coherence of the target object;

[0018] A first adjustment layer, configured to adjust the first spatiotemporal feature according to the first inter-frame continuity feature output by the at least one second calculation layer;

[0019] A downstream combination layer, which includes a classification layer, for processing the adjusted first spatiotemporal features and outputting a motion behavior recognition result of the target object through the classification layer;

[0020] The target object appears in at least some of the frame images of the multiple frame images.

[0021] In a fourth aspect, in one embodiment of the present application, an image processing method is provided. The method comprises:

[0022] Obtain surveillance video of the robot in the venue;

[0023] Determine the spatiotemporal features of the multiple frames of the surveillance video and the inter-frame continuity features of adjacent frames of the multiple frames; wherein the inter-frame continuity features reflect the degree of continuity of the robot's movements in adjacent frames;

[0024] According to the inter-frame continuity feature, adjusting the spatiotemporal feature;

[0025] The adjusted spatiotemporal features are used to identify the actions of the robot in the surveillance video.

[0026] In a fifth aspect, in one embodiment of the present application, an image processing method is provided. The method comprises:

[0027] Acquire a video of a subject performing an operation in a processing place;

[0028] Determine the spatiotemporal features of the multiple frames of images in the video and the inter-frame continuity features of adjacent frame images in the multiple frames of images; wherein the inter-frame continuity features reflect the degree of continuity of the operation actions of the subject in adjacent frame images;

[0029] According to the inter-frame continuity feature, adjusting the spatiotemporal feature;

[0030] The adjusted spatiotemporal features are used to identify the operating actions of the subject appearing in at least part of the frame images in the video.

[0031] In a sixth aspect, in one embodiment of the present application, an image processing method is provided. The method comprises:

[0032] Obtaining videos of the objects being inspected in the security inspection area;

[0033] Determine the spatiotemporal features of the multiple frames of images in the video and the inter-frame continuity features of adjacent frames of images in the multiple frames; wherein the inter-frame continuity features reflect the degree of continuity of the behavior of the object to be detected in adjacent frames of images;

[0034] According to the inter-frame continuity feature, adjusting the spatiotemporal feature;

[0035] The adjusted spatiotemporal features are used to identify the behavior of the security detection object appearing in at least part of the frame images in the video.

[0036] In a seventh aspect, in one embodiment of the present application, an electronic device is provided. The electronic device includes: a memory and a processor; wherein:

[0037] The memory is used to store programs;

[0038] The processor is coupled to the memory and is configured to execute the program stored in the memory to:

[0039] Acquire multiple frames of images;

[0040] Determining a first spatiotemporal feature of the multiple frames of images and a first inter-frame continuity feature of adjacent frame images in the multiple frames of images; wherein the first inter-frame continuity feature reflects the degree of coherence of motion coherence of the target object in adjacent frame images;

[0041] Adjusting the first spatiotemporal feature according to the first inter-frame continuity feature;

[0042] The movement behavior of the target object appearing in at least part of the multiple frames of images is identified by using the adjusted first spatiotemporal feature.

[0043] In an eighth aspect, in another embodiment of the present application, an electronic device is provided. The electronic device includes: a memory and a processor; wherein:

[0044] The memory is used to store programs;

[0045] The processor is coupled to the memory and is configured to execute the program stored in the memory to:

[0046] Acquire multiple frames of images;

[0047] Determining inter-frame continuity features of adjacent frame images in the plurality of frame images, wherein the inter-frame continuity features reflect the degree of coherence of adjacent frame images in terms of motion coherence of the target object;

[0048] Determining the influence of the image features of each frame of the multiple frames in the motion behavior recognition process according to the inter-frame continuity features;

[0049] Based on the image features of the multiple frames and the influence degree corresponding to the image features of each frame in the multiple frames, the movement behavior of the target object appearing in at least part of the frames of the multiple frames is identified.

[0050] The technical solution provided by the embodiment of the present application obtains multiple frames of images from a video, and these multiple frames of images are obtained continuously. When processing multiple frames of images, generally two frames are used as a group to extract the first spatiotemporal feature, and the first inter-frame continuity feature between two adjacent frames is determined. Then, the first spatiotemporal feature is adjusted using the obtained first inter-frame continuity feature, and the motion behavior of the target object is identified using the adjusted first spatiotemporal feature. By adopting the technical solution of the present application, the first spatiotemporal feature of the multiple frames of images can be adjusted using the first inter-frame continuity feature of adjacent frames in the multiple frames of images. When performing feature processing on the multiple frames of images, the adverse effect of discontinuous images on the feature processing results can be effectively avoided, that is, the influence of the inter-frame continuity poor frame images can be weakened, and the correlation between the frame images with strong inter-frame continuity can be strengthened; the spatiotemporal features adjusted using the inter-frame continuity features of adjacent frame images can be used for both behavior recognition in traditional videos and behavior recognition in videos with rapid camera movement or transitions.

[0051] In addition, the method provided in the embodiments of the present application can also be applied to a variety of application scenarios, such as motion recognition of robots (sorting robots, packaging robots, transfer robots, etc. in the logistics industry), behavior recognition of processing robots, robotic arms or employees in processing plants, and behavior recognition of passengers in security inspection sites (such as airport security, railway station security, subway security), etc., by identifying the movement of robots, the operating actions of processing execution subjects (such as processing robots, employees, etc.) during processing, and the behavior of inspected personnel; the identified results can be used to lock in illegal movements (such as sorting action errors, transfer route errors), illegal operations (such as dangerous operations in processing plants), abnormal behaviors of inspected objects, etc., so as to facilitate taking corresponding measures as soon as possible to reduce losses and improve safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, a brief introduction will be given below to the drawings required for use in the embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0053] Figure 1 A flowchart of an image processing method provided in one embodiment of the present application;

[0054] Figure 2 A schematic diagram of a method flow for identifying motion behaviors based on spatiotemporal features provided in an embodiment of the present application;

[0055] Figure 3 A schematic flow chart of a method for determining inter-frame continuity features provided in an embodiment of the present application;

[0056] Figure 4 A schematic flow chart of another method for determining inter-frame continuity features provided in an embodiment of the present application;

[0057] Figure 5 A schematic flow chart of another method for determining inter-frame continuity features provided in an embodiment of the present application;

[0058] Figure 6 A flowchart of another image processing method provided in an embodiment of the present application;

[0059] Figure 7 A schematic diagram of the structure of the recognition model provided in the embodiment of the present application;

[0060] Figure 8 A schematic diagram for illustrating the behavior recognition process provided in the embodiment of the present application;

[0061] Figure 9a A schematic diagram of the structure of an image processing device provided in an embodiment of the present application;

[0062] Figure 9b A schematic diagram of the structure of another image processing device provided in an embodiment of the present application;

[0063] Fig.10 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;

[0064] Fig.11 A schematic diagram of the structure of another electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0065] In order to meet the different video application needs of users, the acquisition and transmission technologies of video images have developed rapidly. In some video application scenarios, it is necessary to identify the motion behavior of a target object in the video image. In the prior art, when performing motion behavior recognition, the requirements for video images are relatively high. The existing spatiotemporal features are mainly obtained by learning from adjacent frames through optical flow features or one-dimensional or three-dimensional convolution. There are three basic assumptions about the existing spatiotemporal features:

[0066] 1. The brightness between adjacent frames is constant.

[0067] 2. The inter-frame continuity between adjacent frames is good, that is, the movement of objects between adjacent frames is relatively small and there will be no large jumps.

[0068] 3. Maintain spatial consistency, that is, the pixels of the same object in consecutive frame images have the same movement.

[0069] However, with the growth of handheld cameras and the advancement of camera equipment, there are more and more cases of fast camera movement in videos. With the development of short videos and the advancement of video processing software and equipment, there are more and more cases of transitions in videos. At this time, the inter-frame continuity between adjacent sampled frames (i.e., the second point above) has been destroyed, and the above three assumptions can no longer be fully met. The existing solutions are mainly designed for the premise that there is good inter-frame continuity between adjacent sampled frames.

[0070] To this end, the present application intends to provide a method that can be used for object behavior recognition in the above three hypothetical videos and is suitable for object behavior recognition in videos where the camera moves or transitions quickly. In order to enable those skilled in the art to better understand the present application, the technical solution in the embodiments of the present application will be described clearly and completely below in conjunction with the accompanying drawings in the embodiments of the present application.

[0071] In some processes described in the specification, claims and the above-mentioned figures of the present application, multiple operations appearing in a specific order are included, and these operations may not be executed or executed in parallel in the order in which they appear in this article. The serial numbers of the operations, such as 101, 102, etc., are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes may include more or fewer operations, and these operations may be executed in sequence or in parallel. It should be noted that the descriptions of "first", "second", etc. in this article are used to distinguish different messages, devices, modules, etc., do not represent the order of precedence, and do not limit "first" and "second" to different types. In addition, the embodiments described below are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0072] like Figure 1 A flowchart of an image processing method provided for an embodiment of the present application. The execution subject of the method may be a client device or a server device. The client device may include but is not limited to: a tablet computer, a mobile phone, a desktop computer, a laptop computer, etc. The server device includes but is not limited to: a local server, a cloud server, a server cluster, a virtual server, etc. Specifically, the method includes the following steps:

[0073] 101: Acquire multiple frames of images.

[0074] 102: Determine a first spatiotemporal feature of the multiple-frame images and a first inter-frame continuity feature of adjacent frame images in the multiple-frame images; wherein the first inter-frame continuity feature reflects the degree of coherence of motion coherence of a target object in adjacent frame images.

[0075] 103: Adjust the first spatiotemporal feature according to the first inter-frame continuity feature.

[0076] 104: Using the adjusted first spatiotemporal feature, identify the motion behavior of the target object appearing in at least part of the multiple frames of images.

[0077] It should be noted that the image mentioned here can be an image acquired by a video acquisition device or a depth image acquisition device. In addition, the method of acquiring multiple frames of images from a video image can be to continuously acquire multiple frames of images intercepted from the video, or to acquire multiple frames of images according to a preset sampling rule (for example, interval sampling, or continuous sampling for a key time period).

[0078] In the above 102, the first spatiotemporal feature is to distinguish the spatiotemporal features that appear below, and "first" is added. The spatiotemporal features mentioned in this article may include temporal features and spatial features. Among them, the temporal feature can be understood as the relevant features of temporal continuity extracted based on multiple frames of images. The spatial feature can be understood as the relevant features of spatial continuity extracted based on multiple frames of images. Spatial features are used to describe the appearance of moving objects in multiple frames of images (which can be understood as moving objects, that is, foreground features), as well as scene configuration features in each frame (which can be understood as background features). Temporal features are used to capture motion information embedded in frames that change over time.

[0079] In practical applications, the first inter-frame continuity can be understood as the degree of coherence that characterizes the motion coherence of the target object contained in adjacent frames or sampled frames. Take a simple example, in the broadcast video of a gymnastics event, the camera is initially on the athlete, then quickly switches to the audience seats, and then switches from the audience seats to the athlete. Assume that there are continuous movements of the athlete in the first ten frames, the eleventh frame to the thirteenth frame are images of the audience seats, and from the fourteenth frame onwards there are motion images of the athlete in the image. Then, the motion coherence of the target object in any two adjacent frames in the first ten frames is good; but the motion coherence of the target object in the tenth frame and the eleventh frame is poor; similarly, the motion coherence of the target object in the thirteenth frame and the fourteenth frame is poor. In the process of image processing, this first inter-frame continuity feature (i.e., the degree of coherence) can be represented by probability or score, and the first spatiotemporal feature is adjusted using probability or score. The specific adjustment process will be explained in the following embodiments.

[0080] The first inter-frame continuity feature mentioned here can be obtained by calculating and processing based on the earlier calculation layer in the machine learning model. In the machine learning model, the earlier calculation layer can perform calculation processing to detect frame images with poor inter-frame continuity as early as possible, and promptly eliminate the adverse effects of frame images with poor continuity on subsequent calculation layers and calculation results in the robot learning model. The first inter-frame continuity feature includes inter-frame continuity feature items of any two adjacent frames in the multi-frame image. Assume that there are 8 frames in the multi-frame image, numbered 1, 2, 3, 4, 5, 6, 7, and 8 respectively. The first inter-frame continuity feature includes: the inter-frame continuity feature item C between the first frame image and the second frame image 12 , the inter-frame continuity feature item C between the second frame image and the third frame image 23 , the inter-frame continuity feature item C between the 3rd frame image and the 4th frame image 34 , the inter-frame continuity feature item C between the 4th frame image and the 5th frame image 45 , the inter-frame continuity feature item C between the 5th frame image and the 6th frame image 56, the inter-frame continuity feature item C between the 6th frame image and the 7th frame image 67 , the inter-frame continuity feature item C between the 7th frame image and the 8th frame image 78 For example, the first inter-frame continuity feature of these 8 frames of images can be represented by the following vector:

[0081] (C 12 , C 23 , C 34 , C 45 , C 56 , C 67 , C 78 )

[0082] The above-mentioned continuity feature items between each frame can be represented by probability or score, and the probability or score is used to adjust the first spatiotemporal feature. Taking the above-mentioned 8 frames of images as an example, the first spatiotemporal feature of the 8 frames of images includes: the spatiotemporal feature item T between the first frame of image and the second frame of image 12 , the spatiotemporal feature term T between the second frame image and the third frame image 23 , the spatiotemporal feature term T between the 3rd and 4th frames 34 , the spatiotemporal feature term T between the 4th frame image and the 5th frame image 45 , the spatiotemporal feature term T between the 5th frame image and the 6th frame image 56 , the spatiotemporal feature term T between the 6th frame image and the 7th frame image 67 , the spatiotemporal feature term T between the 7th frame image and the 8th frame image 78 .

[0083] For example, the first spatiotemporal features of these 8 frames of images can be represented as the following vector:

[0084] (T 12 、T 23 、T 34 、T 45 、T 56 、T 67 、T 78 )

[0085] The better the first inter-frame continuity feature item is (or the higher the probability value or score is), the better the coherence of the adjacent frame images is (it can be understood as the target object and the target object's motion behavior can be accurately identified from the two adjacent frame images, and the target object and its motion behavior are coherent). Conversely, if the first inter-frame continuity feature item is not good (or the lower the probability value or score is), it means that the coherence of the adjacent frame images is not good, which will affect the accuracy of motion behavior recognition.

[0086] In the above 103, the process of adjusting the first spatiotemporal feature according to the first inter-frame continuity feature can be simply understood as the following process:

[0087] Taking the above 8 frames of images as an example again,

[0088] Using C 12 The first spatiotemporal feature T between the first frame image and the second frame image 12 Make adjustments;

[0089] Using C 23 The first spatiotemporal feature T between the second and third frames 23 Make adjustments;

[0090] Using C 34 The first spatiotemporal feature T between the third and fourth frames 34 Make adjustments;

[0091] Using C 45 The first spatiotemporal feature T between the 4th and 5th frames 45 Make adjustments;

[0092] Using C 56 The first spatiotemporal feature T between the 5th and 6th frames 56 Make adjustments;

[0093] Using C 67 The first spatiotemporal feature T between the 6th and 7th frames 67 Make adjustments;

[0094] Using C 78 The first spatiotemporal feature T between the 7th and 8th frames 78 Make adjustments.

[0095] More specifically, the adjustment process can be to change C 12 As the first spatiotemporal feature T 12 The coefficient of , and the product result can be calculated to obtain the adjusted first spatiotemporal feature T 12 . Other T 23 ,……T 78 The adjustment is the same as T 12 .

[0096] Based on the above scheme, it is possible to more accurately identify the motion behavior of the target object contained in multiple frames of images, and avoid the influence of interference factors in the video frames. In other words, the first inter-frame continuity feature can be used to screen multiple frames of images in the video, and the influence of discontinuous frames on the behavior recognition results can be timely shielded, which can effectively improve the accuracy of behavior recognition.

[0097] In step 102, “determining a first spatiotemporal feature of the multiple frames of images and a first inter-frame continuity feature of adjacent frame images in the multiple frames of images” may specifically include:

[0098] 1021. Use the multiple frames of images as input of at least one first computing layer in a first recognition model, execute the at least one first computing layer, and obtain first spatiotemporal features of the multiple frames of images; wherein the first spatiotemporal features include first time features and first spatial features;

[0099] 1022. Use the first spatiotemporal features of the multiple-frame images (specifically, only the first spatial features may be used as input parameters) as input of at least one second computing layer of the first recognition model, and execute the at least one second computing layer to output the first inter-frame continuity features of adjacent frame images in the multiple-frame images.

[0100] In practical applications, when extracting features from multiple frames of images, a first recognition model obtained through pre-training can be used. Specifically, the first recognition model includes at least one first calculation layer, and the first spatiotemporal feature can be extracted based on multiple frames of images using the first calculation layer. When extracting the first inter-frame continuity feature from multiple frames of images, the extraction is performed based on the first spatiotemporal feature output by the first calculation layer. Specifically, the first spatial feature in the first spatiotemporal feature is input into at least one second calculation layer for calculation to obtain the first inter-frame continuity feature. Generally speaking, when calculating the inter-frame continuity feature, the continuity between two adjacent frames of images is calculated, and the obtained first inter-frame continuity feature is used as an adjustment to the latter frame of the two consecutive frames of images.

[0101] For example, it is assumed that in the current first recognition model, the first computing layer is the first convolutional layer (Convolutional Neural Network, CNN), and the second computing layer is the first fully connected layer (Fully-connected layer, FC). In the first recognition model, there may be multiple first convolutional layers, and each convolutional layer may be connected to at least one first fully connected layer. The shallow features for multiple frames of images output by the first convolutional layer are the first spatiotemporal features, and the shallow features (such as color, texture, etc.) can reflect the changes and trends of spatial information. Then, the first spatial features in the first spatiotemporal features are input into the first fully connected layer, and the first inter-frame continuity features are obtained after being processed by at least one first fully connected layer. In practical applications, the number of convolutional layers included in each first recognition model can be set as needed, and the number of fully connected layers to which each convolutional layer is respectively connected can also be set according to actual needs. In addition, the first recognition model may also include pooling layers, fully connected layers, classification layers, and the like.

[0102] Furthermore, the first recognition module may further include an adjustment layer, the input of the adjustment layer is the first spatiotemporal feature and the first inter-frame continuity feature; executing the adjustment layer may output the adjusted first spatiotemporal feature. Figure 8 In the example shown, after the first calculation layer (such as Con* layer) in the first recognition module, there is an adjustment layer. The adjustment layer includes two branches, namely a space branch and a temporal branch. Figure 8 As shown, the input of the spatial branch is the first spatial feature and the first inter-frame continuity feature, and the output is the adjusted first spatial feature. The input of the temporal branch is the first temporal feature and the first inter-frame continuity feature, and the output is the adjusted first temporal feature.

[0103] like Figure 2 A schematic flow chart of a method for identifying motion behavior based on spatiotemporal features provided in an embodiment of the present application. As described in step 102, further, from Figure 2 It can be seen that the motion behavior of the target object in the multiple frames of images is identified by using the adjusted first spatiotemporal feature, which specifically includes the following steps:

[0104] 201: Using the adjusted first spatiotemporal features as input of at least one third computing layer of the first recognition model, executing the at least one third computing layer to output second spatiotemporal features of the multiple-frame images; wherein the second spatiotemporal features include second time features and second spatial features of the multiple-frame images.

[0105] 202: Use the second spatial feature and the first inter-frame continuity feature of adjacent frame images in the multiple frame images output by the at least one second computing layer as input to the at least one fourth computing layer of the first recognition model, and execute the second inter-frame continuity feature of adjacent frame images in the multiple frame images output by the at least one fourth computing layer.

[0106] 203: Adjust the second spatiotemporal feature according to the second inter-frame continuity feature of adjacent frame images in the multiple frame images output by the at least one fourth computing layer.

[0107] 204: Using the adjusted second spatiotemporal features, identify the movement behavior of the target object in the multiple frames of images.

[0108] As mentioned above, the first recognition model may also include multiple spatiotemporal feature calculation layers (i.e., the third calculation layer, for example, a convolutional layer), and a calculation layer based on the continuity feature processing of the spatial features output by the spatiotemporal feature calculation layer (i.e., the fourth calculation layer, for example, a fully connected layer). The first spatiotemporal feature obtained by the first calculation layer is not only input to the second calculation layer, but also the adjusted first spatiotemporal feature is input to the third calculation layer. Specifically, after obtaining the first inter-frame continuity feature as described in the above embodiment, the first inter-frame continuity feature is used to adjust the first spatiotemporal feature, and the second inter-frame continuity feature to be obtained also needs to be adjusted, that is, the first inter-frame continuity feature and the second spatiotemporal feature are simultaneously input to the fourth calculation layer so that the second inter-frame continuity feature can be obtained. The method of adjusting the spatiotemporal feature mentioned here, for example, can be to multiply the feature value (e.g., probability value) corresponding to the first inter-frame continuity feature with the first spatiotemporal feature. The method of adjusting the inter-frame continuity feature mentioned here may be, for example, multiplying the first inter-frame continuity feature and the second spatial feature and then extracting the feature using the fourth calculation layer to obtain the second inter-frame continuity feature.

[0109] It is easy to understand that if there are multiple convolutional layers in the recognition model, we can use Figure 1 and Figure 2 The steps are iteratively processed, that is, the inter-frame continuity features obtained by the previous calculation layer are used to adjust the spatiotemporal features to be input to the next calculation layer and the corresponding inter-frame continuity feature calculation layer. By adopting multiple rounds of iterative processing, the adverse interference of multiple frames of images with poor inter-frame continuity on the behavior recognition process and results can be effectively eliminated.

[0110] In practical applications, the movement behavior of the target object in the multiple frames of images is identified using the adjusted second spatiotemporal features, including: using the adjusted second spatiotemporal features as the input of the classification layer of the first recognition model, executing the classification layer to output the recognition results of the target object in the multiple frames of images. Of course, after the third and fourth calculation layers and before the classification layer, there can also be a fifth calculation layer for outputting the third spatiotemporal features, and a sixth calculation layer for outputting the third inter-frame continuity features, and the third inter-frame continuity features can also adjust the third spatiotemporal features; ... and so on.

[0111] Specifically, after the spatiotemporal features are extracted by multiple computing layers and adjusted by using the continuity between frames, the adjusted spatiotemporal features are input into the classification layer, and the classification layer is used to classify the multiple frames of images, and then the behavior of the target object in the image is recognized. The classification layer mentioned here can use algorithms such as support vector machine (SVM), decision tree, Bayesian, etc.

[0112] The above is to determine the inter-frame continuity of adjacent frame images in multiple frame images by using the corresponding layers in the first recognition model (such as the second calculation layer, the fully connected layer). Figure 3 FIG. 2 is a flow chart of another method for determining inter-frame continuity features provided by an embodiment of the present application. Figure 3 As shown, the multiple frames of images include adjacent first frame images and second frame images; and

[0113] Determining a first inter-frame continuity feature of adjacent frame images in the multiple frame images includes:

[0114] 301: Extracting image features of the first frame image and image features of the second frame image.

[0115] 302: Determine an image feature difference based on the image features of the first frame image and the image features of the second frame image.

[0116] 303: Determine a first inter-frame continuity feature between the first frame image and the second frame image according to the image feature difference.

[0117] In some application scenarios, when calculating the first inter-frame continuity feature between two adjacent frames, if there is at least one target object in the adjacent images, the inter-frame difference algorithm can be used, that is, the area where changes occur in the two adjacent frames is detected. This algorithm uses two consecutive frames in the image sequence to perform a difference, and then binarizes the grayscale difference image to extract the motion information of the target object. The image obtained by the inter-frame change area detection and segmentation distinguishes the background area and the target object area, and then extracts the target object to be detected. By comparing the difference in the grayscale values ​​of the corresponding pixels of the two frames in the image sequence, the first inter-frame continuity feature is determined by subtracting the two frames.

[0118] The first inter-frame continuity feature mentioned here may be a probability value. After obtaining the first inter-frame continuity feature, the first inter-frame continuity feature may be used to adjust the first spatiotemporal feature. Specifically, the inter-frame continuity probability value between the first frame image and the second frame image is used as the adjustment coefficient of the temporal feature information of the second frame image, and the temporal feature information of the second frame image in the first spatiotemporal feature is adjusted. The inter-frame continuity probability value between the first frame image and the second frame image is used as the adjustment coefficient of the spatial feature information of the second frame image, and the spatial feature information of the second frame image in the first spatiotemporal feature is adjusted. Using the adjusted spatiotemporal feature as the input of the subsequent calculation layer can effectively eliminate the adverse effects of discontinuous frame images on the behavior recognition results of the target object.

[0119] In addition to the frame difference algorithm, other algorithms can also be used. For example, Figure 4A flowchart of another method for determining inter-frame continuity features provided in an embodiment of the present application. Specifically, determining the first inter-frame continuity features of adjacent frame images in the multiple frame images includes:

[0120] 401: Obtain a continuity prediction model; wherein the continuity prediction model is obtained after the first machine learning model is trained.

[0121] 402: Using the multiple frames of images as inputs of the continuity prediction model, executing the continuity prediction model to output an inter-frame continuity probability value of any frame image in the multiple frames of images relative to an adjacent frame image. The first inter-frame continuity feature includes an inter-frame continuity probability value of any frame image in the multiple frames of images relative to an adjacent frame image.

[0122] In practical applications, the first machine learning model can be trained to obtain a trained continuity prediction model, and the model can be used to perform continuity prediction on input multiple frames of images. For example, a portion of multiple frames of images that need to be processed is intercepted from a group of videos, and the multiple frames of images are input into the continuity prediction model, and then the continuity probability values ​​of each frame image relative to the previous frame image are output respectively, and the continuity probability value is used as the first inter-frame continuity feature of the current frame image (second frame image) relative to the previous frame image (first frame image). Generally speaking, the larger the continuity probability value, the better the continuity with the previous frame.

[0123] For example, a classification model can also be added after the continuity prediction model. That is, after the continuity prediction model outputs the continuity probability value of the multi-frame image, the continuity probability value and the multi-frame image are input into the classification model. In the classification model, the continuity probability value plays a role in adjusting the spatiotemporal features of the multi-frame image. It should be noted that after being processed by the continuity prediction model, the continuity probability value of each frame image obtained can effectively distinguish the continuity effect of each frame image, that is, the coherence of the motion behavior of the target object in each frame image. The probability value corresponding to the image with poor coherence is relatively low, and the probability value with good coherence is relatively high. Generally speaking, the probability value ranges from 0 to 1, and the larger the probability value, the better the coherence. The process of adjusting the spatiotemporal features using the continuity probability value is as follows: the inter-frame continuity probability value of the first frame image and the second frame image is used as the adjustment coefficient of the temporal feature information of the second frame image, and the temporal feature information of the second frame image in the first spatiotemporal feature is adjusted. The inter-frame continuity probability value of the first frame image and the second frame image is used as the adjustment coefficient of the spatial feature information of the second frame image, and the spatial feature information of the second frame image in the first spatiotemporal feature is adjusted. The first spatiotemporal features and their probability values ​​(first inter-frame continuity features) corresponding to each frame image are input into the classification model together, and in the process of motion behavior recognition, the influence of the frame images with poor coherence can be eliminated, so that the motion behavior of the target object contained in each frame image can be more accurately recognized based on the frame images with good coherence.

[0124] like Figure 5 A schematic diagram of a method for determining spatiotemporal features provided in an embodiment of the present application. The spatiotemporal feature determination process specifically includes the following steps:

[0125] 501: Obtain a second recognition model; wherein the second recognition model is obtained after the second machine learning model is trained.

[0126] The second recognition model here can be the same model as the classification model mentioned above. The second machine learning model can be a deep neural network model, a convolutional neural network model, etc., which is not specifically limited in this embodiment.

[0127] 502: Using the multiple frames of images as input of the second recognition model, and obtaining the first spatiotemporal features through calculation by at least one first calculation layer of the second recognition model; wherein the second recognition model further includes a classification layer for outputting recognition results based on the spatiotemporal features.

[0128] In the embodiment of the present application, the second recognition model mainly includes two parts, one part is to output the first spatiotemporal feature based on multiple frames of images, and the other part is to realize the recognition and classification of multiple frames of images based on the first spatiotemporal feature. The second recognition model described here includes at least one first computing layer, which is used to process multiple frames of images and output the first spatiotemporal feature. Among them, the first spatiotemporal feature includes: the time feature item of each frame image in the multiple frames of images, and the spatial feature item of each frame image in the multiple frames of images. Furthermore, the first spatiotemporal feature can be used to realize the recognition result of the motion behavior of the target object in the multiple frames of images.

[0129] Furthermore, in order to improve the accuracy of the motion behavior recognition of the target object, the continuity prediction model can be combined with the second recognition model. Specifically, after the same set of multi-frame images are input into the continuity prediction model and the second recognition model respectively, the first inter-frame continuity feature and the first spatiotemporal feature (which can be the spatiotemporal feature calculated by any calculation layer in the second recognition model) are obtained; then, the first inter-frame continuity feature is used to adjust the first spatiotemporal feature, and the second recognition model continues to use the adjusted first spatiotemporal feature to output the motion behavior recognition result of the target object in the multi-frame image.

[0130] As an optional embodiment, if the second recognition model includes multiple computing layers (e.g., multiple convolutional layers), the spatiotemporal features corresponding to each computing layer can be output respectively, and the spatiotemporal features to be input to each computing layer can be adjusted using the continuity prediction model. After multiple layers of adjustment, the accuracy of identifying the motion behavior of the target object in the image can be improved.

[0131] Based on the same idea, the present application also provides an image processing method. Figure 6 A flowchart of another image processing method provided in an embodiment of the present application is shown below. Figure 6 As shown, including:

[0132] 601: Acquire multiple frames of images.

[0133] 602: Determine inter-frame continuity features of adjacent frame images in a plurality of frame images, wherein the inter-frame continuity features reflect the degree of coherence of adjacent frame images in terms of motion coherence of a target object.

[0134] 603: Determine, based on the inter-frame continuity feature, the influence degree of the image feature of each frame image in the multiple frames of images in the motion behavior recognition process.

[0135] 604: Based on the image features of the multiple frames of images and the influence degree corresponding to the image features of each frame of the multiple frames of images, identify the movement behavior of the target object appearing in at least part of the frame images of the multiple frames of images.

[0136] In practical applications, when performing motion behavior recognition, the adverse effects of incoherent images or incoherent target objects in the image on the recognition results cannot be ruled out. After acquiring multiple frames of images, the inter-frame continuity features used to characterize the degree of motion coherence of the target object in adjacent frame images can be first acquired. The acquisition method can refer to the following: Figure 3-4 Described each embodiment, just do not repeat here.And then, based on the interframe continuity feature, the influence degree of image feature in the motor behavior identification process is evaluated.Target is: the better the interframe continuity is, the higher the favorable influence degree of image feature in the motor behavior identification process is; the worse the interframe continuity is, the lower the favorable influence degree of image feature in the motor behavior identification process is.Therefore, in carrying out the motor behavior identification process, adopt suitable mode (such as, the mode of adjusting as early as possible or the mode of adjusting repeatedly etc.) to eliminate adverse effects, expand favorable effects.

[0137] There are many ways to obtain inter-frame continuity features. For ease of understanding, specific examples are given below. One way is to use an inter-frame difference method, assuming that a multi-frame image contains adjacent first frame images and second frame images. Determine the inter-frame continuity features of adjacent frame images in the multi-frame image, including: extracting image features of the first frame image and image features of the second frame image. Determine the image feature difference based on the image features of the first frame image and the image features of the second frame image. Determine the inter-frame continuity features between the first frame image and the second frame image based on the image feature difference. Determine the inter-frame continuity features between the first frame image and the second frame image based on the image feature difference.

[0138] The inter-frame continuity feature between two adjacent frames, which is used to characterize the coherence of the target object's motion coherence, can be obtained through the inter-frame difference method. Figure 3 The corresponding embodiments will not be repeated here.

[0139] Another way to obtain inter-frame continuity features can be achieved using a machine learning model. For example, it includes: obtaining a continuity prediction model; wherein the continuity prediction model is obtained after training a first machine learning model. The multiple frames of images are used as input to the continuity prediction model, and the continuity prediction model is executed to output the inter-frame continuity probability value of any frame image in the multiple frames relative to the adjacent frame image. The inter-frame continuity feature includes the inter-frame continuity probability value of any frame image in the multiple frames relative to the adjacent frame image. For details, please refer to the above Figure 4 The corresponding embodiments.

[0140] In practical applications, when evaluating the degree of influence, the inter-frame continuity probability value can be used as the degree of influence of the subsequent frame image in adjacent frame images relative to the previous frame image, and the inter-frame continuity probability value can be used to adjust the spatiotemporal characteristics of multiple frame images. It should be noted that the image features of multiple frame images include spatiotemporal features. When identifying the motion behavior of the target object, the spatiotemporal features of the multiple frame images are adjusted using the degree of influence corresponding to the image features of each frame image in the multiple frame images. Using the adjusted spatiotemporal features, the motion behavior of the target object appearing in at least some of the frame images of the multiple frame images is identified. For the specific adjustment process, please refer to Figure 1-5 The corresponding embodiments will not be repeated here. In this embodiment, when the behavior characteristics of the target object are identified, the spatiotemporal characteristics of each frame image are adjusted based on the degree of influence, and the adverse effects of poorly coherent images on the recognition of motion behaviors are eliminated by using the degree of influence, while the favorable effects of images with good coherence on the recognition of motion behaviors are enhanced.

[0141] Based on the same idea, the present application embodiment also provides a recognition model. Figure 7 A schematic diagram of the structure of the recognition model provided in the embodiment of the present application. Figure 7 As can be seen, the recognition model includes:

[0142] At least one first calculation layer 71, used to determine the first spatiotemporal features of the multiple frames of images; the first spatiotemporal features include first time features and first spatial features;

[0143] at least one second calculation layer 72, used to determine a first inter-frame continuity feature of adjacent frame images in the multiple frame images according to the first spatial feature, wherein the first inter-frame continuity feature reflects the degree of coherence of adjacent frame images in terms of motion coherence of the target object;

[0144] A first adjustment layer 73, configured to adjust the first spatiotemporal feature according to the first inter-frame continuity feature output by the at least one second calculation layer;

[0145] A downstream combination layer 74, which includes a classification layer 74a, for processing the adjusted first spatiotemporal features and outputting the target object's motion behavior recognition result through the classification layer;

[0146] The target object appears in at least some of the frame images of the multiple frame images.

[0147] In practical applications, the first computing layer is used to process multiple frames of images to obtain the first spatiotemporal features. The first computing layer mentioned here may be a convolutional layer. The first spatiotemporal features output by the first computing layer are input to the second computing layer. The second computing layer mentioned here may be a fully connected layer. The second computing layer can output the first inter-frame continuity features. Figure 7 As can be seen in FIG. 1 , in the adjustment layer, the first inter-frame continuity feature is used to adjust the first temporal feature and the first spatial feature in the first spatiotemporal feature. Figure 7 Only a first computing layer and a second computing layer are schematically shown. In actual application, the user can design the number of the first computing layer and the second computing layer according to actual needs.

[0148] The downstream combination layer mentioned here includes a classification layer. In the downstream combination layer, the adjusted first spatiotemporal feature is used as input, the motion behavior of the target object contained in the adjacent frame images is recognized by the classification layer, and the recognition result is output.

[0149] In this recognition model, at least one calculation layer (which may be a fully connected layer) is added horizontally for outputting inter-frame continuity features. Inter-frame continuity is used to adjust the temporal and spatial features of the input of subsequent layers, thereby eliminating the influence of images with poor continuity and enhancing the influence of images with good continuity. It should be noted that, in this embodiment, the first calculation layer can be the first calculation layer in the recognition model, which can adjust the temporal and spatial features in advance when multiple frames of images are input, so as to avoid the adverse effects of images with poor continuity on subsequent motion behavior recognition results.

[0150] As an optional scheme, the downstream combination layer 74 may also include: at least one third calculation layer 74b, used to output the second spatiotemporal features of the multiple frame images based on the adjusted first spatiotemporal features, wherein the second spatiotemporal features include the second time features and the second spatial features of the multiple frame images; at least one fourth calculation layer 74c, used to output the second inter-frame continuity features of adjacent frame images in the multiple frame images based on the second spatial features and the first inter-frame continuity features of adjacent frame images in the multiple frame images output by the at least one second calculation layer.

[0151] In practical applications, in order to more thoroughly eliminate the adverse effects of poor continuity images on recognition results, corresponding inter-frame continuity feature calculation layers and adjustment layers can be set in multiple calculation layers. It is easy to understand that here, the third and fourth calculation layers are included in the downstream combination layer as an example, and in the actual application process, many layers may be included, and the calculation layer structure relationship introduced in the above embodiment can also be adopted.

[0152] It should be noted that, in the recognition model, the location and number of the calculation layers used to output the inter-frame continuity features, as well as the location and number of the adjustment layers can be selected and modulated as needed.

[0153] In order to facilitate the understanding of the technical solution of this application, Figure 8 An example is given to illustrate the overall plan. Figure 8 The following is a schematic diagram of an example behavior recognition process provided in the embodiment of the present application. Figure 8 As can be seen in , multiple consecutive frames of images are input into the model. Figure 8 There are two convolutional layers conv* in the figure. In practical applications, the convolution kernel size and stride of different convolutional layers may be different. The calculation methods used include VGG, resnet, inception, dense net, etc. The convolutional layer can output the spatial feature branch space branch and the temporal feature branch temporal branch. In addition, from Figure 8 It can also be seen that the lateral connection relationship of the convolutional layer also includes a fully connected layer fc, which is processed by the fully connected layer to output the inter-frame continuity features. While each convolutional layer outputs the spatiotemporal features and transmits them downward, it also outputs the corresponding inter-frame continuity features horizontally. Furthermore, the inter-frame continuity features are used to adjust the spatiotemporal features to be input to the next layer, and the adjusted spatiotemporal features are used to perform convolution processing and fully connected layer processing to obtain new inter-frame continuity features. The above method is used to perform multiple iterative adjustments and finally output the behavior recognition results. In the subsequent processing of the adjusted spatiotemporal features, there may be pooling layers, fully connected layers, classification layers, and so on.

[0154] Based on the same idea, the embodiment of the present application also provides an image processing device. Figure 9a A schematic diagram of the structure of an image processing device provided in an embodiment of the present application. The image processing device includes:

[0155] The acquisition module 91 is used to acquire multiple frames of images.

[0156] The determination module 92 is used to determine the first spatiotemporal features of the multiple frames and the first inter-frame continuity features of adjacent frame images in the multiple frames; wherein the first inter-frame continuity features reflect the degree of coherence of the motion coherence of the target object in adjacent frame images.

[0157] The adjustment module 93 is used to adjust the first spatiotemporal feature according to the first inter-frame continuity feature.

[0158] The recognition module 94 is used to recognize the motion behavior of the target object appearing in at least part of the multiple frames of images by using the adjusted first spatiotemporal feature.

[0159] Optionally, the determination module 92 is further configured to use the multiple frames of images as input of at least one first calculation layer in the first recognition model, execute the at least one first calculation layer, and obtain first spatiotemporal features of the multiple frames of images; wherein the first spatiotemporal features include first time features and first spatial features;

[0160] The first spatial features of the multiple frame images are used as input of at least one second calculation layer of the first recognition model, and the at least one second calculation layer is executed to output the first inter-frame continuity features of adjacent frame images in the multiple frame images.

[0161] Optionally, the recognition module 94 is further configured to use the adjusted first spatiotemporal feature as an input of at least one third computing layer of the first recognition model, and execute the at least one third computing layer to output a second spatiotemporal feature of the multiple-frame image; wherein the second spatiotemporal feature includes a second time feature and a second spatial feature of the multiple-frame image;

[0162] Using the second spatial feature and the first inter-frame continuity feature of adjacent frame images in the multiple frame images output by the at least one second computing layer as inputs of at least one fourth computing layer of the first recognition model, and executing the at least one fourth computing layer to output the second inter-frame continuity feature of adjacent frame images in the multiple frame images;

[0163] Adjusting the second spatiotemporal feature according to a second inter-frame continuity feature of adjacent frame images in the multiple frame images output by the at least one fourth computing layer;

[0164] The movement behavior of the target object in the multiple frames of images is identified by using the adjusted second spatiotemporal features.

[0165] Optionally, the recognition module 94 is further configured to use the adjusted second spatiotemporal features as input of a classification layer of the first recognition model, and execute the classification layer to output recognition results of target objects in the multiple image frames.

[0166] Optionally, the multiple frames of images include adjacent first frame images and second frame images; and

[0167] The determination module 92 is used to extract the image features of the first frame image and the image features of the second frame image; determine the image feature difference based on the image features of the first frame image and the image features of the second frame image; and determine the first inter-frame continuity feature between the first frame image and the second frame image based on the image feature difference.

[0168] Optionally, the determination module 92 is used to obtain a continuity prediction model; wherein the continuity prediction model is obtained after the first machine learning model is trained;

[0169] Taking the multiple frames of images as input of the continuity prediction model, executing the continuity prediction model to output an inter-frame continuity probability value of any frame image in the multiple frames of images relative to an adjacent frame image;

[0170] The first inter-frame continuity feature includes an inter-frame continuity probability value of any frame image in the multiple frame images relative to an adjacent frame image.

[0171] Optionally, the determination module 92 is used to obtain a second recognition model; wherein the second recognition model is obtained after the second machine learning model is trained;

[0172] Using the multiple frames of images as input of the second recognition model, and calculating the first spatiotemporal features through at least one first calculation layer of the second recognition model;

[0173] The second recognition model further includes a classification layer for outputting a recognition result based on the first spatiotemporal feature.

[0174] Optionally, the inter-frame continuity feature includes inter-frame continuity probability values ​​of the first frame image and the second frame image; the first spatiotemporal feature includes: time feature information of the first frame image, space feature information of the first frame image, time feature information of the second frame image and space feature information of the second frame image;

[0175] An adjustment module 93, configured to use the inter-frame continuity probability value between the first frame image and the second frame image as an adjustment coefficient of the time feature information of the second frame image, and adjust the time feature information of the second frame image in the first spatiotemporal feature;

[0176] The inter-frame continuity probability value between the first frame image and the second frame image is used as an adjustment coefficient of the spatial feature information of the second frame image to adjust the spatial feature information of the second frame image in the first spatiotemporal feature.

[0177] Based on the same idea, the present application also provides another image processing device. Figure 9b A schematic diagram of the structure of another image processing device provided in an embodiment of the present application. The image processing device includes:

[0178] The acquisition module 95 is used to acquire multiple frames of images.

[0179] The determination module 96 is used to determine the inter-frame continuity feature of adjacent frame images in the multiple frame images, wherein the inter-frame continuity feature reflects the degree of coherence of the adjacent frame images in terms of the motion coherence of the target object.

[0180] The determination module 96 is further configured to determine the influence of the image features of each frame of the multiple frames of images in the process of motion behavior recognition according to the inter-frame continuity features.

[0181] The identification module 97 is used to identify the movement behavior of the target object appearing in at least part of the frames of the multiple images based on the image features of the multiple images and the influence degree corresponding to the image features of each frame of the multiple images.

[0182] The multiple frame images include adjacent first frame images and second frame images; and the determination module 96 is also used to extract image features of the first frame image and image features of the second frame image; based on the image features of the first frame image and the image features of the second frame image, determine the image feature difference; based on the image feature difference, determine the inter-frame continuity features between the first frame image and the second frame image.

[0183] Optionally, the determination module 96 is further used to obtain a continuity prediction model; wherein the continuity prediction model is obtained after the first machine learning model is trained;

[0184] Taking the multiple frames of images as input of the continuity prediction model, executing the continuity prediction model to output an inter-frame continuity probability value of any frame image in the multiple frames of images relative to an adjacent frame image;

[0185] The inter-frame continuity feature includes an inter-frame continuity probability value of any frame image in the multiple frame images relative to an adjacent frame image.

[0186] Optionally, the inter-frame continuity feature contains an inter-frame continuity probability value of a second frame image relative to a first frame image in the multiple frames of images; and

[0187] Optionally, the determination module 96 is further configured to use the inter-frame continuity probability value of the second frame image relative to the first frame image as the influence degree corresponding to the second frame image.

[0188] Optionally, the image features of the multiple frames of images include spatiotemporal features, and

[0189] The recognition module 97 is also used to adjust the spatiotemporal characteristics of the multiple frames of images according to the influence degree corresponding to the image features of each frame image in the multiple frames of images; and use the adjusted spatiotemporal characteristics to recognize the movement behavior of the target object appearing in at least some of the frame images of the multiple frames of images.

[0190] An embodiment of the present application also provides an electronic device. Fig.10The electronic device includes a memory 1001, a processor 1002 and a communication component 1003;

[0191] The memory 1001 is used to store programs;

[0192] The processor 1002 is coupled to the memory and is used to execute the program stored in the memory to: acquire multiple frames of images;

[0193] Determining a first spatiotemporal feature of the multiple frames of images and a first inter-frame continuity feature of adjacent frame images in the multiple frames of images; wherein the first inter-frame continuity feature reflects the degree of coherence of motion coherence of the target object in adjacent frame images;

[0194] Adjusting the first spatiotemporal feature according to the first inter-frame continuity feature;

[0195] The movement behavior of the target object appearing in at least part of the multiple frames of images is identified by using the adjusted first spatiotemporal feature.

[0196] The memory 1001 can be configured to store various other data to support operations on the electronic device. Examples of these data include instructions for any application or method for operating on the electronic device. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0197] Furthermore, the processor 1002 in this embodiment may specifically be: a programmable switching processing chip, in which a data replication engine is configured to replicate received data.

[0198] When the processor 1002 executes the program in the memory, in addition to the above functions, it can also realize other functions, which can be specifically referred to the description of the above embodiments. Fig.10 As shown, the electronic device also includes: a power supply component 1004 and other components.

[0199] Based on the same idea, an embodiment of the present application also provides another electronic device. Fig.11 This is a schematic diagram of the structure of another electronic device provided in an embodiment of the present application. The electronic device includes a memory 1101, a processor 1102 and a communication component 1103; wherein:

[0200] The memory 1101 is used to store programs;

[0201] The processor 1102 is coupled to the memory and is used to execute the program stored in the memory to: acquire multiple frames of images;

[0202] Determining inter-frame continuity features of adjacent frame images in the plurality of frame images, wherein the inter-frame continuity features reflect the degree of coherence of adjacent frame images in terms of motion coherence of the target object;

[0203] Determining the influence of the image features of each frame of the multiple frames in the motion behavior recognition process according to the inter-frame continuity features;

[0204] Based on the image features of the multiple frames and the influence degree corresponding to the image features of each frame in the multiple frames, the movement behavior of the target object appearing in at least part of the frames of the multiple frames is identified.

[0205] The memory 1101 can be configured to store various other data to support operations on the electronic device. Examples of these data include instructions for any application or method for operating on the electronic device. The memory can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0206] Furthermore, the processor 1102 in this embodiment may specifically be: a programmable switching processing chip, in which a data replication engine is configured to replicate received data.

[0207] When the processor 1102 executes the program in the memory, in addition to the above functions, it can also realize other functions, which can be specifically referred to the description of the above embodiments. Fig.11 As shown, the electronic device also includes: a power supply component 1104 and other components.

[0208] Based on the technical solutions related to the above embodiments, it can be applied to security monitoring scenes such as supermarkets and banks to identify the behavior of people entering the monitoring range, and it can also be used to identify the behavior of athletes in sports events. The technical solution of this application can effectively improve the accuracy of target object behavior recognition in the video.

[0209] In summary, the technical solutions provided in each embodiment can be applied to a variety of application scenarios. For example, the motion recognition of robots (sorting robots, packaging robots, transfer robots, etc. in the logistics industry), the behavior recognition of processing robots, mechanical arms or employees in processing plants, the behavior recognition of passengers in security inspections (such as airport security, railway station security, subway security), etc., by identifying the robot movement, the operating actions of the processing execution subject (such as processing robots, employees, etc.) during processing, and the behavior of the inspected personnel; the identified results can be used to lock illegal movements (such as sorting action errors, transfer route errors), illegal operations (such as dangerous operations in processing plants), abnormal behaviors of the inspected objects, etc., so as to take corresponding measures as soon as possible to reduce losses and improve safety. The following will explain the application of the technical solutions of each embodiment of the present application for each application scenario.

[0210] 1. Robot Monitoring Scenario

[0211] Specifically, an embodiment of the present application provides an image processing method applied to a robot monitoring scene. The method comprises the following steps:

[0212] S11, obtaining surveillance video of the robot in the venue;

[0213] S12, determining the spatiotemporal features of the multiple frames of the monitoring video and the inter-frame continuity features of adjacent frames of the multiple frames; wherein the inter-frame continuity features reflect the degree of continuity of the robot's movements in adjacent frames;

[0214] S13, adjusting the spatiotemporal feature according to the inter-frame continuity feature;

[0215] S14. Using the adjusted spatiotemporal features, identify the actions of the robot in the surveillance video.

[0216] The surveillance video in S11 above may be collected by one or more cameras deployed in the venue. The venue may be a warehouse, a sorting site, a restaurant, a hotel, a bank, etc. The functions and responsibilities of the robot will be different for different venues. In a warehouse, the robot's job is to transport goods; in a sorting site, the robot's job is to sort items. In restaurants, hotels, banks and other venues, the robot's job is to provide corresponding services to people, such as delivering dishes, guiding check-in, guiding operations, etc.

[0217] For the contents of S12 to S14, please refer to the corresponding contents in the above embodiments. In addition, it should be noted that the inter-frame continuity feature and the spatiotemporal feature in this embodiment, together with the first inter-frame continuity feature and the first spatiotemporal feature mentioned in the above embodiments, can also be the second inter-frame continuity feature or the second spatiotemporal feature.

[0218] The recognition results obtained after identifying the robot's actions can be used to lock illegal actions and send corresponding instructions to the robot that has committed illegal actions, or send reminder information about the robot that has committed illegal actions to the manager's client device, etc.

[0219] 2. Processing Factory Scene

[0220] Specifically, an embodiment of the present application provides an image processing method applied to a processing factory scene. The method comprises the following steps:

[0221] S21, obtaining a video of a subject performing an operation in a processing place;

[0222] S22, determining the spatiotemporal features of the multiple frames of images in the video and the inter-frame continuity features of adjacent frames of images in the multiple frames of images; wherein the inter-frame continuity features reflect the degree of continuity of the operation actions of the subject in adjacent frames of images;

[0223] S23, adjusting the spatiotemporal feature according to the inter-frame continuity feature;

[0224] S24: using the adjusted spatiotemporal features, identifying the operating actions of the subject appearing in at least part of the frame images in the video.

[0225] The subject that generates the above-mentioned operation action can be a machine or a staff member. The machine can be: a mechanical arm or a robot involved in processing.

[0226] For the contents of S22 to S24, please refer to the corresponding contents in the above embodiments. In addition, it should be noted that the inter-frame continuity feature and the spatiotemporal feature in this embodiment, the first inter-frame continuity feature and the first spatiotemporal feature mentioned in the above embodiments, can also be the second inter-frame continuity feature or the second spatiotemporal feature.

[0227] The recognition result obtained after identifying the action of the subject that generates the operating action can be used to lock the illegal action and send corresponding instructions to the machine where the illegal action occurs, or send reminder information of the machine or staff where the illegal action occurs to the manager's client device, etc., so as to take corresponding measures as soon as possible to reduce losses and improve processing safety.

[0228] 3. Security Detection Scenarios

[0229] Specifically, an embodiment of the present application provides an image processing method applied to a security detection scenario. The method comprises the following steps:

[0230] S31, obtaining a video of the object to be inspected in the security inspection site;

[0231] S32, determining the spatiotemporal features of the multiple frames of images in the video and the inter-frame continuity features of adjacent frames of images in the multiple frames; wherein the inter-frame continuity features reflect the degree of continuity of the behavior of the object to be detected in adjacent frames of images;

[0232] S33, adjusting the spatiotemporal feature according to the inter-frame continuity feature;

[0233] S34: using the adjusted spatiotemporal features, identify the behavior of the security detection object appearing in at least part of the frame images in the video.

[0234] Among them, the safety inspection sites can be: subways, railway stations, airports, buildings, residential areas, etc.

[0235] For the contents of S32 to S34, please refer to the corresponding contents in the above embodiments. In addition, it should be noted that the inter-frame continuity feature and the spatiotemporal feature in this embodiment, the first inter-frame continuity feature and the first spatiotemporal feature mentioned in the above embodiments, can also be the second inter-frame continuity feature or the second spatiotemporal feature.

[0236] The identification results obtained after identifying the behavior of the security detection object can be used to lock suspicious persons and send and display prompt information of suspicious persons to the manager's client device so that corresponding measures can be taken as soon as possible to improve security.

[0237] The above only shows a few specific scenarios. The embodiments of the present application can also be applied to other scenarios that require recognition of behaviors, movements, and actions, which are not listed one by one in this article.

[0238] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0239] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, it can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0240] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. An image processing method, characterized in that: include: Acquire multiple frames of images; Determine a first spatiotemporal feature of the multiple frames of images and a first inter-frame continuity feature of adjacent frame images in the multiple frames of images; wherein the first inter-frame continuity feature reflects the degree of coherence in terms of motion coherence of a target object in adjacent frame images; the spatial feature in the first spatiotemporal feature is used to describe foreground features and background features in the image, and the temporal feature in the first spatiotemporal feature is used to capture motion information embedded in the image that changes over time; Adjusting the first spatiotemporal feature according to the first inter-frame continuity feature; The movement behavior of the target object appearing in at least part of the multiple frames of images is identified by using the adjusted first spatiotemporal feature.

2. The method according to claim 1, characterized in that Determining a first spatiotemporal feature of the multiple frames of images and a first inter-frame continuity feature of adjacent frame images in the multiple frames of images includes: Using the multiple frames of images as input of at least one first calculation layer in the first recognition model, executing the at least one first calculation layer, and obtaining first spatiotemporal features of the multiple frames of images; wherein the first spatiotemporal features include first time features and first spatial features; The first spatial features of the multiple frame images are used as input of at least one second calculation layer of the first recognition model, and the at least one second calculation layer is executed to output the first inter-frame continuity features of adjacent frame images in the multiple frame images.

3. The method according to claim 2, characterized in that Using the adjusted first spatiotemporal feature to identify the motion behavior of the target object in the multiple frames of images includes: Using the adjusted first spatiotemporal feature as input of at least one third computing layer of the first recognition model, executing the at least one third computing layer to output second spatiotemporal features of the multiple-frame images; wherein the second spatiotemporal feature includes a second time feature and a second spatial feature of the multiple-frame images; Using the second spatial feature and the first inter-frame continuity feature of adjacent frame images in the multiple frame images output by the at least one second computing layer as inputs of at least one fourth computing layer of the first recognition model, and executing the at least one fourth computing layer to output the second inter-frame continuity feature of adjacent frame images in the multiple frame images; Adjusting the second spatiotemporal feature according to a second inter-frame continuity feature of adjacent frame images in the multiple frame images output by the at least one fourth computing layer; The movement behavior of the target object in the multiple frames of images is identified by using the adjusted second spatiotemporal features.

4. The method according to claim 3, characterized in that Using the adjusted second spatiotemporal feature to identify the motion behavior of the target object in the multiple frames of images includes: The adjusted second spatiotemporal features are used as inputs of a classification layer of the first recognition model, and the classification layer is executed to output recognition results of target objects in the multiple frames of images.

5. The method according to claim 1, characterized in that The multiple frames of images include adjacent first frame of image and second frame of image; as well as Determining the first inter-frame continuity feature of adjacent frame images includes: Extracting image features of the first frame image and image features of the second frame image; Determining an image feature difference based on an image feature of the first frame of image and an image feature of the second frame of image; A first inter-frame continuity feature between the first frame image and the second frame image is determined according to the image feature difference.

6. The method according to claim 1, characterized in that Determining a first inter-frame continuity feature of adjacent frame images in the multiple frame images includes: Obtaining a continuity prediction model; wherein the continuity prediction model is obtained after the first machine learning model is trained; Taking the multiple frames of images as input of the continuity prediction model, executing the continuity prediction model to output an inter-frame continuity probability value of any frame image in the multiple frames of images relative to an adjacent frame image; The first inter-frame continuity feature includes an inter-frame continuity probability value of any frame image in the multiple frame images relative to an adjacent frame image.

7. The method according to claim 1, characterized in that Determining a first spatiotemporal feature of the multiple frames of images includes: Obtain a second recognition model; wherein the second recognition model is obtained after training the second machine learning model; Using the multiple frames of images as input of the second recognition model, and calculating the first spatiotemporal features through at least one first calculation layer of the second recognition model; The second recognition model further includes a classification layer for outputting a recognition result based on the first spatiotemporal feature.

8. The method according to any one of claims 1 to 7, characterized in that The inter-frame continuity feature includes the inter-frame continuity probability value of the first frame image and the second frame image; the first spatiotemporal feature includes: time feature information of the first frame image, space feature information of the first frame image, time feature information of the second frame image and space feature information of the second frame image; Adjusting the first spatiotemporal feature according to the first inter-frame continuity feature includes: Using the inter-frame continuity probability value between the first frame image and the second frame image as an adjustment coefficient of the time feature information of the second frame image, and adjusting the time feature information of the second frame image in the first spatiotemporal feature; The inter-frame continuity probability value between the first frame image and the second frame image is used as an adjustment coefficient of the spatial feature information of the second frame image to adjust the spatial feature information of the second frame image in the first spatiotemporal feature.

9. An image processing method, characterized in that: include: Acquire multiple frames of images; Determining inter-frame continuity features of adjacent frame images in the plurality of frame images, wherein the inter-frame continuity features reflect the degree of coherence of adjacent frame images in terms of motion coherence of the target object; According to the inter-frame continuity feature, determining the influence of the image feature of each frame image in the multiple frames of images in the process of motion behavior recognition; the image feature includes a spatiotemporal feature, the spatial feature in the spatiotemporal feature is used to describe the foreground feature and the background feature in the image, and the time feature in the spatiotemporal feature is used to capture the motion information embedded in the image that changes continuously over time; Adjusting the spatiotemporal features of the multiple image frames according to the influence degree corresponding to the image features of each image frame in the multiple image frames; The adjusted spatiotemporal features are used to identify the motion behavior of the target object appearing in at least part of the multiple frames of images.

10. The method according to claim 9, characterized in that The multiple frames of images include adjacent first frame of image and second frame of image; as well as Determine the inter-frame continuity features of adjacent frame images, including: Extracting image features of the first frame image and image features of the second frame image; Determining an image feature difference based on an image feature of the first frame of image and an image feature of the second frame of image; According to the image feature difference, an inter-frame continuity feature between the first frame image and the second frame image is determined.

11. The method according to claim 9, characterized in that Determining inter-frame continuity features of adjacent frame images in the multiple frame images includes: Obtaining a continuity prediction model; wherein the continuity prediction model is obtained after the first machine learning model is trained; Taking the multiple frames of images as input of the continuity prediction model, executing the continuity prediction model to output an inter-frame continuity probability value of any frame image in the multiple frames of images relative to an adjacent frame image; The inter-frame continuity feature includes an inter-frame continuity probability value of any frame image in the multiple frame images relative to an adjacent frame image.

12. The method according to claim 11, characterized in that The inter-frame continuity feature contains an inter-frame continuity probability value of a second frame image relative to a first frame image in the multiple frames of images; as well as Determining the influence of the image features of each frame image in the plurality of frames of images in the process of motion behavior recognition according to the inter-frame continuity features includes: The inter-frame continuity probability value of the second frame image relative to the first frame image is used as the influence degree corresponding to the second frame image.

13. A recognition model, characterized in that include: At least one first calculation layer, used to determine first spatiotemporal features of multiple frames of images; The first spatiotemporal feature includes a first time feature and a first space feature, the first time feature is used to describe the foreground feature and the background feature in the image, and the first space feature is used to capture motion information embedded in the image that changes over time; at least one second calculation layer, used to determine, based on the first spatial feature, a first inter-frame continuity feature of adjacent frame images in the multiple frame images, wherein the first inter-frame continuity feature reflects the degree of coherence of adjacent frame images in terms of motion coherence of the target object; A first adjustment layer, configured to adjust the first spatiotemporal feature according to the first inter-frame continuity feature output by the at least one second calculation layer; A downstream combination layer, which includes a classification layer, for processing the adjusted first spatiotemporal features and outputting a motion behavior recognition result of the target object through the classification layer; The target object appears in at least some of the frame images of the multiple frame images.

14. The recognition model according to claim 13, characterized in that The downstream combined layers include: At least one third computing layer, used for outputting second spatiotemporal features of the multiple-frame images according to the adjusted first spatiotemporal features, wherein the second spatiotemporal features include second time features and second spatial features of the multiple-frame images; At least one fourth computing layer is used to output a second inter-frame continuity feature of adjacent frame images in the multiple frame images based on the second spatial feature and the first inter-frame continuity feature of adjacent frame images in the multiple frame images output by the at least one second computing layer.

15. An image processing method, characterized in that: include: Obtain surveillance video of the robot in the venue; Determine the spatiotemporal features of multiple frames of images in the surveillance video and the inter-frame continuity features of adjacent frames of images in the multiple frames; wherein the inter-frame continuity features reflect the degree of continuity of robot movements in adjacent frames; the spatial features in the spatiotemporal features are used to describe the foreground features and background features in the image, and the temporal features in the spatiotemporal features are used to capture motion information embedded in the image that changes over time; According to the inter-frame continuity feature, adjusting the spatiotemporal feature; The adjusted spatiotemporal features are used to identify the actions of the robot in the surveillance video.

16. An image processing method, characterized in that: include: Acquire a video of a subject performing an operation in a processing place; Determine the spatiotemporal features of multiple frames of images in the video and the inter-frame continuity features of adjacent frames of images in the multiple frames; wherein the inter-frame continuity features reflect the degree of continuity of the operation actions of the subject in adjacent frames of images; the spatial features in the spatiotemporal features are used to describe the foreground features and background features in the image, and the temporal features in the spatiotemporal features are used to capture motion information embedded in the image that changes over time; According to the inter-frame continuity feature, adjusting the spatiotemporal feature; The adjusted spatiotemporal features are used to identify the operating actions of the subject appearing in at least part of the frame images in the video.

17. An image processing method, characterized in that: include: Obtaining videos of the objects being inspected in the security inspection area; Determine the spatiotemporal features of multiple frames of images in the video and the inter-frame continuity features of adjacent frames of images in the multiple frames; wherein the inter-frame continuity features reflect the degree of continuity of the behavior of the object to be detected in adjacent frames of images; the spatial features in the spatiotemporal features are used to describe the foreground features and background features in the image, and the temporal features in the spatiotemporal features are used to capture motion information embedded in the image that changes over time; According to the inter-frame continuity feature, adjusting the spatiotemporal feature; The adjusted spatiotemporal features are used to identify the behavior of the security detection object appearing in at least part of the frame images in the video.

18. An electronic device, characterized in that: comprising a memory and a processor; wherein, The memory is used to store programs; The processor is coupled to the memory and is configured to execute the program stored in the memory to: Acquire multiple frames of images; Determine a first spatiotemporal feature of the multiple frames of images and a first inter-frame continuity feature of adjacent frame images in the multiple frames of images; wherein the first inter-frame continuity feature reflects the degree of coherence in terms of motion coherence of a target object in adjacent frame images; the spatial feature in the first spatiotemporal feature is used to describe foreground features and background features in the image, and the temporal feature in the first spatiotemporal feature is used to capture motion information embedded in the image that changes over time; Adjusting the first spatiotemporal feature according to the first inter-frame continuity feature; The movement behavior of the target object appearing in at least part of the multiple frames of images is identified by using the adjusted first spatiotemporal feature.

19. An electronic device, characterized in that: comprising a memory and a processor; wherein, The memory is used to store programs; The processor is coupled to the memory and is configured to execute the program stored in the memory to: Acquire multiple frames of images; Determining inter-frame continuity features of adjacent frame images in the plurality of frame images, wherein the inter-frame continuity features reflect the degree of coherence of adjacent frame images in terms of motion coherence of the target object; According to the inter-frame continuity feature, determining the influence of the image feature of each frame image in the multiple frames of images in the process of motion behavior recognition; the image feature includes a spatiotemporal feature, the spatial feature in the spatiotemporal feature is used to describe the foreground feature and the background feature in the image, and the time feature in the spatiotemporal feature is used to capture the motion information embedded in the image that changes continuously over time; Adjusting the spatiotemporal features of the multiple image frames according to the influence degree corresponding to the image features of each image frame in the multiple image frames; The adjusted spatiotemporal features are used to identify the motion behavior of the target object appearing in at least part of the multiple frames of images.

Images