Image Recognition Method, Device, Electronic Device and Computer Readable Storage Medium

Abstract

An embodiment of the present application provides an image recognition method, apparatus, electronic device, and computer-readable storage medium, which relate to the field of computer technology. The method includes: determining a target frame image being currently processed in a target video sequence, inputting the target frame image into a first recognition model, obtaining first position information of a target sliding window sub-image output after the first recognition model performs sliding window slicing on the target frame image in the target frame image, and then inputting the target frame image and the first position information into a second recognition model, obtaining coordinate positions of each target object in the target frame image after the second recognition model performs anchor box slicing on the target frame image based on the first position information. In the embodiment of the present application, the area position where a target object may exist is first determined based on sliding window slicing, and then the area position is accurately recognized based on anchor box slicing, which can improve the accuracy of image recognition and reduce the amount of calculation in the image recognition process.

Description

Technical Field

[0001] This application relates to the field of computer technology. Specifically, this application relates to an image recognition method, apparatus, electronic device, and computer-readable storage medium. Background Art

[0002] The main task of multi-object tracking (MOT) is to find moving objects in a given image sequence based on the appearance features, positions, motion states, etc. of the objects, and associate the different moving objects detected in different image frames with different identities (IDs) one by one, and finally obtain the motion trajectories of different objects.

[0003] Large-scale scene image sequences contain richer object information and can perform long-term and long-distance analysis of objects and object activities within a large-scale spatio-temporal range. However, due to the wide coverage of the images and lighting conditions, etc., there are significant differences in the appearance of the same object at different times; moreover, in some scenes, the objects are dense, the scale span is large, and there are serious occlusion and blur phenomena, making the matching difficult; the video frame rate is high and the pixels are extremely high, resulting in low recognition accuracy and large computational complexity of traditional algorithms. Summary of the Invention

[0004] The purpose of the embodiments of this application is to solve the problems of low recognition accuracy and large computational complexity for target trajectory recognition in high-frame-rate images.

[0005] According to one aspect of the embodiments of this application, an image recognition method is provided. The method includes:

[0006] Determine the target frame image being currently processed in the target video sequence, where the target frame image includes at least one target object;

[0007] Input the target frame image into a first recognition model, and obtain the first position information of the target sliding window sub-image output by the first recognition model after performing sliding window slicing processing on the target frame image. The target sliding window sub-image is a sliding window sub-image including at least one target object;

[0008] Input the target frame image and the first position information into a second recognition model, and obtain the coordinate positions of each target object in the target frame image output by the second recognition model after performing anchor box slicing processing on the target frame image based on the first position information.

[0009] Optionally, the method further includes:

[0010] Determine the feature information of each target object in the target frame image according to the coordinate positions of each target object in the target frame image;

[0011] Determine the feature information and movement trajectory information of the identified object in the reference image;

[0012] Determine the movement trajectory information of each target object according to the feature information of each target object, the coordinate positions of each target object in the target frame image, the feature information of the identified object, and the movement trajectory information;

[0013] Among them, the reference image is a frame image in the target video sequence that is before the target frame image; the movement trajectory information of the identified object includes the coordinate position of the identified object in the corresponding reference image, and the predicted coordinate position of the identified object in the target frame image predicted based on the coordinate position of the identified object in the reference image.

[0014] Optionally, determining the movement trajectory information of each target object according to the feature information of each target object, the coordinate positions of each target object in the target frame image, the feature information of the identified object, and the movement trajectory information includes:

[0015] For any target object, if it is determined that there is an identified object that matches the target object according to the feature information of the target object and the feature information of each identified object, then obtain the movement trajectory information of the target object according to the coordinate position of the target object in the target frame image and the movement trajectory information of the matching identified object.

[0016] Optionally, the movement trajectory information of the target object includes the predicted coordinate position of the target object in the predicted image; the predicted image is a frame image in the target video sequence that is after the target frame image.

[0017] Optionally, determining the movement trajectory information of each target object according to the feature information of each target object, the coordinate positions of each target object in the target frame image, the feature information of the identified object, and the movement trajectory information further includes:

[0018] If it is determined that there is no identified object that matches the target object according to the feature information of the target object and the feature information of each identified object, then match the coordinate position of the target object in the target frame image with the predicted coordinate positions of each identified object in the target frame image;

[0019] If it is determined that there is an identified object that matches the target object according to the coordinate position of the target object in the target frame image and the predicted coordinate positions of each identified object in the target frame image, then obtain the movement trajectory information of the target object according to the coordinate position of the target object in the target frame image and the movement trajectory information of the matching identified object.

[0020] Optionally, the training steps of the second recognition model are as follows:

[0021] Obtain multiple sample frame images in a sample video sequence. The sample frame images include at least one sample object and a sample label corresponding to the sample object. The sample label includes the coordinate position of the sample object in the sample frame image;

[0022] Input the sample frame image into the initial recognition model, and obtain the initial coordinate positions of each sample object in the sample frame image after the initial recognition model performs anchor box cropping on the sample frame image. The initial coordinate positions are determined by the heat map detection module, the center offset detection module, and the box size detection module in the initial recognition model based on the General Focal Loss V2 loss calculation method;

[0023] According to the initial coordinate positions output by the initial recognition model and the coordinate positions of the sample objects in the sample frame image, determine the loss value of the General Focal Loss V2 loss calculation method, and iteratively train the initial recognition model based on the loss value until the loss value meets the training end condition, then stop training and determine the initial recognition model at the end of training as the second recognition model.

[0024] Optionally, input the target frame image and the first position information into the second recognition model, and obtain the coordinate positions of each target object in the target frame image after the second recognition model performs anchor box cropping on the target frame image based on the first position information, including:

[0025] Determine the anchor boxes corresponding to each target object after the target frame image is subjected to anchor box cropping;

[0026] Determine the feature maps of the anchor boxes corresponding to each target object, and input the feature maps into the heat map detection module, the center offset detection module, and the box size detection module of the second recognition model respectively to obtain the center coordinate positions, center offset data of each target object in the corresponding feature map, and the height and width of the anchor boxes corresponding to each target object;

[0027] According to the center offset data of each target object in the corresponding feature map, update the center coordinate positions of each target object in the corresponding feature map, and according to the updated center coordinate positions of each target object in the corresponding feature map and the height and width of the anchor boxes corresponding to each target object, determine the coordinate positions of each target object in the target frame image.

[0028] Optionally, the General Focal Loss V2 loss calculation method includes a quality coordination loss calculation method;

[0029] Input the feature map into the heatmap detection module, center offset detection module, and box size detection module of the second recognition model respectively to obtain the center coordinate positions, center offset data of each target object in the corresponding feature map, and the height and width of the anchor box corresponding to each target object, including:

[0030] Input the feature map into the heatmap detection module, and determine the heatmap detection loss of the feature map based on the quality coordination loss calculation method;

[0031] According to the heatmap detection loss, perform heatmap detection on the feature map to determine the center coordinate positions of each target object in the corresponding feature map.

[0032] Optionally, the General Focal Loss V2 loss calculation method includes the distribution coordination loss calculation method;

[0033] Input the feature map into the heatmap detection module, center offset detection module, and box size detection module of the second recognition model respectively to obtain the center coordinate positions, center offset data of each target object in the corresponding feature map, and the height and width of the anchor box corresponding to each target object, and further include:

[0034] Input the feature map into the center offset detection module, and determine the center offset detection loss of the feature map based on the distribution coordination loss calculation method;

[0035] According to the center offset detection loss, perform center offset detection on the feature map to determine the center offset data of each target object in the corresponding feature map.

[0036] Optionally, input the feature map into the heatmap detection module, center offset detection module, and box size detection module of the second recognition model respectively to obtain the center coordinate positions, center offset data of each target object in the corresponding feature map, and the height and width of the anchor box corresponding to each target object, and further include:

[0037] Input the feature map into the box size detection module, and determine the box size detection loss of the feature map based on the distribution coordination loss calculation method;

[0038] According to the box size detection loss, perform box size detection on the feature map to determine the height and width of the anchor box corresponding to each target object.

[0039] According to another aspect of the embodiments of the present application, an image recognition device is provided, and the device includes:

[0040] A determination module, configured to determine the target frame image currently processed in the target video sequence, and the target frame image includes at least one target object;

[0041] The first recognition module is configured to input the target frame image into the first recognition model, and obtain the first position information of the target sliding sub-image in the target frame image output by the first recognition model after performing sliding window slicing on the target frame image, where the target sliding sub-image is a sliding sub-image including at least one target object;

[0042] The second recognition module is configured to input the target frame image and the first position information into the second recognition model, and obtain the coordinate positions of each target object in the target frame image output by the second recognition model after performing anchor box slicing on the target frame image based on the first position information.

[0043] Optionally, the apparatus further includes:

[0044] The feature determination module is configured to determine the feature information of each target object in the target frame image according to the coordinate positions of each target object in the target frame image;

[0045] The reference determination module is configured to determine the feature information and the movement trajectory information of the recognized objects in the reference image;

[0046] The trajectory determination module is configured to determine the movement trajectory information of each target object according to the feature information of each target object, the coordinate positions of each target object in the target frame image, the feature information of the recognized objects, and the movement trajectory information;

[0047] Wherein, the reference image is a frame image in the target video sequence before the target frame image; the movement trajectory information of the recognized objects includes the coordinate positions of the recognized objects in the corresponding reference image, and the predicted coordinate positions of the recognized objects in the target frame image predicted based on the coordinate positions of the recognized objects in the reference image.

[0048] Optionally, the trajectory determination module is specifically configured to:

[0049] For any target object, if it is determined that there is a recognized object matching the target object according to the feature information of the target object and the feature information of each recognized object, then the movement trajectory information of the target object is obtained according to the coordinate position of the target object in the target frame image and the movement trajectory information of the matching recognized object.

[0050] Optionally, the movement trajectory information of the target object includes the predicted coordinate position of the target object in the predicted image; the predicted image is a frame image in the target video sequence after the target frame image.

[0051] Optionally, the trajectory determination module is further configured to:

[0052] If it is determined that there is no recognized object that matches the target object according to the feature information of the target object and the feature information of each recognized object, then the coordinate position of the target object in the target frame image is matched with the predicted coordinate positions of each recognized object in the target frame image;

[0053] If it is determined that there is a recognized object that matches the target object according to the coordinate position of the target object in the target frame image and the predicted coordinate positions of each recognized object in the target frame image, then the movement trajectory information of the target object is obtained according to the coordinate position of the target object in the target frame image and the movement trajectory information of the matched recognized object.

[0054] Optionally, the training steps of the second recognition model are as follows:

[0055] Obtain multiple sample frame images in the sample video sequence. Each sample frame image includes at least one sample object and a sample label corresponding to the sample object. The sample label includes the coordinate position of the sample object in the sample frame image;

[0056] Input the sample frame image into the initial recognition model, and obtain the initial coordinate positions of each sample object in the sample frame image output after the initial recognition model performs anchor box cropping processing on the sample frame image; the initial coordinate positions are determined by the heat map detection module, the center offset detection module, and the box size detection module in the initial recognition model based on the General Focal Loss V2 loss calculation method;

[0057] According to the initial coordinate positions output by the initial recognition model and the coordinate positions of the sample objects in the sample frame image, determine the loss value of the General Focal Loss V2 loss calculation method, and perform iterative training on the initial recognition model based on the loss value until the loss value meets the training end condition, then stop training and determine the initial recognition model at the end of training as the second recognition model.

[0058] Optionally, the second determination module includes:

[0059] A cropping module, configured to determine the anchor boxes corresponding to each target object obtained after the target frame image is subjected to anchor box cropping processing;

[0060] A detection module, configured to determine the feature maps of the anchor boxes corresponding to each target object, and input the feature maps into the heat map detection module, the center offset detection module, and the box size detection module of the second recognition model respectively, to obtain the center coordinate positions, center offset data, and the heights and widths of the anchor boxes corresponding to each target object in the corresponding feature maps;

[0061] A second determination sub-module, configured to update the central coordinate positions of each target object in the corresponding feature map according to the central offset data of each target object in the corresponding feature map, and determine the coordinate positions of each target object in the target frame image according to the updated central coordinate positions of each target object in the corresponding feature map and the height and width of the anchor box corresponding to each target object.

[0062] Optionally, the General Focal Loss V2 loss calculation method includes a quality coordination loss calculation method;

[0063] The detection module is specifically configured to:

[0064] Input the feature map into the heat map detection module, and determine the heat map detection loss of the feature map based on the quality coordination loss calculation method;

[0065] Perform heat map detection on the feature map according to the heat map detection loss, and determine the central coordinate positions of each target object in the corresponding feature map.

[0066] Optionally, the General Focal Loss V2 loss calculation method includes a distribution coordination loss calculation method;

[0067] The detection module is further configured to:

[0068] Input the feature map into the central offset detection module, and determine the central offset detection loss of the feature map based on the distribution coordination loss calculation method;

[0069] Perform central offset detection on the feature map according to the central offset detection loss, and determine the central offset data of each target object in the corresponding feature map.

[0070] Optionally, the detection module is further configured to:

[0071] Input the feature map into the box size detection module, and determine the box size detection loss of the feature map based on the distribution coordination loss calculation method;

[0072] Perform box size detection on the feature map according to the box size detection loss, and determine the height and width of the anchor box corresponding to each target object.

[0073] According to another aspect of the embodiments of the present application, an electronic device is provided. The electronic device includes a memory, a processor, and a computer program stored on the memory. The processor executes the computer program to implement the steps of the image recognition method in any of the above aspects.

[0074] According to still another aspect of the embodiments of the present application, a computer-readable storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the steps of the image recognition method in any of the above aspects are implemented.

[0075] The beneficial effects brought by the technical solution provided in the embodiment of the present application are as follows:

[0076] First, based on sliding window slicing, the regional positions where target objects may exist are determined, and then based on anchor box slicing, the regional positions are accurately identified, which improves the accuracy of image recognition and reduces the computational amount in the process of image recognition. Description of the Drawings

[0077] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following will briefly introduce the drawings required for the description in the embodiments of the present application.

[0078] Figure 1 It is one of the schematic flowcharts of an image recognition method provided in the embodiment of the present application;

[0079] Figure 2 It is another schematic flowchart of an image recognition method provided in the embodiment of the present application;

[0080] Figure 3 It is still another schematic flowchart of an image recognition method provided in the embodiment of the present application;

[0081] Figure 4 It is yet another schematic flowchart of an image recognition method provided in the embodiment of the present application;

[0082] Figure 5 It is a schematic flowchart of a process for training a second recognition model provided in the embodiment of the present application;

[0083] Figure 6 It is a schematic structural diagram of an image recognition device provided in the embodiment of the present application;

[0084] Figure 7 It is a schematic structural diagram of an electronic device for image recognition provided in the embodiment of the present application. Detailed Embodiments

[0085] The following describes the embodiments of the present application with reference to the drawings in the present application. It should be understood that the embodiments described below in conjunction with the drawings are exemplary descriptions for explaining the technical solutions of the embodiments of the present application, and do not constitute a limitation on the technical solutions of the embodiments of the present application.

[0086] Those skilled in the art can understand that, unless specifically stated otherwise, the singular forms "a", "an", "the" and "said" used herein may also include the plural forms. It should be further understood that the terms "comprising" and "including" used in the embodiments of the present application mean that the corresponding features can be implemented as the presented features, information, data, steps, operations, elements and / or components, but do not exclude the implementation of other features, information, data, steps, operations, elements, components and / or their combinations supported by the technical field of the present application. It should be understood that when we say that an element is "connected" or "coupled" to another element, the one element can be directly connected or coupled to the other element, or it can mean that the one element and the other element establish a connection relationship through an intermediate element. In addition, the "connection" or "coupling" used herein may include wireless connection or wireless coupling. The term "and / or" used herein indicates at least one of the items defined by the term. For example, "A and / or B" can be implemented as "A", or as "B", or as "A and B".

[0087] To make the objectives, technical solutions and advantages of the present application clearer, the embodiments of the present application will be further described in detail below with reference to the accompanying drawings.

[0088] The technical solutions of the embodiments of the present application and the technical effects produced by the technical solutions of the present application will be described below through the description of several exemplary embodiments. It should be noted that the following embodiments can be referenced, learned from or combined with each other. For the same terms, similar features and similar implementation steps in different embodiments, they will not be described repeatedly.

[0089] An image recognition method is provided in an embodiment of the present application, as Figure 1 shown, including:

[0090] Step S101, determining a target frame image currently processed in a target video sequence, where the target frame image includes at least one target object.

[0091] The target video sequence is a video sequence composed of a series of video frame images arranged in chronological order, and the video frame image is the smallest unit constituting the video sequence.

[0092] For example, a web camera captures a scene video of a certain area in real time and uploads the scene video or the target video sequence corresponding to the scene video to a network server. When the web camera directly uploads the scene video to the network server, the network server performs frame extraction processing on the scene video, extracts video frame images from the scene video at a certain frame interval, and a number of video frame images form a target video sequence. When the web camera uploads the target video sequence corresponding to the scene video to the network server, the web camera or other devices connected to the web camera perform frame extraction processing on the scene video to obtain the target video sequence.

[0093] The scene video captured by the network camera can be a high-frame-rate video. For example, the scene video of the PANDA large-scale scene video dataset is collected using a billion-pixel camera. The global view of this scene video covers a natural scene of about 1 square kilometer, and the natural scene can accommodate about 4,000 people. There is a scale change of more than 100 times for each target object, and the appearance features of each target object are still clearly visible in the maximum magnification view.

[0094] A series of video frame images are extracted from the scene video to form a target video sequence with a low frame rate. Among them, each video frame image includes at least one target object. For example, when detecting the pedestrian flow in a commercial area, the target object is the pedestrians appearing in this commercial area; when detecting the vehicle flow on a road, the target object is the various vehicles driving on this road. This will not be elaborated one by one in this embodiment.

[0095] For a number of video frame images in the target video sequence, image recognition processing is performed one by one in chronological order, and the currently processed video frame image is determined as the target frame image.

[0096] Step S102: Input the target frame image into the first recognition model to obtain the first position information of the target sliding sub-image output after the first recognition model performs sliding window cutting on the target frame image. The target sliding sub-image is a sliding sub-image including at least one target object.

[0097] The first recognition model is an integrated model for recognizing multiple target objects in an image. The first recognition model performs a sliding window slicing process on the target frame image to obtain a number of target sliding window sub-images. Specifically, there are slight differences between the sliding window slicing process applied in the embodiments of the present application and the conventional techniques. In the embodiments of the present application, instead of using sliding window windows of different sizes for processing, a fixed-size sliding window window is used for processing at a fixed overlapping rate, and then, based on the result after convolutional operation and the trained classifier, the probability of the existence of target objects in each sliding window sub-image is determined, and further, each target sliding window sub-image with at least one target object therein and the first position information of each target sliding window sub-image in the target frame image are determined.

[0098] For example, the target frame image is reduced to 1 / 4 of its original size, and then a sliding window window with a size of 2000*2000 is used to sequentially slice the reduced target frame image in the order from left to right and from top to bottom at an overlapping rate of 0.5 to obtain a number of sliding window sub-images. Further, each target sliding window sub-image with at least one target object therein is determined, and based on the vertex coordinates of each target sliding window sub-image, the first position information of each target sliding window sub-image in the target frame image is determined.

[0099] It should be noted that in this embodiment, reducing the size of the target frame image in advance can be processed before step S101 or can be processed by the first recognition model, so that the processed target frame image is a low-frame-rate image with a smaller size. In this embodiment, the size of the target frame image is reduced in advance because there are certain limitations on the size of the sliding window window, and for the original high-frame-rate image, it is easy to slice out incomplete target objects, which affects the result of the sliding window slicing process.

[0100] Step S103: Input the target frame image and the first position information into the second recognition model, and obtain the coordinate positions of each target object in the target frame image output by the second recognition model after performing an anchor box slicing process on the target frame image based on the first position information.

[0101] The second recognition model is an integrated model for recognizing multiple target objects in an image. The first recognition model and the second recognition model can be implemented by aggregating the same neural network model, or can be represented as two sub-models in the same model, and the first recognition model and the second recognition model can be synchronously trained. The second recognition model performs an anchor box slicing process on the target frame image based on the first position information to obtain a number of target anchor box sub-images.

[0102] For example, based on the first position information, determine the regional positions of the target objects in the target frame image. Based on the regional positions in the target frame image, determine the anchor boxes of the target objects. For the anchor box of a target object, expand it to the surrounding by a preset size and then cut it to obtain a larger target anchor box sub-image containing the anchor box. Further, the second recognition model recognizes each target anchor box sub-image to determine the coordinate positions of the target objects in the target frame image.

[0103] It should be noted that in this embodiment, a target anchor box sub-image slightly larger than the anchor box is determined to calculate the overlap rate between the anchor box and the corresponding target anchor box sub-image, and mask processing is performed on the target anchor box sub-image based on the overlap rate to ensure the integrity and effectiveness of the target objects in the target anchor box sub-image.

[0104] Apply the image recognition method provided in the embodiment of the present application. Determine the target frame image currently processed in the target video sequence, input the target frame image into the first recognition model, and obtain the first position information of the target sliding window sub-image output after the first recognition model performs sliding window slicing processing on the target frame image. Then, input the target frame image and the first position information into the second recognition model, and obtain the coordinate positions of the target objects in the target frame image output after the second recognition model performs anchor box slicing processing on the target frame image based on the first position information.

[0105] The embodiment of the present application first determines the regional positions where target objects may exist based on sliding window slicing, and then accurately recognizes the regional positions based on anchor box slicing, improving the accuracy of image recognition and reducing the computational amount in the image recognition process.

[0106] An image recognition method is provided in the embodiment of the present application, as Figure 2 shown, including:

[0107] Step S201, determine the target frame image currently processed in the target video sequence, where the target frame image includes at least one target object.

[0108] For several video frame images in the target video sequence, image recognition processing needs to be performed one by one in chronological order, and the currently processed video frame image is determined as the target frame image. Further, after determining the coordinate positions of the target objects in multiple video frame images, the movement trajectory information of the target objects in the target video sequence can be determined.

[0109] Step S202, input the target frame image into the first recognition model, and obtain the first position information of the target sliding window sub-image output after the first recognition model performs sliding window slicing processing on the target frame image, where the target sliding window sub-image is a sliding window sub-image including at least one target object.

[0110] The first recognition model performs sliding window slicing on the target frame image to obtain a number of sliding window sub-images, then determines the probability of the existence of target objects in each sliding window sub-image, and further determines each target sliding window sub-image in which there is at least one target object, as well as the first position information of each target sliding window sub-image in the target frame image.

[0111] Specifically, when the first recognition model determines the first position information of each target sliding window sub-image in the target frame image, it is necessary to perform boundary fusion processing on each target sliding window sub-image because the target objects in each target sliding window sub-image may be incomplete. One possible situation is that when performing sliding window slicing on the target frame image, a target object is sliced into multiple adjacent sliding window sub-images, making the target object in one sliding window sub-image incomplete; another possible situation is that the target objects in a certain area are relatively concentrated, or there are obstacles such as buildings, making the target objects incomplete; still another possible situation is that the light in a certain area is too strong or too weak, making the target objects incomplete.

[0112] Generally speaking, if the integrity of a certain target object in a sliding window sub-image is not less than 50%, then it is determined that there is such a target object in this sliding window sub-image, and this sliding window sub-image is a target sliding window sub-image; correspondingly, if the integrity of a certain target object in a sliding window sub-image is less than 50%, then it is determined that there is no such target object in this sliding window sub-image. For the target objects that appear in all target sliding window sub-images, boundary fusion processing is performed on the target sliding window sub-images so that the same target object appears only in one target sliding window sub-image, avoiding multiple statistics of the position of the same target object.

[0113] Step S203: Input the target frame image and the first position information into the second recognition model, and obtain the coordinate positions of each target object in the target frame image output by the second recognition model after performing anchor box slicing on the target frame image based on the first position information.

[0114] The second recognition model performs anchor box slicing on the target frame image based on the first position information to obtain a number of target anchor box sub-images.

[0115] For example, based on the first position information, determine the regional position of each target object in the target frame image, determine the anchor box of each target object at the regional position in the target frame image, and for the anchor box of a target object, expand it to the surrounding by a preset size and then slice it to obtain a larger target anchor box sub-image containing the anchor box. Further, the second recognition model recognizes each target anchor box sub-image to determine the coordinate positions of each target object in the target frame image.

[0116] In a preferred embodiment of the present application, as Figure 3 shown, step S203 includes:

[0117] Step S2031: Determine the anchor boxes corresponding to each target object obtained after the target frame image is processed by anchor box cropping.

[0118] Step S2032: Determine the feature maps of the anchor boxes corresponding to each target object, and input the feature maps into the heatmap detection module, center offset detection module, and box size detection module of the second recognition model respectively to obtain the center coordinate positions, center offset data of each target object in the corresponding feature map, and the height and width of the anchor boxes corresponding to each target object.

[0119] The anchor box corresponding to a target object is the smallest rectangular box containing the target object. The second recognition model performs anchor box cropping processing on the target frame image to obtain the anchor boxes corresponding to each target object and the target anchor box sub-images corresponding to each anchor box. Each target anchor box sub-image contains at least one target object, and the target anchor box sub-image has all the features of the corresponding anchor box. Therefore, the feature map of the anchor box corresponding to each target object can also be understood as the feature map of the target anchor box sub-image corresponding to each target object.

[0120] The Encoder-Decoder framework in the second recognition model uses the DLA-34 network. An image of C*H*W of the target anchor box sub-image is input into the network. After multiple downsamplings, upsamplings, feature summations, and skip connections, a feature map with a shape of C*H / 4*W / 4 is finally output, thereby determining the feature map corresponding to each target anchor box sub-image, that is, the feature map of the anchor box corresponding to each target object.

[0121] Among them, the DLA-34 network is a variant of the DLA network (Deep Layer Aggregation, deep layer aggregation network). Compared with the DLA network, the DLA-34 network has more skip connections between low-level features and high-level features, and all convolutions in the upsampling module are optimized into deformable convolutions, which can better adapt to the changes in the scale and shape of target objects, and to a certain extent alleviates the problem of target center point alignment existing in the Anchor-Based target detector in the second recognition model.

[0122] Input the feature maps into the Heatmap heatmap detection module, Center offset center offset detection module, and Box size box size detection module of the second recognition model respectively to obtain the center coordinate positions, center offset data of each target object in the corresponding feature map, and the height and width of the anchor boxes corresponding to each target object.

[0123] In a preferred embodiment of the present application, the General Focal Loss V2 loss calculation method includes a QFL (Quality Focal Loss) calculation method and a DFL (Distribution Focal Loss) calculation method.

[0124] Among them, the QFL method can solve the problem of inconsistent classification and quality estimation existing in the second recognition model during training and application, and the DFL method can solve the problems of boundary ambiguity and uncertainty existing during the regression of anchor boxes corresponding to each target object.

[0125] Step S2032 includes;

[0126] Input the feature map into the heatmap detection module, and based on the quality focal loss calculation method, determine the heatmap detection loss of the feature map;

[0127] According to the heatmap detection loss, perform heatmap detection on the feature map to determine the central coordinate positions of each target object in the corresponding feature map.

[0128] Based on the QFL calculation method, determine the heatmap detection loss Heatmaploss in the heatmap detection module, where the calculation formula is as follows:

[0129]

[0130] In the formula, N represents the number of detection boxes, represents the quality label from 0 to 1, represents the probability that there is a target at (x, y) in the heatmap. When it obtains the global minimum solution of L heatmap Here, it is optimal when the adjustment factor β = 2 is taken.

[0131] According to the heatmap detection loss, adopt the heatmap-based representation method to determine the central coordinate positions of each target object in the corresponding feature map.

[0132] Input the feature map into the center offset detection module, and based on the distribution focal loss calculation method, determine the center offset detection loss of the feature map;

[0133] According to the center offset detection loss, perform center offset detection on the feature map to determine the center offset data of each target object in the corresponding feature map.

[0134] Determine the center offset loss in the center offset detection module based on the DFL calculation method. Further, determine the center offset data of each target object in the corresponding feature map. Specifically, the center offset loss is generated when downsampling the feature map, and the center offset loss is a slight position offset that objectively exists.

[0135] Input the feature map into the box size detection module, and determine the box size detection loss of the feature map based on the distribution coordination loss calculation method;

[0136] According to the box size detection loss, perform box size detection on the feature map to determine the height and width of the anchor box corresponding to each target object.

[0137] Determine the box size loss in the box size detection module based on the DFL calculation method, where the calculation formula is as follows:

[0138]

[0139] In the formula, y i and y i+1 are the label points adjacent to the left and right of the true label point y respectively, and S i and S i+1 are the results after softmax processing.

[0140] According to the box size detection loss, determine the anchor point positions of the anchor boxes corresponding to each target object. Further, determine the height and width of the anchor boxes.

[0141] Step S2033: According to the center offset data of each target object in the corresponding feature map, update the center coordinate positions of each target object in the corresponding feature map, and based on the updated center coordinate positions of each target object in the corresponding feature map, as well as the height and width of the anchor boxes corresponding to each target object, determine the coordinate positions of each target object in the target frame image.

[0142] According to the center offset data, update the center coordinate positions of each target object in the corresponding feature map to achieve more accurate positioning of each center coordinate position. Then, based on the height and width of the anchor boxes corresponding to each target object, the updated center coordinate positions can be mapped to the target frame image, thereby determining the coordinate positions of each target object in the target frame image.

[0143] Specifically, when the second recognition model determines the coordinate positions of each target object in the target frame image, it is necessary to perform result fusion processing on each target anchor box sub-image to avoid the situation where the detection results of the target anchor box sub-images are repeated or incomplete when fused onto the target frame image. In the embodiments of the present application, the WBF (Weighted Boxes Fusion) method is used to achieve result fusion. The WBF method calculates the IOU (Intersection over Union) between each target anchor box sub-image and the corresponding anchor box. When the overlap degree is greater than the preset overlap degree threshold, the confidence and coordinates of the anchor boxes of each target object are weighted and fused to obtain the coordinate positions of each target object in the target frame image.

[0144] Step S204: Determine the feature information of each target object in the target frame image according to the coordinate positions of each target object in the target frame image.

[0145] The Embedding embedding module in the second recognition model uses a convolutional layer with 128 kernels to extract the identity embedding features of the target object at each coordinate position, and generates feature information that can distinguish different target objects.

[0146] The Embedding loss in the embedding module is implemented using CrossEntropy Loss. The calculation formula is as follows:

[0147]

[0148] In the formula, p(k) is the probability distribution of the predicted category (ID number) of the k-th object, and L(k) is the true one-hot encoding of the k-th object.

[0149] Step S205: Determine the feature information and movement trajectory information of the recognized objects in the reference image;

[0150] Among them, the reference image is the frame image in the target video sequence that is before the target frame image; the movement trajectory information of the recognized object includes the coordinate position of the recognized object in the corresponding reference image, and the predicted coordinate position of the recognized object in the target frame image predicted based on the coordinate position of the recognized object in the reference image.

[0151] Generally speaking, the target video sequence includes multiple video frame images. Image recognition processing is sequentially performed on the multiple video frame images according to their arrangement order, and the processed video frame images are used as reference images, the currently processed video frame image is used as the target frame image, and the video frame image to be processed is used as the predicted image. That is to say, the reference image is the frame image in the target video sequence that is before the target frame image, and the predicted image is the frame image in the target video sequence that is after the target frame image.

[0152] The target object in the reference image is an identified object. The movement trajectory information of the identified object includes the identification information of the identified object and the coordinate position of the identified object in the reference image. In the embodiments of the present application, the feature information and movement trajectory information of the identified object can be synchronously stored.

[0153] Each object in the target video sequence has a unique identification information. The identification information of the same object appearing in different video frame images is the same. In the embodiments of the present application, based on the Embedding embedding module, the feature information of each object is determined, and further, the identification information of the object is determined.

[0154] The movement trajectory information of the identified object further includes the predicted coordinate position of the identified object in the next video frame image. The feature information and movement trajectory information of the identified object in the pre-stored reference image are obtained.

[0155] Step S206: Determine the movement trajectory information of each target object according to the feature information of each target object, the coordinate position of each target object in the target frame image, the feature information of the identified object, and the movement trajectory information.

[0156] By calculating the similarity according to the feature information of each target object in the target frame image and the feature information of each identified object in the reference image, the same object in the target frame image and the reference image can be determined. Then, according to the movement trajectory information stored in the reference image of the same object, the identification information of the same object can be obtained. Further, according to the coordinate position of the same object in the reference image and the coordinate position in the target frame image, the predicted coordinate position of the target object in the next frame image can be predicted.

[0157] Determine the movement trajectory information of each target object according to the identification information, feature information, coordinate position in the reference image, coordinate position in the target frame image, and predicted coordinate position in the predicted image of the target object.

[0158] In a preferred embodiment of the present application, as Figure 4 shown, step S206 includes:

[0159] Step S2061: For any target object, if it is determined that there is a recognized object that matches the target object based on the feature information of the target object and the feature information of each recognized object, then based on the coordinate position of the target object in the target frame image and the movement trajectory information of the matching recognized object, obtain the movement trajectory information of the target object.

[0160] The movement trajectory information of the target object includes the predicted coordinate position of the target object in the predicted image; the predicted image is a frame image in the target video sequence that is after the target frame image.

[0161] Specifically, the method for matching the recognized object in the reference image and the target object in the target frame image in the embodiments of the present application includes at least two steps. Among them, the first step is to perform matching based on feature information.

[0162] For any target object, determine the similarity matrix of the feature information of the target object and the feature information of all recognized objects in the reference image, and determine whether there is a recognized object that matches the target object in the reference image through the similarity matrix.

[0163] If there is, obtain the movement trajectory information of the recognized object. The movement trajectory information of the recognized object includes the identification information, feature information, coordinate position in the reference image, and predicted coordinate position in the target frame image of the recognized object.

[0164] Update the feature information of the recognized object according to the feature information of the target object, update the predicted coordinate position in the movement trajectory information of the recognized object according to the coordinate position of the target object in the target frame image, and predict the predicted coordinate position of the target object in the predicted image according to the coordinate positions of the target object in the reference image and the target frame image, so as to obtain the updated movement trajectory information of the target object.

[0165] Step S2062: If it is determined that there is no recognized object that matches the target object based on the feature information of the target object and the feature information of each recognized object, then match the coordinate position of the target object in the target frame image with the predicted coordinate positions of each recognized object in the target frame image.

[0166] Since there may be target objects with unclear feature information in the target frame image, in order to avoid affecting the result of object matching, in the embodiments of the present application, matching based on coordinate position is used as the supplementary second step.

[0167] If it is determined that there is no recognized object that matches the target object when calculating the similarity based on the feature information, then match the coordinate position of the target object in the target frame image with the predicted coordinate positions of each recognized object in the target frame image.

[0168] Specifically, in the step of matching based on the coordinate positions, not all the coordinate positions of the target objects are matched with the predicted coordinate positions of all the recognized objects. Instead, on the basis of matching based on the feature information, the target objects and recognized objects that fail to match are screened out, and then a second matching is performed.

[0169] For example, there are two recognized objects A and B in the reference image, and there are three target objects C, D, and E in the target frame image. If A and C are successfully matched when matching based on the feature information, then when matching based on the coordinate positions, only the predicted coordinate position of B needs to be matched with the coordinate positions of D and E, reducing the computational amount.

[0170] Step S2063, if it is determined that there is a recognized object that matches the target object according to the coordinate position of the target object in the target frame image and the predicted coordinate positions of the recognized objects in the target frame image, then according to the coordinate position of the target object in the target frame image and the movement trajectory information of the matched recognized object, the movement trajectory information of the target object is obtained.

[0171] If the matching is successful, the feature information of the recognized object is updated according to the feature information of the target object, the predicted coordinate position in the movement trajectory information of the recognized object is updated according to the coordinate position of the target object in the target frame image, and the predicted coordinate position of the target object in the predicted image is predicted according to the coordinate positions of the target object in the reference image and the target frame image, so as to obtain the updated movement trajectory information of the target object.

[0172] Applying the image recognition method provided by the embodiment of the present application, the target frame image currently processed in the target video sequence is determined, the target frame image is input into the first recognition model, and the first position information of the target sliding sub-image output after the first recognition model performs sliding window slicing processing on the target frame image in the target frame image is obtained. Then, the target frame image and the first position information are input into the second recognition model, and the coordinate positions of the target objects in the target frame image output after the second recognition model performs anchor box slicing processing on the target frame image based on the first position information are obtained.

[0173] In the embodiments of the present application, first, the area position where the target object may exist is determined based on sliding window image slicing, and then the area position is accurately recognized based on anchor box image slicing, which improves the accuracy of image recognition and reduces the computational amount in the process of image recognition. The GFLv2 loss function is introduced into the model to solve the problem of inconsistent processing methods for heatmap, box size, and center offset estimation during model training and model application, as well as the uncertainty problem of the bounding box representation in traditional methods. In addition, by using the uncertainty statistic of the bounding box to efficiently guide the positioning quality estimation, the detection performance of the model can be better improved. The WBF method is introduced into the model to fuse the repeated or incomplete detection boxes that appear when merging the detection results of the target anchor box sub-images into the target frame image. Compared with the traditional NMS processing method, the WBF method can obtain better target box confidence and coordinate information. Secondary matching is performed based on the feature information and coordinate position of the target object, effectively improving the success rate of multi-target matching and being applicable to multi-target trajectory matching in large scenes.

[0174] In the embodiments of the present application, a step of generating mobile trajectory information is provided, including:

[0175] 1. Calculate the cost matrix of the ID Embeddings of each target object in the target frame image and the appearance features of each recognized object in the mobile trajectory information Tracklets of the reference image;

[0176] 2. Use the Kalman filter to predict the mean and covariance of the mobile trajectory information in the target frame image based on the mobile trajectory information of the reference image;

[0177] 3. Calculate the Mahalanobis distance between the predicted mobile trajectory information and the actual detection results Detections of the target frame image, and assign the appearance cost matrix greater than the pre-set distance threshold to infinity;

[0178] 4. Use the Hungarian algorithm to match the targets and trajectories:

[0179] Set the matched trajectory as the trajectory of the target frame image, and update its appearance features and the mean and variance parameters of the Kalman filter;

[0180] For the unmatched trajectories, continue the next step of matching;

[0181] 5. Calculate the CIOU of the unmatched detections and trajectories in the previous step, and use the Hungarian algorithm for matching again:

[0182] Set the matched trajectory as the trajectory of the target frame image, and similarly update its appearance features and the mean and variance parameters of the Kalman filter;

[0183] Count the unmatched trajectories into UnactivatedTracklets;

[0184] 6. Delete the UnactivatedTracklets that have not been matched for 15 consecutive frames.

[0185] The appearance feature of the trajectory adopts a feature smoothing update method:

[0186] feat = feat / l2(feat)

[0187]

[0188] smooth_feat = smooth_feat / l2(smooth_feat)

[0189] In the formula, feat is the Embedding of the current target, l2(·) is the L2 norm, smooth_feat is the appearance feature after feature smoothing, and α is the feature smoothing coefficient.

[0190] Considering that after frame extraction from a high-frame-rate target video sequence, when the same target object in two adjacent frames is sent into the network for detection and tracking, there is a larger displacement difference and a smaller CIOU. Therefore, the parameters of the Kalman filter are increased here, and the threshold of the Hungarian algorithm during CIOU Matching is decreased;

[0191] Secondly, the coverage range of large-scene images is wide, and there are differences such as illumination, resulting in a large difference in the appearance of the same target at different times. Therefore, the weight value of the trajectory template update is decreased here, and the threshold of the Hungarian algorithm for Cosine Metric is increased;

[0192] Finally, after the above optimization process, there may still be a problem that the correct detection box of the target frame image cannot be matched with the trajectory. Therefore, the detection boxes that are detected for the first time but cannot be matched with the previous frame trajectory are set to the activated state here.

[0193] Based on the problems existing in pedestrian target tracking in the low-frame-rate images after image extraction, the embodiments of this application optimize the Kalman filter parameters, the matching algorithm threshold of CIOU matching, the trajectory template update weight, and the matching algorithm threshold of Cosine Metric, reasonably adjust the intermediate state of the detection box and the trajectory, and smooth the trajectory features, which is more suitable for multi-pedestrian target tracking in low-frame-rate large-scene images.

[0194] The embodiments of this application provide a training step for a second recognition model, as Figure 5 shown, including:

[0195] Step S501: Obtain multiple sample frame images in a sample video sequence. The sample frame images include at least one sample object and a sample label corresponding to the sample object. The sample label includes the coordinate position of the sample object in the sample frame image.

[0196] Step S502: Input the sample frame images into the initial recognition model, and obtain the initial coordinate positions of each sample object in the sample frame image after the initial recognition model performs anchor box cropping processing on the sample frame images. The initial coordinate positions are determined by the heatmap detection module, center offset detection module, and box size detection module in the initial recognition model based on the GeneralFocal Loss V2 loss calculation method.

[0197] Step S503: According to the initial coordinate positions output by the initial recognition model and the coordinate positions of the sample objects in the sample frame image, determine the loss value of the General Focal Loss V2 loss calculation method, and iteratively train the initial recognition model based on the loss value until the loss value meets the training end condition, then stop training and determine the initial recognition model at the end of training as the second recognition model.

[0198] Perform annotation processing on the sample frame images, then perform anchor box cropping processing on the sample frame images according to the annotation information, and use the Encoder-Decoder network for feature extraction and multi-layer feature fusion. Among them, the high-resolution feature map extracted by the Encoder-Decoder network is used for four detection modules, namely the heatmap module, boxsize module, center offset module for target object detection, and the Embeddings module for target object Re-ID. The model training mainly minimizes the loss functions of the above four detection modules through multiple iterations, that is, to obtain the optimal multi-target object trajectory detection model. The loss calculations for target object detection and target object Re-ID are respectively implemented by General Focal Loss V2 and CrossEntropy Loss.

[0199] The first recognition model and the second recognition model can aggregate the same neural network model, or be presented as two sub-models in the same model. The first recognition model and the second recognition model can be synchronously trained. In other words, the first recognition model can also be implemented based on the above training steps.

[0200] An image recognition device is provided in an embodiment of the present application, as Figure 6 shown, including:

[0201] Determination module 601, configured to determine a target frame image being currently processed in a target video sequence, where the target frame image includes at least one target object;

[0202] First recognition module 602, configured to input the target frame image into a first recognition model, and obtain first position information of a target sliding sub-image in the target frame image output after the first recognition model performs a sliding window slicing process on the target frame image, where the target sliding sub-image is a sliding sub-image including at least one target object;

[0203] Second recognition module 603, configured to input the target frame image and the first position information into a second recognition model, and obtain coordinate positions of each target object in the target frame image output after the second recognition model performs an anchor box slicing process on the target frame image based on the first position information.

[0204] In a preferred embodiment of the present application, the apparatus further includes:

[0205] Feature determination module, configured to determine feature information of each target object in the target frame image according to the coordinate positions of each target object in the target frame image;

[0206] Reference determination module, configured to determine feature information and movement trajectory information of an identified object in a reference image;

[0207] Trajectory determination module, configured to determine movement trajectory information of each target object according to the feature information of each target object, the coordinate positions of each target object in the target frame image, the feature information of the identified object, and the movement trajectory information;

[0208] Wherein, the reference image is a frame image in the target video sequence before the target frame image; the movement trajectory information of the identified object includes the coordinate position of the identified object in the corresponding reference image, and the predicted coordinate position of the identified object in the target frame image predicted based on the coordinate position of the identified object in the reference image.

[0209] In a preferred embodiment of the present application, the trajectory determination module is specifically configured to:

[0210] For any target object, if it is determined that there is an identified object matching the target object according to the feature information of the target object and the feature information of each identified object, then the movement trajectory information of the target object is obtained according to the coordinate position of the target object in the target frame image and the movement trajectory information of the matching identified object.

[0211] In a preferred embodiment of the present application, the movement trajectory information of the target object includes the predicted coordinate position of the target object in a predicted image; the predicted image is a frame image in the target video sequence after the target frame image.

[0212] In a preferred embodiment of the present application, the trajectory determination module is further configured to:

[0213] If it is determined, according to the feature information of the target object and the feature information of each recognized object, that there is no recognized object matching the target object, then match the coordinate position of the target object in the target frame image with the predicted coordinate positions of each recognized object in the target frame image;

[0214] If it is determined, according to the coordinate position of the target object in the target frame image and the predicted coordinate positions of each recognized object in the target frame image, that there is a recognized object matching the target object, then obtain the movement trajectory information of the target object according to the coordinate position of the target object in the target frame image and the movement trajectory information of the matched recognized object.

[0215] In a preferred embodiment of the present application, the training steps of the second recognition model 603 are as follows:

[0216] Obtain multiple sample frame images in a sample video sequence. The sample frame images include at least one sample object and a sample label corresponding to the sample object. The sample label includes the coordinate position of the sample object in the sample frame image;

[0217] Input the sample frame images into the initial recognition model, and obtain the initial coordinate positions of each sample object in the sample frame image output after the initial recognition model performs anchor box cropping processing on the sample frame images; the initial coordinate positions are determined by the heat map detection module, the center offset detection module, and the box size detection module in the initial recognition model based on the General Focal Loss V2 loss calculation method;

[0218] According to the initial coordinate positions output by the initial recognition model and the coordinate positions of the sample objects in the sample frame images, determine the loss value of the General Focal Loss V2 loss calculation method, and perform iterative training on the initial recognition model based on the loss value until the loss value meets the training end condition, then stop training and determine the initial recognition model at the end of training as the second recognition model.

[0219] In a preferred embodiment of the present application, the second determination module includes:

[0220] A cropping module, configured to determine the anchor boxes corresponding to each target object obtained after the target frame image is subjected to anchor box cropping processing;

[0221] The detection module is used to determine the feature maps of the anchor boxes corresponding to each target object, and input the feature maps into the heatmap detection module, the center offset detection module, and the box size detection module of the second recognition model respectively, to obtain the center coordinate positions, center offset data of each target object in the corresponding feature map, and the height and width of the anchor boxes corresponding to each target object;

[0222] The second determination sub-module is used to update the center coordinate positions of each target object in the corresponding feature map according to the center offset data of each target object in the corresponding feature map, and determine the coordinate positions of each target object in the target frame image according to the updated center coordinate positions of each target object in the corresponding feature map and the height and width of the anchor boxes corresponding to each target object.

[0223] In a preferred embodiment of the present application, the General Focal Loss V2 loss calculation method includes a quality coordination loss calculation method;

[0224] Specifically, the detection module is used for:

[0225] Input the feature map into the heatmap detection module, and determine the heatmap detection loss of the feature map based on the quality coordination loss calculation method;

[0226] According to the heatmap detection loss, perform heatmap detection on the feature map to determine the center coordinate positions of each target object in the corresponding feature map.

[0227] In a preferred embodiment of the present application, the General Focal Loss V2 loss calculation method includes a distribution coordination loss calculation method;

[0228] The detection module is further used for:

[0229] Input the feature map into the center offset detection module, and determine the center offset detection loss of the feature map based on the distribution coordination loss calculation method;

[0230] According to the center offset detection loss, perform center offset detection on the feature map to determine the center offset data of each target object in the corresponding feature map.

[0231] In a preferred embodiment of the present application, the detection module is further used for:

[0232] Input the feature map into the box size detection module, and determine the box size detection loss of the feature map based on the distribution coordination loss calculation method;

[0233] According to the box size detection loss, perform box size detection on the feature map to determine the height and width of the anchor boxes corresponding to each target object.

[0234] Apply the image recognition device provided in the embodiments of the present application to determine the target frame image being currently processed in the target video sequence, input the target frame image into the first recognition model, obtain the first position information of the target sliding sub-image output after the first recognition model performs sliding window slicing on the target frame image in the target frame image, and then input the target frame image and the first position information into the second recognition model to obtain the coordinate positions of each target object in the target frame image after the second recognition model performs anchor box slicing on the target frame image based on the first position information.

[0235] In the embodiments of the present application, the area position where target objects may exist is first determined based on sliding window slicing, and then the area position is accurately recognized based on anchor box slicing, which improves the accuracy of image recognition and reduces the computational amount in the image recognition process.

[0236] In the embodiments of the present application, an electronic device (computer device / equipment / system) is provided, including a memory, a processor, and a computer program stored on the memory. The processor executes the above computer program to implement the steps of the image recognition method. Compared with the related art, it can be realized that: the area position where target objects may exist is first determined based on sliding window slicing, and then the area position is accurately recognized based on anchor box slicing, which improves the accuracy of image recognition and reduces the computational amount in the image recognition process.

[0237] In an optional embodiment, an electronic device is provided, as Figure 7 shown Figure 7 The electronic device 7000 shown includes: a processor 7001 and a memory 7003. Among them, the processor 7001 and the memory 7003 are connected, such as connected through a bus 7002. Optionally, the electronic device 7000 may further include a transceiver 7004, and the transceiver 7004 can be used for data interaction between the electronic device and other electronic devices, such as data sending and / or data receiving, etc. It should be noted that in practical applications, the transceiver 7004 is not limited to one, and the structure of the electronic device 7000 does not constitute a limitation to the embodiments of the present application.

[0238] The processor 7001 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. It can implement or execute various exemplary logical blocks, modules, and circuits described in connection with the disclosure of this application. The processor 7001 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0239] The bus 7002 may include a path for transmitting information between the above components. The bus 7002 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus, etc. The bus 7002 may be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, Figure 7 only a thick line is used to represent it in the figure, but it does not mean that there is only one bus or one type of bus.

[0240] The memory 7003 may be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or it may also be an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory), or other optical disc storage, optical disc storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), magnetic disk storage media, other magnetic storage devices, or any other medium that can be used to carry or store computer programs and can be read by a computer, which is not limited herein.

[0241] The memory 7003 is used to store the computer program for implementing the embodiments of the present application, and is controlled by the processor 7001 to execute. The processor 7001 is used to execute the computer program stored in the memory 7003 to implement the steps shown in the foregoing method embodiments.

[0242] The embodiments of the present application provide a computer-readable storage medium, on which a computer program is stored. When the computer program is executed by a processor, the steps and corresponding contents of the foregoing method embodiments can be implemented.

[0243] The embodiments of the present application further provide a computer program product, including a computer program. When the computer program is executed by a processor, the steps and corresponding contents of the foregoing method embodiments can be implemented.

[0244] The terms "first", "second", "third", "fourth", "1", "2", etc. (if any) in the specification, claims and the above drawings of the present application are used to distinguish similar objects, and do not necessarily describe a specific order or sequence. It should be understood that such data can be interchanged under appropriate circumstances so that the embodiments of the present application described herein can be implemented in an order other than that shown or described in words.

[0245] It should be understood that although the flowchart of the embodiments of the present application indicates each operation step by an arrow, the execution order of these steps is not limited to the order indicated by the arrow. Unless otherwise clearly stated in this application, in some implementation scenarios of the embodiments of the present application, the implementation steps in each flowchart can be executed in other orders according to requirements. In addition, some or all of the steps in each flowchart may include multiple sub-steps or multiple stages based on the actual implementation scenario. Some or all of these sub-steps or stages can be executed at the same time, and each sub-step or stage of these sub-steps or stages can also be executed at different times. In the scenario where the execution times are different, the execution order of these sub-steps or stages can be flexibly configured according to requirements, and the embodiments of the present application do not limit this.

[0246] The above are only optional implementation manners of some implementation scenarios of the present application. It should be noted that for those of ordinary skill in the art, without departing from the technical concept of the solution of the present application, other similar implementation means based on the technical idea of the present application also belong to the protection scope of the embodiments of the present application.

Claims

1. An image recognition method, characterized in that, Including: Determine the target frame image being currently processed in the target video sequence, where the target frame image includes at least one target object; Input the target frame image into a first recognition model, and obtain the first position information of the target sliding window sub-image output by the first recognition model after performing sliding window slicing on the target frame image. The target sliding window sub-image is a sliding window sub-image including at least one target object; the target sliding window sub-image undergoes boundary fusion processing so that the same target object appears only in one target sliding window sub-image; Input the target frame image and the first position information into a second recognition model, and obtain the coordinate positions of each target object in the target frame image output by the second recognition model after performing anchor box slicing on the target frame image based on the first position information; Wherein, the coordinate position of each target object is determined by the following method: Determine the regional positions of each target object in the target frame image, and determine the anchor boxes of each target object based on the regional positions; For each target object, expand the anchor box of the target object to a preset size and then slice it to obtain a target anchor box sub-image containing the anchor box, and perform recognition on the target anchor box sub-image to determine the coordinate position of the target object in the target frame image.

2. The image recognition method according to claim 1, wherein The method further includes: Determine the feature information of each target object in the target frame image according to the coordinate positions of each target object in the target frame image; Determine the feature information and movement trajectory information of the recognized objects in the reference image; Determine the movement trajectory information of each target object according to the feature information of each target object, the coordinate positions of each target object in the target frame image, the feature information of the recognized objects, and the movement trajectory information; Wherein, the reference image is a frame image in the target video sequence before the target frame image; the movement trajectory information of the recognized objects includes the coordinate positions of the recognized objects in the corresponding reference image, and the predicted coordinate positions of the recognized objects in the target frame image predicted based on the coordinate positions of the recognized objects in the reference image.

3. The image recognition method according to claim 2, characterized in that, The determining the movement trajectory information of each target object according to the feature information of each target object, the coordinate positions of each target object in the target frame image, the feature information of the recognized objects, and the movement trajectory information includes: For any one target object, if it is determined that there is a recognized object matching the target object according to the feature information of the target object and the feature information of each recognized object, then obtain the movement trajectory information of the target object according to the coordinate position of the target object in the target frame image and the movement trajectory information of the matching recognized object.

4. The image recognition method according to claim 3, wherein The movement trajectory information of the target object includes the predicted coordinate position of the target object in the predicted image; the predicted image is a frame image in the target video sequence after the target frame image.

5. The image recognition method according to claim 2, wherein Determining the movement trajectory information of each target object based on the feature information of each target object, the coordinate positions of each target object in the target frame image, the feature information of the identified objects, and the movement trajectory information further includes: If it is determined, based on the feature information of the target object and the feature information of each identified object, that there is no identified object matching the target object, then match the coordinate position of the target object in the target frame image with the predicted coordinate positions of each identified object in the target frame image; If it is determined, based on the coordinate position of the target object in the target frame image and the predicted coordinate positions of each identified object in the target frame image, that there is an identified object matching the target object, then obtain the movement trajectory information of the target object based on the coordinate position of the target object in the target frame image and the movement trajectory information of the matching identified object.

6. The image recognition method according to claim 1, characterized in that, The training steps of the second recognition model are as follows: Obtain multiple sample frame images in the sample video sequence, where each sample frame image includes at least one sample object and the sample label corresponding to the sample object, and the sample label includes the coordinate position of the sample object in the sample frame image; Input the sample frame image into the initial recognition model to obtain the initial coordinate positions of each sample object in the sample frame image output after the initial recognition model performs anchor box cropping on the sample frame image; the initial coordinate positions are determined by the heat map detection module, the center offset detection module, and the box size detection module in the initial recognition model based on the General Focal Loss V2 loss calculation method; Determine the loss value of the General Focal Loss V2 loss calculation method based on the initial coordinate positions output by the initial recognition model and the coordinate positions of the sample objects in the sample frame image, and perform iterative training on the initial recognition model based on the loss value until the loss value meets the training end condition, then stop training and determine the initial recognition model at the end of training as the second recognition model.

7. The image recognition method according to claim 6, characterized in that The step of inputting the target frame image and the first position information into the second recognition model to obtain the coordinate positions of each target object in the target frame image output after the second recognition model performs anchor box cropping on the target frame image based on the first position information includes: Determine the anchor boxes corresponding to each target object obtained after the target frame image is subjected to the anchor box cropping process; Determine the feature maps of the anchor boxes corresponding to each target object, and input the feature maps into the heat map detection module, the center offset detection module, and the box size detection module of the second recognition model respectively to obtain the center coordinate positions, center offset data of each target object in the corresponding feature maps, and the height and width of the anchor boxes corresponding to each target object; Update the center coordinate positions of each target object in the corresponding feature map according to the center offset data of each target object in the corresponding feature map, and determine the coordinate positions of each target object in the target frame image according to the updated center coordinate positions of each target object in the corresponding feature map and the height and width of the anchor box corresponding to each target object.

8. The image recognition method according to claim 7, wherein, The General Focal Loss V2 loss calculation method includes a quality coordination loss calculation method; The step of inputting the feature map into the heat map detection module, the center offset detection module, and the box size detection module of the second recognition model respectively to obtain the center coordinate positions, center offset data, and the height and width of the anchor box corresponding to each target object in the corresponding feature map includes: Input the feature map into the heat map detection module, and determine the heat map detection loss of the feature map based on the quality coordination loss calculation method; Perform heat map detection on the feature map according to the heat map detection loss to determine the center coordinate positions of each target object in the corresponding feature map.

9. The image recognition method according to claim 8, characterized in that, The General Focal Loss V2 loss calculation method includes a distribution coordination loss calculation method; The step of inputting the feature map into the heat map detection module, the center offset detection module, and the box size detection module of the second recognition model respectively to obtain the center coordinate positions, center offset data, and the height and width of the anchor box corresponding to each target object in the corresponding feature map further includes: Input the feature map into the center offset detection module, and determine the center offset detection loss of the feature map based on the distribution coordination loss calculation method; Perform center offset detection on the feature map according to the center offset detection loss to determine the center offset data of each target object in the corresponding feature map.

10. The image recognition method according to claim 9, wherein, The step of inputting the feature map into the heat map detection module, the center offset detection module, and the box size detection module of the second recognition model respectively to obtain the center coordinate positions, center offset data, and the height and width of the anchor box corresponding to each target object in the corresponding feature map further includes: Input the feature map into the box size detection module, and determine the box size detection loss of the feature map based on the distribution coordination loss calculation method; Perform box size detection on the feature map according to the box size detection loss to determine the height and width of the anchor box corresponding to each target object.

11. An image recognition device, characterized in that, It includes: A determination module for determining the target frame image being currently processed in the target video sequence, where the target frame image includes at least one target object; A first recognition module for inputting the target frame image into a first recognition model to obtain the first position information of the target sliding sub-image in the target frame image output after the first recognition model performs sliding window slicing on the target frame image, where the target sliding sub-image is a sliding sub-image including at least one target object; the target sliding sub-image undergoes boundary fusion processing so that the same target object appears only in one target sliding sub-image; A second recognition module, configured to input the target frame image and the first position information into a second recognition model, and obtain the coordinate positions of each target object in the target frame image output after the second recognition model performs anchor box slicing processing on the target frame image based on the first position information; Specifically, the second recognition module is configured to: Determine the regional positions of each target object in the target frame image, and determine the anchor boxes of each target object based on the regional positions; For each target object, expand the anchor box of the target object to a preset size and then slice it to obtain a target anchor box sub-image containing the anchor box, and recognize the target anchor box sub-image to determine the coordinate position of the target object in the target frame image.

12. An electronic device, characterized in that, The electronic device includes: One or more processors; A memory; One or more applications, where the one or more applications are stored in the memory and configured to be executed by the one or more processors, and the one or more programs are configured to: execute the image recognition method according to any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the image recognition method according to any one of claims 1 to 10.

Images