Object recognition method, device, server and computer-readable storage medium

Abstract

The embodiment of the present invention provides an object recognition method, device, server and computer-readable storage medium, wherein the method comprises: obtaining a feature mapping set of each frame of N frames of pictures included in the collected video data, the feature mapping set being used to indicate the features of the object included in each frame of the picture; determining a target convolution set of the N frames of pictures and auxiliary data of the object in each frame of the picture according to the feature mapping set of each frame of the picture and a recorded reference feature mapping set, the auxiliary data being used to indicate the image area corresponding to the object in the picture, the reference feature mapping set being a convolution set of a previous batch of N frames of pictures processed, the reference feature mapping set being used to indicate the morphological features of the object; performing classification and regression processing on the auxiliary data of the object and the target convolution set to obtain category indication data and corresponding position data of each object included in the N frames of the picture, thereby improving the efficiency of object recognition in the video.

Description

Technical Field

[0001] The present invention relates to the field of computer technology, and in particular to an object recognition method, device, server and computer-readable storage medium. Background Art

[0002] At present, in order to accurately identify the corresponding target object in the image, the instance segmentation mask-rcnn technology can be used. This technology can accurately identify the target object in the image, but this technology only processes pictures and cannot recognize videos. The Long Short-Term Memory (LSTM) technology can recognize the target object in the video, but the implementation of this technology depends on the processing of the previous frame of the picture frame, and can only process the picture frame by frame, which greatly affects the efficiency of object recognition in the video. Therefore, how to improve the efficiency of object recognition in the video is a problem that needs to be solved at present. Summary of the invention

[0003] The embodiments of the present invention provide an object recognition method, device, server and computer-readable storage medium, which can process multiple frames of images included in video data in parallel and improve the efficiency of object recognition in the video.

[0004] A first aspect of an embodiment of the present invention provides an object recognition method, including:

[0005] Acquire a feature mapping set of each frame of N frames of images included in the collected video data, where the feature mapping set is used to indicate features of an object included in each frame of the image, where N is an integer greater than or equal to 1;

[0006] Determining a target convolution set of the N frames of images and auxiliary data of objects in each frame of images according to the feature mapping set of each frame of images and the recorded reference feature mapping set, wherein the auxiliary data is used to indicate an image area corresponding to the object in the image, and the reference feature mapping set is a convolution set of a previous batch of N frames of images processed, and the reference feature mapping set is used to indicate morphological features of the object;

[0007] The auxiliary data of the object and the target convolution set are subjected to classification regression processing to obtain category indication data and corresponding position data of each object included in the N frames of pictures.

[0008] A second aspect of an embodiment of the present invention provides an object recognition device, including:

[0009] An acquisition module, used to acquire a feature mapping set of each frame of N frames of images included in the collected video data, wherein the feature mapping set is used to indicate the features of an object included in each frame of the image, and N is an integer greater than or equal to 1;

[0010] A determination module, configured to determine a target convolution set of the N frames of images and auxiliary data of an object in each frame of images according to a feature mapping set of each frame of images and a recorded reference feature mapping set, wherein the auxiliary data is used to indicate an image area corresponding to the object in the image, and the reference feature mapping set is a convolution set of a previous batch of N frames of images processed, and the reference feature mapping set is used to indicate morphological features of the object;

[0011] A processing module is used to perform classification regression processing on the auxiliary data of the object and the target convolution set to obtain category indication data and corresponding position data of each object included in the N frames of pictures.

[0012] A third aspect of an embodiment of the present invention provides a server, comprising a processor, a network interface and a storage device, wherein the processor, the network interface and the storage device are interconnected, wherein the network interface is controlled by the processor for sending and receiving data, the storage device is used to store a computer program, the computer program includes program instructions, and the processor is configured to call the program instructions to execute the object recognition method described in the first aspect above.

[0013] A fourth aspect of the embodiments of the present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores program instructions, and when the program instructions are executed, the object recognition method described in the first aspect is implemented.

[0014] A fifth aspect of the embodiments of the present invention provides a computer program product or a computer program, the computer program product or the computer program including computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the object recognition method described in the first aspect above.

[0015] In an embodiment of the present invention, the server obtains a feature mapping set of each frame of N frames included in the collected video data, and determines a target convolution set of N frames and auxiliary data of objects in each frame according to the feature mapping set of each frame and a recorded reference feature mapping set, wherein the auxiliary data is used to indicate an image area corresponding to the object in the picture, and the reference feature mapping set is a convolution set of a previous batch of N frames of pictures processed, and the reference feature mapping set is used to indicate morphological features of the object, and then the auxiliary data of the object and the target convolution set are subjected to classification and regression processing to obtain category indication data and corresponding position data of each object included in the N frames of pictures, so that objects in multiple frames of pictures in the video can be identified in parallel, thereby improving the efficiency of object recognition in the video. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0017] Figure 1a It is a structural diagram of an instance segmentation mask-rcnn provided by an embodiment of the present invention;

[0018] Figure 1b It is a schematic diagram of the structure of an LSTM provided by an embodiment of the present invention;

[0019] Figure 2 is a structural schematic diagram of an object recognition model provided by an embodiment of the present invention;

[0020] Figure 3 is a flow chart of an object recognition method provided by an embodiment of the present invention;

[0021] Figure 4a is a schematic diagram of an input frame of an image provided by an embodiment of the present invention;

[0022] Figure 4b is a schematic diagram of another frame of picture input provided by an embodiment of the present invention;

[0023] Figure 5 is a flow chart of another object recognition method provided by an embodiment of the present invention;

[0024] Figure 6 is a structural schematic diagram of an object recognition device provided by an embodiment of the present invention;

[0025] Figure 7 It is a structural diagram of a server provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0027] At present, instance segmentation mask-rcnn technology and LSTM technology are used for object recognition in images and videos respectively. The process of object recognition in images using the instance segmentation mask-rcnn technology is as follows: Figure 1a As shown in the figure, each frame of the image is processed through the skeleton network, convolution kernel, candidate set network, classification regression, etc., so as to realize the object recognition in the image. The classification regression can be semantic segmentation, bounding box prediction and classification, but this technology cannot be used for object recognition in the video. The LSTM technology can be used to realize the process of object recognition in the video. The object recognition process is as follows Figure 1b As shown in the figure, each time a frame of image is input, the frame of image is processed by the embedding layer, and then processed by each hidden layer and the fully connected layer, and finally the object recognition in the video can be realized through classification processing, but Figure 1b It can be seen that the processing of each frame of the image needs to rely on the processing result of the previous hidden layer before the processing of the next hidden layer can be performed. For example, the processing of hidden layer 12 needs to rely on the processing result of hidden layer 11. Therefore, when recognizing objects in the video, it is necessary to recognize them frame by frame, which leads to low efficiency in object recognition in the video.

[0028] Based on the defects of the above-mentioned instance segmentation mask-rcnn technology and LSTM technology, an embodiment of the present invention provides an object recognition method, which can process multiple frames of pictures included in a video in parallel, thereby realizing the recognition of objects in the video and improving the efficiency of object recognition. In a specific implementation, a feature mapping set of each frame of the N frames of pictures included in the collected video data can be first obtained, and the target convolution set of the N frames of pictures and the auxiliary data of the objects in each frame of the picture can be determined according to the feature mapping set of each frame of the picture and the recorded reference feature mapping set, and the auxiliary data of the object and the target convolution set are classified and regressed to obtain the category indication data and corresponding position data of each object included in the N frames of the picture. Among them, the auxiliary data is used to indicate the image area corresponding to the object in the picture, the reference feature mapping set is the convolution set of the previous batch of N frames of pictures processed, and the reference feature mapping set is used to indicate the morphological features of the object.

[0029] See also Figure 2 , Figure 2 A schematic diagram of the structure of an object recognition model provided by an embodiment of the present invention. The object recognition model may be a modified mask-rcnn model, such as Figure 2 As shown, the object recognition model includes an input module 201, a skeleton network 202, a processing module 203, and a training module 204, wherein:

[0030] The input module 201 can obtain N frames of images included in the collected video data, and input the N frames of images in parallel into the skeleton network 202. The skeleton network 202 can process the input N frames of images to obtain a feature mapping set of each frame of the N frames of images. The processing module 203 can calculate the N frames of images to obtain the weight of each frame of images to determine the importance of the corresponding object features in each frame of images. Further, the processing module 203 can also determine the feature vector set of each frame of images and the reference feature mapping set of the previous batch of N frames of images to perform similarity calculation, and obtain the similarity of each frame of images, determine the weight of each frame of images according to the similarity of each frame of images, and perform weighted average of the weight of each frame of images and the feature mapping set of each frame of images, so as to obtain an average feature mapping set, which can also be called a convolution set, and the convolution set is saved and used as a reference feature mapping set of the next batch of N frames of images. The training module 204 can put the convolution set into the candidate set network for training, as well as classification regression, and then obtain the category indication data and corresponding position data of the objects in the N frames of images.

[0031] See also Figure 3 , Figure 3 A flowchart of an object recognition method provided by an embodiment of the present invention. The object recognition method described in this embodiment can be executed by a server, and the server can execute any module in the above object recognition model, including:

[0032] 301. Obtain a feature mapping set of each frame of N frames of images included in the collected video data.

[0033] The feature mapping set is used to indicate the features of the object included in each frame of the picture, N is an integer greater than or equal to 1, and the feature mapping set may refer to a nested set of multidimensional arrays, such as a nested set of three-dimensional arrays, and the values ​​in the nested set of multidimensional arrays may be used to indicate the features of the object included in each frame of the picture. Alternatively, the feature mapping set for each frame of the picture includes a feature vector of the object, and the feature vector is used to indicate the features of the object included in each frame of the picture.

[0034] In a feasible embodiment, the above-mentioned N frames of pictures can be N frames of unprocessed pictures obtained from the collected video data, or the above-mentioned N frames of pictures can be obtained by the server first obtaining N frames of unprocessed pictures from the collected video data and pre-processing the unprocessed N frames of pictures.

[0035] In a feasible embodiment, the server collects video data, divides the video data into original N frames of pictures, and preprocesses each of the original N frames of pictures to obtain N frames of pictures included in the video data, wherein the preprocessing includes grayscale processing of each original frame of pictures and padding processing of the size of each frame of pictures after grayscale processing. The division of the video data into the original N frames of pictures can be set according to user needs, for example, the video data can be divided into 30 frames of pictures.

[0036] In a feasible embodiment, the server can obtain N frames of pictures included in the collected video data, and input the N frames of pictures into the skeleton network to perform convolution, pooling, activation and other operations in sequence, thereby obtaining a feature mapping set for each frame of the N frames of pictures.

[0037] 302. Determine a target convolution set of N frames of images and auxiliary data of objects in each frame of images according to a feature mapping set of each frame of images and a recorded reference feature mapping set.

[0038] Among them, the auxiliary data is used to indicate the image area corresponding to the object in the picture, the reference feature mapping set is a convolution set of the previous batch of N frames of pictures processed, and the reference feature mapping set is used to indicate the morphological features of the object. For example, the object can be a pig, then the reference feature mapping set is used to indicate the morphological features of the pig.

[0039] Specifically, the server determines the target convolution set based on the feature mapping set of each frame and the convolution set of the previous batch of N frames of images, and trains the target convolution set in the candidate set network to determine the auxiliary data of the object in each frame of the image. It can be understood that training the target convolution set in the candidate set network can remove the background data in each frame of the image and the redundant candidate frame data of the object, thereby determining the auxiliary data of the object in each frame of the image.

[0040] In a feasible embodiment, if the server obtains a feature mapping set of each frame of N frames of images included in the collected video data for the first time, the reference feature mapping set recorded above can be pre-set with a value, for example, the reference feature mapping set can be set to 1.

[0041] In a feasible embodiment, the server determines the target convolution set of N frames of pictures based on the feature mapping set of each frame of pictures and the recorded reference feature mapping set by comparing the feature mapping set of each frame of pictures with the recorded reference feature mapping set to determine the similarity between each frame of pictures and the reference feature mapping set; after determining the similarity of each frame of pictures, determine the weight of each frame of pictures according to the similarity of each frame of pictures; and determine the target convolution set according to the weight of each frame of pictures and the feature mapping set of each frame of pictures.

[0042] 303. Perform classification regression processing on the auxiliary data of the object and the target convolution set to obtain category indication data and corresponding position data of each object included in the N frames of pictures.

[0043] The category indication data of each object can be preset, for example, the category indication data of each object can be set when each object is framed in each frame, and a number is set to represent a pig or a person when the object is framed, such as the number 1 represents a pig, etc., or the category indication data of each object can be the contour information of each object. The position data refers to a certain position of the object, which can be represented by coordinates.

[0044] In a feasible embodiment, the server obtains two frames of pictures, namely Figure 4a and Figure 4b , after being processed by the above object recognition model and object recognition method, pigs can be recognized, thereby improving the efficiency of object recognition in the video. The output result is:

[0045] Category indicator data: [1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1 1], 1 represents the target is identified as a pig. Indicates that 23 pigs were identified.

[0046] Pig's coordinates: [[1233 480 1331 724],[312 462 461 626],[144 772 268935],[682 74 796 207],[528 228 670 321],[380 286 529 407],[749 158 817 326],[338 294 418 462],[274 469 359 648],[438 139 560 206],[798 126 984 201],[10457 1098 117],[994 0 1051 107],[775 15 886 90],[1190 4 1271 76],[873 18 97198],[357 185 459 276],[515 34 637 83],[547 83 625 150],[634 43 696 123],[938110 1077 169],[421 199 527 285],[1124 10 1180 103]]

[0047] Among them, [1233 480 1331 724] represents the two coordinate positions (upper left corner coordinate and lower right corner coordinate) of one of the pigs, that is, y1, x1, y2, x2.

[0048] In an embodiment of the present invention, the server obtains a feature mapping set of each frame of N frames included in the collected video data, and determines a target convolution set of N frames and auxiliary data of objects in each frame according to the feature mapping set of each frame and a recorded reference feature mapping set, wherein the auxiliary data is used to indicate an image area corresponding to the object in the picture, and the reference feature mapping set is a convolution set of a previous batch of N frames of pictures processed, and the reference feature mapping set is used to indicate morphological features of the object, and then the auxiliary data of the object and the target convolution set are subjected to classification and regression processing to obtain category indication data and corresponding position data of each object included in the N frames of pictures, so that objects in multiple frames of pictures in the video can be identified in parallel, thereby improving the efficiency of object recognition in the video.

[0049] See also Figure 5 , Figure 5 A flowchart of another object recognition method provided by an embodiment of the present invention. The object recognition method described in this embodiment can be executed by a server, and the server can execute any module in the above object recognition model, including:

[0050] 501. Obtain a feature mapping set of each frame of N frames of images included in the collected video data.

[0051] In a feasible embodiment, the server obtains N frames of images included in the collected video data, and inputs each of the N frames of images into a skeleton network for convolution, pooling and activation processing, thereby obtaining a feature mapping set for each frame of images.

[0052] In a feasible embodiment, if each of the N frames of pictures is obtained through preprocessing, the server divides the collected video data into original N frames of pictures before obtaining the N frames of pictures included in the collected video data, and preprocesses each of the original N frames of pictures. In a specific implementation, each of the original frames of pictures is gray-scaled, wherein the gray-scale calculation formula of the pixel points in each of the original frames of pictures is as shown in Formula 1.1:

[0053] f(i,j)=(R(i,j)+G(i,j)+B(i,j)) / 3 Equation 1.1

[0054] Among them, f(i,j) represents the grayscale of the pixel, R(i,j) represents the change of the red color channel, G(i,j) represents the change of the green color channel, and B(i,j) represents the change of the blue color channel.

[0055] In a feasible embodiment, after grayscale processing is performed on each of the original N frames of pictures, N frames of pictures after grayscale processing are obtained, and the server fills the size of the grayscale image corresponding to each frame of the picture so that the object recognition model can adapt to pictures of different sizes. In a specific implementation, the server obtains the maximum length and maximum width of the N frames of pictures (i.e., the N frames of pictures after the grayscale processing mentioned above) included in the collected video data, and determines the maximum value of the maximum length and the maximum width, and uses the maximum value as the filling value, thereby obtaining the filling value, and fills the size of each frame of the N frames of pictures according to the filling value. It can be understood that, according to the filling value, the size of each frame of the N frames of pictures is filled, and each frame of the picture can be filled into a picture of the same size. For example, the server obtains the maximum length of the N frames of pictures included in the collected video data as a and the maximum width as b. If the maximum width a is greater than the width b, the maximum value is determined to be a, and a is used as the filling value, then the size of the filled N frames of pictures is the same.

[0056] In a feasible embodiment, further, after obtaining the filling value, the server can determine the filling rule for each frame of the picture according to the filling value. In a specific implementation, for the target picture in the N frames of pictures, the server calculates the picture zoom factor corresponding to the target picture according to the length and width of the target picture, and the server determines whether the product of the larger value of the length and width of the target picture and the picture zoom factor is greater than the filling value. If the product of the larger value of the length and width of the target picture and the picture zoom factor is greater than the filling value, the picture zoom factor is adjusted, and the size of the target picture is filled according to the adjusted picture zoom factor; if the product of the larger value of the length and width of the target picture and the picture zoom factor is less than or equal to the filling value, the size of the target picture is filled according to the picture zoom factor, wherein the target picture is any one of the N frames of pictures.

[0057] For example, taking the obtained padding value a as an example, for the target image in the N frames of images, the actual width of the target image is represented by h, and the length of the target image is represented by w. The server calculates the scaling factor of the target image, which is represented by scale. The scaling factor scale=max(1,b / min(h,w)), b is the above-mentioned maximum width. The server determines that the product of the larger value of the length w and the width h of the target image and the scaling factor is greater than the padding value a, that is, max(h,w)*scale>a, and then adjusts the image scaling factor, that is, adjusts the image scaling factor to be expressed as scale'=a / max(h,w), and obtains the processed width h'=h*scale' of the target image, and the processed length w'=w*scale' of the target image, thereby achieving padding processing of the size of the target image according to the adjusted image scaling factor.

[0058] In a feasible embodiment, the server performs padding processing on the size of the target image according to the adjusted image zoom factor in a specific implementation manner: padding is performed on the top, bottom, left and right of the target image respectively, the top padding value of the target image = int((padding value a - processed width h' of the target image) / 2), the bottom padding value of the target image = padding value a - processed width h' of the target image - top padding value, the left padding value of the target image = int((padding value a - processed length w' of the target image) / 2), the right padding value of the target image = padding value a - processed length w' of the target image - left padding value, wherein int means rounding without decimal points.

[0059] It should be noted that the features of the objects in the N frames of images after the grayscale processing and the N frames of images included in the above-mentioned collected video data have not changed. It is only for filling the images so that the object recognition model can adapt to images of different sizes.

[0060] 502. Obtain a reference feature mapping set of records.

[0061] The reference feature map set is a convolution set of the previous batch of N frames of images processed, and the reference feature map set is used to indicate the morphological features of the object. In the present invention application, the morphological features here refer to the main features that can represent the object.

[0062] 503. Determine the weight of each frame of the picture according to the feature mapping set of each frame of the picture and the reference feature mapping set.

[0063] Specifically, the server calculates the similarity between the feature mapping set of each frame of the image and the reference feature mapping set, and determines the weight of each frame of the image according to the obtained similarity of each frame of the image.

[0064] In a feasible embodiment, the server calculates the similarity between the feature mapping set of each frame of the picture and the reference feature mapping set, obtains the similarity between the feature mapping set of each frame of the picture and the reference feature mapping set, and determines the weight of each frame of the picture according to the similarity corresponding to each frame of the picture. The feature mapping set of each frame of the picture can be represented by a i Indicates that the reference feature map set can be represented by b, and the feature map set a of each frame image is i Calculate the similarity with the reference feature map set b respectively to obtain the feature map set a of each frame i The similarity with the reference feature mapping set b is calculated as shown in formula 1.2:

[0065] similarity i (ai ,b)=(a i *b i ) / ||a i ||*||b|| Formula 1.2

[0066] In a feasible embodiment, after obtaining the feature mapping set a of each frame of the picture i After comparing the similarity with the reference feature map set b, the server performs feature map set a on each frame of the image. i The similarity with the reference feature map set b is normalized to obtain the weight of each frame. The weight of each frame is calculated as shown in formula 1.3:

[0067]

[0068] Among them, softmax() is the normalization function and sum() is the summation function.

[0069] 504. Determine a target convolution set of N frames of images according to the weight and feature mapping set of each frame of images.

[0070] Specifically, the server performs weighted averaging based on the weight and feature mapping set of each frame of the image to obtain a target convolution set of N frames of images.

[0071] In a feasible embodiment, the server performs weighted processing on the weight and feature mapping set of each frame of the picture to obtain a total feature mapping set of N frames of the picture, and averages the total feature mapping set of N frames of the picture to obtain a target convolution set of N frames of the picture. The weight of each frame of the picture is represented by w i Indicates that the feature map set of each frame is X i Indicates that the target convolution set is denoted by C, then the target convolution set calculation formula for N frames of pictures is as shown in formula 1.4:

[0072] C = sum(w i *X i ) / i Formula 1.4

[0073] In a feasible embodiment, after the server determines the target convolution product set of N frames of pictures, the server will use the target convolution set to update the reference feature mapping set. In a specific implementation, the server uses the obtained target convolution set as a reference feature mapping set to determine the convolution set corresponding to the next batch of N frames of pictures.

[0074] In a feasible embodiment, after the server obtains the feature mapping set of each frame of N frames included in the collected video data for the first time, the server needs to determine the target convolution set based on the feature mapping set of each frame of N frames and the recorded reference feature mapping set. Since there is no reference feature mapping set at this time, the reference feature mapping set can be pre-set during the initial data training, and the reference feature mapping set can generally be set to 1.

[0075] 505. Use the candidate set network to process the target convolution set to obtain auxiliary data of the object in each frame.

[0076] Specifically, the server uses the candidate set network to perform the first data processing on the target convolution set, removes the background data in each frame of the picture and the redundant candidate frame data of the object, thereby obtaining the auxiliary data of the object in each frame of the picture. The auxiliary data is used to indicate the image area corresponding to the object in the picture.

[0077] 506. Perform classification regression processing on the auxiliary data of the object and the target convolution set to obtain category indication data and corresponding position data of each object included in the N frames of pictures.

[0078] Specifically, after obtaining the auxiliary data of the object in each frame of the image, the server performs secondary processing, classifies and regresses the auxiliary data of the object with the target convolution set, and obtains the category indication data and corresponding position data of each object included in the N frames of the image. Through secondary processing, objects can be identified more accurately.

[0079] In an embodiment of the present invention, a server obtains a feature mapping set of each N frame of pictures included in the collected video data and a recorded reference feature mapping set, and determines the weight of each frame of pictures according to the feature mapping set of each frame of pictures and the reference feature mapping set, and then determines the target convolution set of N frames of pictures according to the weight of each frame of pictures and the feature mapping set, and uses a candidate set network to process the target convolution set to obtain auxiliary data of objects in each frame of pictures, and then classifies and regresses the auxiliary data of the objects with the target convolution set to obtain category indication data and corresponding position data of each object included in the N frames of pictures, so that objects in multiple frames of pictures in the video can be recognized in parallel, thereby improving the efficiency of object recognition in the video.

[0080] See also Figure 6 , is a schematic diagram of the structure of an object recognition device provided by an embodiment of the present invention. The object recognition device described in this embodiment can be executed by a server, including:

[0081] An acquisition module 601 is used to acquire a feature mapping set of each frame of N frames of images included in the collected video data, wherein the feature mapping set is used to indicate the features of an object included in each frame of the image, and N is an integer greater than or equal to 1;

[0082] A determination module 602 is used to determine a target convolution set of the N frames of pictures and auxiliary data of the object in each frame of pictures according to the feature mapping set of each frame of pictures and the recorded reference feature mapping set, wherein the auxiliary data is used to indicate an image area corresponding to the object in the picture, and the reference feature mapping set is a convolution set of a previous batch of N frames of pictures processed, and the reference feature mapping set is used to indicate morphological features of the object;

[0083] The processing module 603 is used to perform classification regression processing on the auxiliary data of the object and the target convolution set to obtain category indication data and corresponding position data of each object included in the N frames of pictures.

[0084] In a feasible embodiment, the determining module 602 is specifically configured to:

[0085] Get the reference feature map set of the record;

[0086] Determine the weight of each frame of the picture according to the feature mapping set of each frame of the picture and the reference feature mapping set;

[0087] Determine a target convolution set of the N frames of pictures according to the weight and feature mapping set of each frame of pictures;

[0088] The target convolution set is processed by using the candidate set network to obtain auxiliary data of the object in each frame of the picture.

[0089] In a feasible embodiment, the determining module 602 is specifically configured to:

[0090] Calculate the similarity between the feature mapping set of each frame of the picture and the reference feature mapping set to obtain the similarity between the feature mapping set of each frame of the picture and the reference feature mapping set;

[0091] The weight of each frame of the picture is determined according to the similarity corresponding to each frame of the picture.

[0092] In a feasible embodiment, the determining module 602 is specifically configured to:

[0093] Performing weighted processing on the weight and feature mapping set of each frame of the picture to obtain a total feature mapping set of the N frames of pictures;

[0094] The total feature map set of the N frames of pictures is averaged to obtain a target convolution set of the N frames of pictures.

[0095] In a feasible embodiment, the processing module 603 is further used for:

[0096] The reference feature map set is updated using the target convolution set.

[0097] In a feasible embodiment, before obtaining the feature mapping set of each frame of the N frames of pictures included in the collected video data, the device further includes: a filling module 604, wherein:

[0098] The acquisition module 601 is further used to acquire the maximum length and maximum width of N frames of pictures included in the collected video data;

[0099] The determination module 602 is further configured to determine the maximum value of the maximum length and the maximum width to obtain a padding value;

[0100] The padding module 604 is used to perform padding processing on the size of each frame of the N frames of pictures according to the padding value.

[0101] In a feasible embodiment, the filling module 604 is specifically used for:

[0102] For a target picture in the N frames of pictures, a picture zoom factor corresponding to the target picture is calculated according to the length and width of the target picture, where the target picture is any one of the N frames of pictures;

[0103] If the product of the larger value of the length and the width of the target image and the image zoom factor is greater than the padding value, the image zoom factor is adjusted, and the size of the target image is padded according to the adjusted image zoom factor;

[0104] If the product of the larger value of the length and the width of the target image and the image zoom factor is less than or equal to the padding value, the size of the target image is padded according to the image zoom factor.

[0105] In a feasible embodiment, the acquisition module 601 is specifically used for:

[0106] Obtain N frames of images included in the collected video data;

[0107] Each of the N frames of images is input into a skeleton network for convolution, pooling and activation processing to obtain a feature mapping set for each frame of the image.

[0108] It can be understood that the functions of the functional modules of the object recognition device of this embodiment can be specifically implemented according to the method in the above method embodiment, and the specific implementation process can refer to the above method embodiment. Figure 3 or Figure 5 The relevant description will not be repeated here.

[0109] See also Figure 7 , is a schematic diagram of the structure of a server provided in an embodiment of the present invention. The server described in this embodiment includes: a processor 701, a network interface 702, and a memory 703. The processor 701, the network interface 702, and the memory 703 may be connected via a bus or other means, and the embodiment of the present invention takes the connection via a bus as an example.

[0110] Among them, the processor 701 (or central processing unit (CPU)) is the computing core and control core of the server. The network interface 702 may optionally include a standard wired interface, a wireless interface (such as WI-FI, a mobile communication interface, etc.), which is controlled by the processor 701 for sending and receiving data. The memory 703 (Memory) is a memory device of the server, which is used to store programs and data. It can be understood that the memory 703 here can be a high-speed RAM memory, or a non-volatile memory (non-volatile memory), such as at least one disk storage; optionally, it can also be at least one storage device located away from the aforementioned processor 701. The memory 703 provides a storage space, which stores the operating system and executable program code of the server, which may include but is not limited to: Windows system (an operating system), Linux (an operating system) system, etc., and the present invention is not limited to this.

[0111] In the embodiment of the present invention, the processor 701 performs the following operations by running the executable program code in the memory 703:

[0112] Acquire a feature mapping set of each frame of N frames of images included in the collected video data, where the feature mapping set is used to indicate features of an object included in each frame of the image, where N is an integer greater than or equal to 1;

[0113] Determining a target convolution set of the N frames of images and auxiliary data of objects in each frame of images according to the feature mapping set of each frame of images and the recorded reference feature mapping set, wherein the auxiliary data is used to indicate an image area corresponding to the object in the image, and the reference feature mapping set is a convolution set of a previous batch of N frames of images processed, and the reference feature mapping set is used to indicate morphological features of the object;

[0114] The auxiliary data of the object and the target convolution set are subjected to classification regression processing to obtain category indication data and corresponding position data of each object included in the N frames of pictures.

[0115] In a feasible embodiment, the processor 701 is specifically configured to:

[0116] Get the reference feature map set of the record;

[0117] Determine the weight of each frame of the picture according to the feature mapping set of each frame of the picture and the reference feature mapping set;

[0118] Determine a target convolution set of the N frames of pictures according to the weight and feature mapping set of each frame of pictures;

[0119] The target convolution set is processed by using the candidate set network to obtain auxiliary data of the object in each frame of the picture.

[0120] In a feasible embodiment, the processor 701 is specifically configured to:

[0121] Calculate the similarity between the feature mapping set of each frame of the picture and the reference feature mapping set to obtain the similarity between the feature mapping set of each frame of the picture and the reference feature mapping set;

[0122] The weight of each frame of the picture is determined according to the similarity corresponding to each frame of the picture.

[0123] In a feasible embodiment, the processor 701 is specifically configured to:

[0124] Performing weighted processing on the weight and feature mapping set of each frame of the picture to obtain a total feature mapping set of the N frames of pictures;

[0125] The total feature map set of the N frames of pictures is averaged to obtain a target convolution set of the N frames of pictures.

[0126] In a feasible embodiment, the processor 701 is further configured to:

[0127] The reference feature map set is updated using the target convolution set.

[0128] In a feasible embodiment, before the processor 701 obtains the feature mapping set of each frame of the N frames of pictures included in the collected video data, it is also used to:

[0129] Get the maximum length and maximum width of N frames of images included in the collected video data;

[0130] Determine the maximum value of the maximum length and the maximum width to obtain a padding value;

[0131] The size of each frame of the N frames of pictures is padded according to the padded value.

[0132] In a feasible embodiment, the processor 701 is specifically configured to:

[0133] For a target picture in the N frames of pictures, a picture zoom factor corresponding to the target picture is calculated according to the length and width of the target picture, where the target picture is any one of the N frames of pictures;

[0134] If the product of the larger value of the length and the width of the target image and the image zoom factor is greater than the padding value, the image zoom factor is adjusted, and the size of the target image is padded according to the adjusted image zoom factor;

[0135] If the product of the larger value of the length and the width of the target image and the image zoom factor is less than or equal to the padding value, the size of the target image is padded according to the image zoom factor.

[0136] In a feasible embodiment, the processor 701 is specifically configured to:

[0137] Obtain N frames of images included in the collected video data;

[0138] Each of the N frames of images is input into a skeleton network for convolution, pooling and activation processing to obtain a feature mapping set for each frame of the image.

[0139] In a specific implementation, the processor 701, the network interface 702 and the memory 703 described in the embodiment of the present invention can execute the implementation method described in the process of an object recognition method provided in an embodiment of the present invention, and can also execute the implementation method described in an object recognition device provided in an embodiment of the present invention, which will not be repeated here.

[0140] A computer-readable storage medium is also provided in an embodiment of the present invention. The computer-readable storage medium stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the steps performed in the above-mentioned object recognition embodiment can be executed.

[0141] The present application also provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the steps performed in the above-mentioned object recognition embodiment.

[0142] A person skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing the relevant hardware through a computer program, and the program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0143] What is disclosed above is only a preferred embodiment of the present invention, and it certainly cannot be used to limit the scope of rights of the present invention. Ordinary technicians in this field can understand that all or part of the processes of the above embodiment and equivalent changes made according to the claims of the present invention still fall within the scope of the invention.

Claims

1. An object recognition method, characterized in that: The method comprises: Acquire a feature mapping set of each frame of N frames of images included in the collected video data, where the feature mapping set is used to indicate features of an object included in each frame of the image, where N is an integer greater than or equal to 1; Get the reference feature map set of the record; Calculate the similarity between the feature mapping set of each frame of the picture and the reference feature mapping set to obtain the similarity between the feature mapping set of each frame of the picture and the reference feature mapping set; Determine the weight of each frame of the picture according to the similarity corresponding to each frame of the picture; perform weighted processing on the weight of each frame of the picture and the feature mapping set to obtain the total feature mapping set of the N frames of the picture; Averaging the total feature map set of the N frames of pictures to obtain a target convolution set of the N frames of pictures; Using the candidate set network to process the target convolution set, the auxiliary data of the object in each frame of the picture is obtained; the auxiliary data is used to indicate the image area corresponding to the object in the picture, the reference feature map set is the convolution set of the previous batch of N frames of pictures processed, and the reference feature map set is used to indicate the morphological features of the object; The auxiliary data of the object and the target convolution set are subjected to classification regression processing to obtain category indication data and corresponding position data of each object included in the N frames of pictures.

2. The method according to claim 1, characterized in that The method further comprises: The reference feature map set is updated using the target convolution set.

3. The method according to claim 1, characterized in that Before obtaining a feature mapping set of each frame of N frames of pictures included in the collected video data, the method further includes: Get the maximum length and maximum width of N frames of images included in the collected video data; Determine the maximum value of the maximum length and the maximum width to obtain a padding value; The size of each frame of the N frames of pictures is padded according to the padded value.

4. The method according to claim 3, characterized in that The filling process of the size of each frame of the N frames of pictures according to the filling value includes: For a target picture in the N frames of pictures, a picture zoom factor corresponding to the target picture is calculated according to the length and width of the target picture, where the target picture is any one of the N frames of pictures; If the product of the larger value of the length and the width of the target image and the image zoom factor is greater than the padding value, the image zoom factor is adjusted, and the size of the target image is padded according to the adjusted image zoom factor; If the product of the larger value of the length and the width of the target image and the image zoom factor is less than or equal to the padding value, the size of the target image is padded according to the image zoom factor.

5. The method according to claim 1, characterized in that The feature mapping set of each frame of the N frames of the video data acquired includes: Obtain N frames of images included in the collected video data; Each of the N frames of images is input into a skeleton network for convolution, pooling and activation processing to obtain a feature mapping set for each frame of the image.

6. An object recognition device, characterized in that: The device comprises: An acquisition module, used to acquire a feature mapping set of each frame of N frames of images included in the collected video data, wherein the feature mapping set is used to indicate the features of an object included in each frame of the image, and N is an integer greater than or equal to 1; A determination module is used to obtain a recorded reference feature mapping set; perform similarity calculation on the feature mapping set of each frame of the picture and the reference feature mapping set to obtain the similarity between the feature mapping set of each frame of the picture and the reference feature mapping set; determine the weight of each frame of the picture according to the similarity corresponding to each frame of the picture; perform weighted processing on the weight of each frame of the picture and the feature mapping set to obtain the total feature mapping set of the N frames of the picture; perform average processing on the total feature mapping set of the N frames of the picture to obtain the target convolution set of the N frames of the picture; perform data processing on the target convolution set using a candidate set network to obtain auxiliary data of the object in each frame of the picture; the auxiliary data is used to indicate the image area corresponding to the object in the picture, the reference feature mapping set is the convolution set of the previous batch of N frames of the picture that were processed, and the reference feature mapping set is used to indicate the morphological features of the object; A processing module is used to perform classification regression processing on the auxiliary data of the object and the target convolution set to obtain category indication data and corresponding position data of each object included in the N frames of pictures.

7. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a processor, the processor is caused to execute the method according to any one of claims 1 to 5.

Images