Image detection and recognition method and apparatus

Abstract

The application relates to an image detection and recognition method and device, wherein the method comprises: performing frame extraction processing on video stream data to obtain each frame image; performing target detection on each frame image, and merging images with similar targets according to the target detection result to obtain a same-target data set; performing target recognition on a first image in the same-target data set, and taking the recognition result of the first image as the recognition result of each frame image. The method extracts each frame image in the video stream data, performs target detection on each frame image, and then merges images with similar targets into a data set as a same-target data set. When performing target recognition on each frame image, target recognition is only needed on one frame image in the same-target data set, and the recognition result of the image with the recognized target is directly used as the recognition result of other frame images, so that the number of times of target recognition is greatly reduced, and the calculation amount is effectively reduced.

Description

Technical Field

[0001] This application relates to the field of image processing technology, and in particular to an image detection and recognition method and apparatus. Background Technology

[0002] With the rapid advancement of hardware computing power and deep learning technology, artificial intelligence applications, represented by facial recognition, have been widely applied in the security technology field. In related technologies, the facial recognition process can generally be divided into two steps: first, detecting and cropping small images of faces; and second, presenting and recognizing these small images. That is, faces in each frame of the video stream are first cropped into small images, and then each small image containing a face is recognized. The computational load for small image recognition is greater than that for face detection, often resulting in higher latency in practical applications. Furthermore, in security scenarios, continuous frame-by-frame facial recognition of video streams is a relatively wasteful computational resource. Since adjacent video frames often have similarities, repeated recognition of multiple frames leads to redundant calculations, significantly increasing the computational load. Summary of the Invention

[0003] In view of this, this application proposes an image detection and recognition method that can effectively reduce the amount of computation in the process of target detection and recognition of images.

[0004] According to one aspect of this application, an image detection and recognition method is provided, comprising:

[0005] Frame extraction is performed on the video stream data to obtain each frame image in the video stream data;

[0006] Target detection is performed on each frame of the image. Based on the target detection results, images with similar targets are merged to obtain a dataset of the same target.

[0007] Target recognition is performed on the first image in the same target data set, and the recognition result of the first image is used as the recognition result of other frames in the same target data set.

[0008] In one possible implementation, images containing similar targets are merged based on the target detection results to obtain a dataset of images containing the same target, including:

[0009] Obtain the target detection results of the current frame image;

[0010] When the target detection result of the current frame image is that a target exists, the target overlap between the current frame image and the second image is calculated, and the current frame image is merged into the same target data set according to the target overlap between the current frame image and the second image;

[0011] The second image is the image whose target was detected before the target detection was performed on the current frame image, and which is the image whose sorting is closest to the current frame image.

[0012] In one possible implementation, the target overlap between the current frame image and the second image is calculated by calculating the similarity between the target detection bounding box data of the current frame image and the target detection bounding box data of the second image.

[0013] In one possible implementation, when calculating the similarity between the target detection bounding box data of the current frame image and the target detection bounding box data of the second image, the target detection bounding box data includes the coordinate values ​​of the target detection bounding box, as well as the height and width of the target detection bounding box.

[0014] In one possible implementation, merging the current frame image into the same target data set based on the target overlap between the current frame image and the second image includes:

[0015] Determine whether the target overlap between the current frame image and the second image satisfies the similarity condition;

[0016] When it is determined that the target overlap between the current frame image and the second image meets the similarity condition, the current frame image is merged into the same target data set, and the current frame image is updated to the second image.

[0017] In one possible implementation, the similarity condition is that the target overlap between the current frame image and the second image is greater than a similarity threshold.

[0018] The similarity threshold value ranges from 0.85 to 0.9.

[0019] In one possible implementation, before performing target recognition on the first image in the same target data set and using the recognition result of the first image as the recognition result of other frames in the same target data set, the method further includes: a step of judging whether the same target data set satisfies the first target deduplication condition;

[0020] When it is determined that the set of images with the same target satisfies the first target deduplication condition, the operation of performing target recognition on the first image in the set of images with the same target is performed, and the recognition result of the first image is used as the recognition result of the other images in the set of images with the same target is performed.

[0021] In one possible implementation, the first deduplication condition is:

[0022] The number of image frames in the same target data set is not less than the preset number of frames, and the maximum sequence number of the images in the same target data set is greater than or equal to the preset number of prior images.

[0023] In one possible implementation, before performing target recognition on the first image in the same target data set and using the recognition result of the first image as the recognition result of other frames in the same target data set, the method further includes:

[0024] Statistical analysis is performed on the images in the same target dataset;

[0025] When the proportion of images with the same first ranking result in the same target data set reaches a preset proportion, the operation of performing target recognition on the first image in the same target data set is performed, and the recognition result of the first image is used as the recognition result of other images in the same target data set.

[0026] According to another aspect of this application, an image detection and recognition device is also provided, comprising: a frame extraction processing module, an image detection and merging module, and an image recognition module;

[0027] The frame extraction processing module is configured to perform frame extraction processing on the video stream data to obtain each frame image in the video stream data;

[0028] The image detection and merging module is configured to perform target detection on each frame of images, and merge images with similar targets based on the target detection results to obtain a set of data with the same target.

[0029] The image recognition module is configured to perform target recognition on a first image in the same target data set, and use the recognition result of the first image as the recognition result of other frames in the same target data set.

[0030] After extracting each frame of the video stream data, target detection is performed on each frame. Images containing similar targets are then merged into a dataset called the same-target dataset. Next, target recognition is performed on the first image in the same-target dataset. For all other frames in the same dataset, the recognition result of the first image is directly used as the recognition result for each frame. Since the targets are the same in all frames of the same-target dataset, target recognition only needs to be performed on one frame in each dataset. The results of the images with identified targets are used as the recognition results for other frames. Compared to related technologies that perform target recognition on every single frame, this significantly reduces the number of target recognition operations, effectively reducing computational load and avoiding a large amount of redundant calculations.

[0031] Other features and aspects of this application will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0032] The accompanying drawings, which are included in and form part of this specification, illustrate exemplary embodiments, features, and aspects of this application together with the specification and serve to explain the principles of this application.

[0033] Figure 1 This diagram illustrates the hardware physical structure of the image detection and recognition method according to an embodiment of the present application when applied to a face recognition scenario in a security device equipped with a camera.

[0034] Figure 2 A flowchart illustrating the image detection and recognition method according to an embodiment of this application is shown;

[0035] Figure 3 Another flowchart of the image detection and recognition method according to an embodiment of this application is shown;

[0036] Figure 4 This document illustrates a flowchart of the training process for the target detection network model and the target recognition network model in the image detection and recognition method according to an embodiment of this application.

[0037] Figure 5 This is a structural block diagram of an image detection and recognition device according to an embodiment of this application. Detailed Implementation

[0038] Various exemplary embodiments, features, and aspects of this application will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0039] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0040] Furthermore, to better illustrate this application, numerous specific details are provided in the following detailed embodiments. Those skilled in the art should understand that this application can be implemented without certain specific details. In some instances, methods, means, components, and circuits well-known to those skilled in the art have not been described in detail in order to highlight the main points of this application.

[0041] First, it should be noted that the image detection and recognition method of this application embodiment can be applied in the security field to detect the video stream collected by the video acquisition device in the security equipment, thereby discovering whether there is a dangerous situation in the video stream based on the detection results, so as to achieve the purpose of security protection.

[0042] Specifically, the image detection and recognition method of this application embodiment can be used in various security scenarios based on video surveillance, such as face recognition scenarios for security devices with cameras.

[0043] In this scenario, hardware devices such as Figure 1 As shown, firstly, the captured video stream is uploaded to the first server (i.e., server 1) via a camera. The first server converts the video stream data into a frame sequence, then performs face detection on each frame in the converted frame sequence, and crops out the small face images and sends them to the second server (i.e., server 2). The second server then searches and recognizes the received face images to obtain a candidate ranking of recognition results with confidence levels. Finally, the candidate ranking is transmitted via cable to the subsequent devices, the third server and the front-end display device (i.e., server 3 and the front-end display device), thereby achieving the purpose of face detection and recognition in the video stream.

[0044] However, in the process of implementing face detection and recognition in video streams based on the above hardware physical structure, the second server needs to search and recognize each frame of image detected by the first server. This results in a large amount of computation for the second server, and the same face image needs to be calculated repeatedly, leading to redundant calculations.

[0045] Based on this, this application provides an image detection and recognition method that can effectively reduce the computational load of the face recognition process and avoid redundant calculations. It will be understood by those skilled in the art that the "target" in the image detection and recognition method of this application can be a face or other scenes. That is to say, the image detection and recognition method of this application can also be applied to other security application scenarios, and is not limited to face detection and recognition.

[0046] Figure 2 A flowchart illustrating an image detection and recognition method according to an embodiment of this application is shown. Figure 2 As shown, the method includes: step S100, performing frame extraction processing on the video stream data to obtain each frame image in the video stream data. Here, those skilled in the art will understand that performing frame extraction processing on the video stream data refers to converting the video stream data into a frame sequence, and then extracting each frame image from the frame sequence. The conversion of the video stream data and the extraction of images can both be implemented using conventional techniques in the art, and will not be elaborated upon in this application.

[0047] Step S200 involves performing target detection on each frame of images. Based on the target detection results, images containing similar targets are merged to obtain a set of images with the same target. It should be explained that target detection on each frame can be performed using conventional image detection methods in the field, such as neural network models with image detection capabilities. Merging images with similar targets in this step means storing and recording images containing detected targets that are similar as a single set. For example, in the face recognition scenario of a security device with a camera, where the target is a face, multiple faces may appear in a video stream, and each face may appear multiple times. Therefore, when performing target detection on each frame, images containing faces that are similar need to be merged into the same set. This set is the set of images with the same target. In other words, the elements in the set of images with the same target obtained in this step include frames containing images with similar targets. Furthermore, there can be multiple sets of images with the same target, and the targets present in different sets may vary.

[0048] Step S300 involves performing target recognition on the first image in the same target data set, and using the recognition result of the first image as the recognition result for all other frames in the same target data set. Those skilled in the art will understand that this step is a target recognition process. In this process, the method of this embodiment, after recognizing the first image in the same target data set and obtaining the recognition result of the first image, does not perform target recognition on other frames in the same target data set, directly using the recognition result of the first image as the recognition result for all other frames. This is because the targets detected in each frame in the same target data set belong to the same target, therefore only the first image needs to be recognized, thus achieving the purpose of multi-frame target deduplication. In one possible implementation, the first image can be any frame in the same target data set.

[0049] Therefore, the image detection and recognition method of this application extracts each frame of video stream data, performs target detection on each frame, and then merges images with similar targets into a dataset to form a common target data set. Furthermore, after performing target recognition on the first image in the common target data set, for each other frame in the common target data set, the recognition result of the first image is directly used as the recognition result for each frame. Since the targets in each frame of the common target data set are the same, when performing target recognition on each frame, only one frame in each common target data set needs to be recognized. The results of the images with already identified targets are used as the recognition results for other frames. Compared to the related technology that performs target recognition on every frame, this greatly reduces the number of target recognition operations, effectively reducing the computational load and avoiding a large amount of redundant computation.

[0050] In one possible implementation, step S200, merging images with similar targets based on the target detection results to obtain a set of data with the same target, can be achieved in the following way.

[0051] First, obtain the target detection result of the current frame image and determine whether the target detection result of the current frame image indicates the presence of a target. If the target detection result of the current frame image indicates the presence of a target, calculate the target overlap between the current frame image and the second image, and merge the current frame image into the same target data set based on the target overlap between the current frame image and the second image.

[0052] In the above implementation, the second image is the image whose frame number is closest to the current frame image among the images where a target is detected before target detection is performed on the current frame image. For example, if the current frame image has the sequence number i, and target detection has already been performed on the previous i frames, then the targets are present in frames 1, 3, 4, and (i-3). Here, the second image refers to the (i-3)th frame image. In other words, the second image is the image whose frame number is closest to the current frame image among the images where a target is detected before target detection is performed on the current frame image.

[0053] Additionally, it should be noted that after obtaining the target detection results of the current frame image, before calculating the target overlap between the current frame image and the second image, it is possible to first determine whether the second image exists. That is, determine whether a second image exists in the target detection results before performing target detection on the current frame image. If the second image does not exist, it indicates that the current frame image is the first frame image with a target detected during the target detection process, and in this case, the frame image can be directly merged into the same target data set. If the second image exists, it indicates that the current frame image is not the first frame image with a target detected during the target detection process, and in this case, it is necessary to calculate the target overlap between the current frame image and the images with detected targets, and then merge the same target images based on the calculated target overlap.

[0054] In other words, in the method of this application embodiment, when performing target detection on the frame sequence obtained by converting the video stream, the detection is performed frame by frame according to the order of each frame image in the frame sequence. Whether the current frame image can be merged into the same target data set depends on the target detection results of the previous frames. For images in which a target is detected for the first time during the target detection process, they are directly merged into the same target data set. For images in which a target is detected again during the target detection process, the target overlap with the previously detected target images is calculated, and the data is merged according to the target overlap. This ensures that the elements contained in the same target data set are all images with a target that is the same or similar, avoiding the situation where there are images with large differences in the target in the same target data set. This makes the target detection and recognition results obtained by this application embodiment more accurate and effectively ensures the reliability of the method of this application embodiment.

[0055] Furthermore, when calculating the target overlap between the current frame image and the second image, this can be done by calculating the similarity between the target detection bounding box data of the current frame image and the target detection bounding box data of the second image. In one possible implementation, when calculating the similarity between the target detection bounding box data of the current frame image and the target detection bounding box data of the second image, the target detection bounding box data includes the coordinate values ​​of the target detection bounding box, as well as the height and width of the target detection bounding box.

[0056] For example, if the target is a face, and a video stream contains K consecutive frames, each potentially containing a face, these frames can be represented in an ordered manner as: PIC1, PIC2, ​​PIC3, ..., PIC K After frame-by-frame object detection, these images yield an ordered set of faces: DETCTION1, DETECTION2, DETECTION3, ..., DETECTION K .

[0057] For any DETECTION i Let i ∈ {1, 2, ..., K}, each containing the top-left corner coordinates of a series of face detection boxes (i.e., object detection boxes), as well as the width and height data of the face detection boxes, denoted as:

[0058] DETECTION i ={FACE d1 FACE d2 FACE d3 , ...} = {[x d1 y d1 w d1 h d1 ], [x d2 y d2 w d2 h d2 ], [x d3 y d3 w d3 h d3 ],……}(1)

[0059] Here, it will be understood by those skilled in the art that x d1 Let y be the x-coordinate of the top-left corner of the target detection box. d1 w is the ordinate of the top-left corner of the target detection box. d1 h is the width of the target detection bounding box. d1 The height of the target detection box.

[0060] Specifically, for those who failed to [do something] in PIC i If a face is detected in i∈{1,2,…,K}, then:

[0061]

[0062] Therefore, DETECTION is defined. i The largest face detection bounding box in i∈{1,2,…,K} is:

[0063]

[0064] In defining the correlation of faces between frames, this application uses IOU (Intersection over Union) to calibrate two face detection bounding boxes (FACE). i FACE j The similarity of (i, j ∈ {1, 2, ..., K}) is shown in Equation 4.

[0065]

[0066] Therefore, the similarity between the target detection box data of the current frame image and the target detection box data of the second image is calculated by the above formula (4), and then the target overlap between the current frame image and the second image is judged according to the value of the calculated similarity. Then, the current frame image is processed to determine whether to merge it into the same target data set according to the judgment result.

[0067] According to the above formula (4), those skilled in the art can understand that when calculating the similarity between two frames of images, the calculation can be performed using the coordinate data, width and height data of the target detection boxes in the two frames of images.

[0068] Specifically, the similarity value is 0 when any of the following conditions are met when calculating the coordinates, width, and height of the target detection bounding boxes in two images:

[0069] That is, the sum of the x-coordinate of the target detection box in the current frame image and the width of the target detection box in the current frame image is less than or equal to the x-coordinate of the target detection box in the second image.

[0070] The x-coordinate of the target detection box in the current frame image is greater than or equal to the sum of the x-coordinate of the target detection box in the second image and the width of the target detection box in the second image.

[0071] The sum of the ordinate of the target detection box in the current frame image and the height of the target detection box in the current frame image is less than or equal to the ordinate of the target detection box in the second image.

[0072] The ordinate of the target detection box in the current frame image is greater than or equal to the sum of the ordinate of the target detection box in the second image and the height of the target detection box in the first image.

[0073] If the coordinate, width, and height data of the target detection bounding box in two frames do not meet all the above conditions, then the formula applies: The target overlap value between the two frames can be obtained by calculation.

[0074] Here, it is necessary to explain w i，j h represents the value of the overlap between the width of the target detection bounding box in the current frame and the width of the target detection bounding box in the second image. i，j w represents the value of the overlap between the height of the target detection bounding box in the current frame and the height of the target detection bounding box in the second image. i h is the width value of the target detection bounding box in the second image. i w represents the height of the target detection bounding box in the second image. j h is the width of the target detection bounding box in the current frame image. j This represents the height of the target detection bounding box in the current frame image.

[0075] Furthermore, the determination of target overlap between the current frame image and the second image can be achieved by checking whether their target overlap meets a similarity condition. If the target overlap between the current frame image and the second image meets the similarity condition, it indicates that the targets detected in the current frame image and the second image are similar or identical targets. Therefore, the current frame image can be merged into a dataset containing the second image that includes the same target. Simultaneously, the current frame image is updated with the second image to achieve iterative updates of the second image, providing an effective reference for comparison and judgment in the detection of the next frame image.

[0076] It should be noted that the similarity condition can be flexibly set according to the actual situation. In one possible implementation, the similarity condition can be set as follows: the overlap between the target in the current frame image and the second image is greater than the similarity threshold IOU. th The similarity threshold value can be set based on experience. For example, the similarity threshold value range can be set to 0.6 to 1.0. Preferably, the similarity threshold value can be 0.85 to 1.0.

[0077] After merging images with similar targets into a set of images with the same target through any of the above methods, in order to further improve the accuracy of the method in this application embodiment, before performing target recognition on the first image in the set of images with the same target and using the recognition result of the first image as the recognition result of other frames in the set of images with the same target, a step of judging whether the set of images with the same target meets the first target deduplication condition can also be taken.

[0078] Specifically, when it is determined that the set of data with the same target meets the first deduplication condition, it indicates that the current set of data with the same target can be used for target recognition using a multi-frame target deduplication method. That is, only the first image in the set of data with the same target needs to be recognized, and the target recognition result of the first image can be directly used as the recognition result of each other image.

[0079] Furthermore, the setting of the first target deduplication condition can be flexibly configured according to actual conditions. In one possible implementation, the first target deduplication condition can be set as follows: the number of image frames in the same target data set is not less than a preset number of frames, and the maximum sequence number of images in the same target data set is greater than or equal to a preset number of pre-detected images. Here, those skilled in the art will understand that the number of image frames in the same target data set refers to the number of elements in the same target data set. The maximum sequence number of images in the same target data set refers to the number of images extracted when converting video stream data into a frame sequence and performing frame extraction processing on the video stream data, where each extracted frame image is sequentially numbered according to the order of the frame sequence. The maximum sequence number of images is the last frame image in the sequence among all frames included in the same target data set. The preset number of pre-detected images refers to a pre-set number of pre-detected images.

[0080] In the method of this application embodiment, the preset frame number can be set based on empirical values. In one possible implementation, the preset frame number can be set to 3 to 7, preferably 5 or 7. The preset number of pre-checked frames can also be set based on empirical values, such as: the preset number of pre-checked frames can be set to 3 to 7, preferably 5 or 7.

[0081] After determining whether the data set with the same target meets the first target deduplication condition, the target recognition result of each frame image in the data set with the same target can be obtained by using the target deduplication method.

[0082] Furthermore, to further improve the reliability and accuracy of the method in this application embodiment, before performing target recognition on the first image in the target data set and using the recognition result of the first image as the recognition result of other frames in the same target data set, the method further includes: performing statistics on the images in the same target data set; when the proportion of the number of images with the same first ranking result in the same target data set reaches a preset proportion, performing target recognition on the first image in the same target data set and using the recognition result of the first image as the recognition result of other images in the same target data set.

[0083] Here, those skilled in the art will understand that after a single image is sent for recognition, the recognition engine will return a series of candidate recognition results sorted from high to low according to the similarity score, with the first result being the one with the highest similarity score. That is, the first result is the candidate recognition result with the highest similarity score among the candidate recognition results returned after the single-frame image is sent for recognition. The value of the preset ratio can be set based on experience, such as: the preset ratio can be set to 0.6 to 1.0, preferably 0.8 to 1.0.

[0084] In one possible implementation, the statistical calculation of the proportion of images with the same first-order result within the same target dataset can be performed as follows: Q images are selected from the same target dataset, and then it is calculated whether any of these Q images contain the same number of images. The first order of the images is the same. If there exists If the first-order results of the images are the same, it means that the proportion of images with the same first-order results in the current target data set has reached the preset proportion. Therefore, the multi-frame target deduplication method can be used to obtain the target recognition results of each frame image in the target data set.

[0085] Where h is the number of sheets threshold coefficient, and the value of h can be set according to the preset ratio. The value of Q can be set to: 60% of the total number of elements in the same target dataset to 90% of the total number of elements in the same target dataset.

[0086] It should also be noted that in the method of this application embodiment, if subsequently received images also meet the requirements of the aforementioned target data set, the recognition results in the aforementioned target data set can be directly used. However, if a single image does not meet the requirements of the aforementioned target data set, it is not necessary to wait for the results of subsequent images to be integrated together; the second server can directly recognize the single image.

[0087] In addition, it should be noted that before using the target recognition result of the first image as the recognition result of each image in each frame of the same target data set, the two judgment criteria of the first deduplication target condition and the proportion of the number of images with the same first ranking result in the same target data set can be considered together, or one of them can be selected.

[0088] When considering both the first deduplication target condition and the proportion of images with the same first-order result in the same target dataset, the following steps can be taken: First, determine if the first deduplication target condition is met. If it is, then the images in the same target dataset are statistically analyzed to determine if the proportion of images with the same first-order result in the same target dataset reaches a preset proportion. If the proportion of images with the same first-order result in the same target dataset reaches the preset proportion, then multi-frame target deduplication processing is performed on the same target dataset.

[0089] After determining, through the above steps, that the target data set satisfies the multi-frame target deduplication method for target recognition, target recognition can be performed using one frame image (i.e., the first image) from the same target data set, and the recognition result of this frame image can be used as the recognition result for other frames. It should be noted that the first image can be arbitrarily selected from its set of images with the same target. In one possible implementation, the first image can be the second image mentioned earlier, i.e., the first image and the second image use the same frame image, or other frame images can be selected.

[0090] Therefore, the method of this application embodiment uses IOU detection to determine the continuity of the target in each frame of the image containing the target, and judges the reliability of the continuity of each frame of the obtained target data set. When it is determined that the continuity of each frame of the target data set is reliable, the recognition result of the first image in the target data set is directly used as the recognition result of the first image as the recognition result of other frames in the target data set. Thus, it is not necessary to perform target recognition processing on each frame of the image, which greatly reduces the pressure on recognition computing power and reduces the amount of data calculation while ensuring the accuracy and reliability of the recognition result.

[0091] To more clearly illustrate the method of the embodiments of this application, the following uses face recognition in a video stream as an example for more detailed explanation.

[0092] See Figure 3 In this specific embodiment, the application scenario is as follows: A video stream contains K consecutive images, all of which may contain human faces. These images can be represented in an ordered manner as: PIC1, PIC2, ​​PIC3, ..., PIC K .

[0093] Based on the above application scenarios, when performing target detection and recognition on the aforementioned video stream, step S100 is first used to perform frame extraction processing on the video stream, and then the images obtained from the frame extraction processing are numbered, i.e., PIC1, PIC2, ​​PIC3, ..., PIC K .

[0094] Then, in step S210, face detection is performed frame by frame on each image, and in step S220, it is determined whether a face is detected in the current image.

[0095] If no face is detected, step S220' counts the number of consecutive undetected image frames M, and step S230' determines whether the current number of consecutive undetected image frames M is greater than a preset frame number m. If the number of consecutive undetected image frames M is greater than the preset frame number m, it indicates that the target face is likely not present in the currently acquired video stream data. Therefore, step S240' can directly set the set of identical faces S to an empty set. It should be noted that the value of the preset frame number m can be flexibly set according to actual conditions, and is not specifically limited here.

[0096] If a face is detected in step S220, then step S231 is executed to determine whether there is an image with a detected face (i.e., FACE) before the detection of the current frame image. last If it is determined that there is no image in which a face has been detected, it means that the current frame image is the first image in which a face has been detected. Therefore, the current frame image can be directly added to the same face set S (i.e., the same target data set mentioned above) through step S240.

[0097] Meanwhile, in step S230, when it is determined that there is an image in which a face has been detected, it indicates that the current frame image is not the first image in which a face has been detected. Therefore, it is still necessary to calculate the target overlap between the current frame image and the previously detected face image in step S232, and determine whether the target overlap between the current frame image and the previously detected face image is greater than the similarity threshold.

[0098] When it is determined that the target overlap between the current frame image and previously detected face images is greater than the similarity threshold, it indicates that the current frame image and previously detected face images are continuous. Therefore, the current frame image can be added to the same-face set S in step S240. When it is determined that the target overlap between the current frame image and previously detected face images is less than or equal to the similarity threshold, step S240' is executed, and the same-face set S is set to an empty set.

[0099] In step S240, after adding the current frame image to the same-face set S, it is also necessary to update the current frame image with the image of the closest detected face to the current frame image (i.e., perform FACE). last =FACE i This is to facilitate the determination of continuity of the next frame image using the current frame image (FACE). i () is used as the criterion for judgment.

[0100] The above steps yield the same-face set data S corresponding to the video stream data. It should be noted that the obtained same-face set S contains images with multiple frames showing the same or similar faces. After obtaining the same-face set S, step S031 is executed to determine whether the maximum image index i in the same-face set is greater than or equal to the preset number of prior images P, and whether the number of elements in the same-face set S is greater than or equal to the preset number of images Q.

[0101] When it is determined that the maximum sequence number i in the same face set is greater than or equal to the preset number of prior detection images P, and the number of elements in the same face set S is greater than or equal to the preset number of images Q, it indicates that the number of image frames in the same face set S meets the initial conditions for target detection and recognition using the multi-frame target deduplication method. Therefore, step S032 can be continued to extract Q face images from the same face set, and count whether there are more than [Q·h] images with the same first order result among the Q face images. If there are more than [Q·h] images with the same first order result, it indicates that the continuity of each frame image in the currently obtained same face set S has reached the multi-frame target deduplication standard. Therefore, the recognition result of the first image in the same face set S can be obtained directly through step S300, and the recognition result of the first image can be used as the recognition result of other frames.

[0102] In step S031 above, if it is determined that the maximum sequence number i of the images in the same-face set S is less than the preset number of images P, or the number of elements in the same-face set S is less than the preset number Q, it indicates that the number of image frames in the currently acquired same-face set S is insufficient to support target detection and recognition using multi-frame target deduplication. Therefore, step S310' can be executed at this time to perform target recognition on each frame of the image, obtain the recognition results of each frame, and set the same-face set S to an empty set. Next, step S320' is executed to filter out the recognition results (Cans) of each frame of the image. i In the score, i Candidate results that are greater than or equal to the scoring threshold (s) are selected, and the first-ranked candidate result is accepted through step S330'. It should be noted that when calculating the score for the recognition results of each frame, conventional score calculation methods in this field can be used, which will not be elaborated here. Furthermore, the value of the scoring threshold s can be flexibly set according to the actual situation, and no specific limitation is made here.

[0103] Furthermore, it should be noted that when performing target detection and recognition on each frame of the image, conventional detection and recognition methods in this field can be used, or artificial intelligence methods can be employed. That is, by leveraging existing target detection and recognition network models, a sample database is constructed, and after labeling the sample data in the database, the target detection and recognition network models are trained to determine the parameters of each network model. Then, the trained network models are used to perform target detection and recognition on the actual video stream as described above.

[0104] It should also be noted that the training process for each network model can be achieved using the following steps.

[0105] That is, refer to Figure 4 First, in step S100', the video stream is processed by frame extraction, and the resulting frames are numbered as PIC1, PIC2, ​​PIC3, ..., PIC K .

[0106] Then, step S210” performs face detection frame by frame on each image, and step S220” determines whether a face is detected in the current image. If a face is detected in step S220”, step S310” is executed to perform target recognition frame by frame on each image, obtain the recognition results of each image, and set the set of identical faces S to an empty set. Next, step S320” is executed to filter out candidate results from the recognition results of each image, with scores greater than or equal to the score threshold, and step S330” selects the first candidate result for MMS.

[0107] Accordingly, based on any of the target detection and recognition methods described above, this application also provides a target detection and recognition device. Since the working principle of the target detection and recognition device provided in this application is the same as or similar to the principle of the target detection and recognition method in this application, the repeated details will not be elaborated further.

[0108] See Figure 5 The target detection and recognition device 100 provided in this application includes a frame extraction processing module 110, an image detection and merging module 120, and an image recognition module 130. The frame extraction processing module 110 is configured to extract frames from video stream data to obtain individual frames of the video stream data. The image detection and merging module 120 is configured to perform target detection on each frame of the image, and merge images with similar targets based on the target detection results to obtain a target-same data set. The image recognition module 130 is configured to perform target recognition on a first image in the target-same data set, and use the recognition result of the first image as the recognition result for other frames in the target-same data set.

[0109] The various embodiments of this application have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. An image detection and recognition method, characterized in that, include: Frame extraction is performed on the video stream data to obtain each frame image in the video stream data; Target detection is performed on each frame of the image. Based on the target detection results, images with similar targets are merged to obtain a dataset of the same target. Target recognition is performed on the first image in the same target data set, and the recognition result of the first image is used as the recognition result of other frames in the same target data set; wherein, the first image is any frame in the same target data set; Before performing target recognition on the first image in the same target data set, and using the recognition result of the first image as the recognition result of other frames in the same target data set, the method further includes: Statistical analysis is performed on the images in the same target dataset; When the proportion of images with the same first ranking result in the same target data set reaches a preset proportion, the operation of performing target recognition on the first image in the same target data set is performed, and the recognition result of the first image is used as the recognition result of other images in the same target data set. The first ranking result is the candidate recognition result with the highest similarity score among the candidate recognition results returned after a single frame image is sent for recognition; The statistical calculation of the proportion of images with the same first-order result in the same target dataset can be performed as follows: Select Q images from the same target dataset, and then calculate whether there are ⌈Q•h⌉ images with the same first-order result among these Q images. If there are ⌈Q•h⌉ images with the same first-order result, it indicates that the proportion of images with the same first-order result in the same target dataset has reached the preset proportion. Where h is the number of sheets threshold coefficient, and the value of h is set according to the preset ratio; the value of Q is set as: the total number of elements in the same target data set. 60% ~ the total number of elements in the same target dataset 90%.

2. The method according to claim 1, characterized in that, Based on the target detection results, images with similar targets are merged to obtain a dataset of images containing the same target, including: Obtain the target detection results of the current frame image; When the target detection result of the current frame image is that a target exists, the target overlap between the current frame image and the second image is calculated, and the current frame image is merged into the same target data set according to the target overlap between the current frame image and the second image; The second image is the image whose target was detected before the target detection was performed on the current frame image, and which is the image whose sorting is closest to the current frame image.

3. The method according to claim 2, characterized in that, When calculating the target overlap between the current frame image and the second image, the similarity between the target detection box data of the current frame image and the target detection box data of the second image is calculated.

4. The method according to claim 3, characterized in that, When calculating the similarity between the target detection bounding box data of the current frame image and the target detection bounding box data of the second image, the target detection bounding box data includes the coordinate values ​​of the target detection bounding box, as well as the height and width of the target detection bounding box.

5. The method according to claim 2, characterized in that, Based on the target overlap between the current frame image and the second image, the current frame image is merged into the same target data set, including: Determine whether the target overlap between the current frame image and the second image satisfies the similarity condition; When it is determined that the target overlap between the current frame image and the second image meets the similarity condition, the current frame image is merged into the same target data set, and the current frame image is updated to the second image.

6. The method according to claim 5, characterized in that, The similarity condition is that the target overlap between the current frame image and the second image is greater than a similarity threshold. The similarity threshold value ranges from 0.85 to 0.

9.

7. The method according to any one of claims 1 to 6, characterized in that, Before performing target recognition on the first image in the same target data set and using the recognition result of the first image as the recognition result of other frames in the same target data set, the method further includes: a step of judging whether the same target data set satisfies the first target deduplication condition; When it is determined that the set of images with the same target satisfies the first target deduplication condition, the operation of performing target recognition on the first image in the set of images with the same target is performed, and the recognition result of the first image is used as the recognition result of the other images in the set of images with the same target is performed.

8. The method according to claim 7, characterized in that, The first deduplication condition is: The number of image frames in the same target data set is not less than the preset number of frames, and the maximum sequence number of the images in the same target data set is greater than or equal to the preset number of prior images.

9. An image detection and recognition device, characterized in that, include: Frame extraction processing module, image detection and merging module, and image recognition module; The frame extraction processing module is configured to perform frame extraction processing on the video stream data to obtain each frame image in the video stream data; The image detection and merging module is configured to perform target detection on each frame of images, and merge images with similar targets based on the target detection results to obtain a set of data with the same target. The image recognition module is configured to perform target recognition on a first image in the same target data set, and use the recognition result of the first image as the recognition result of other frames in the same target data set; wherein, the first image is any frame in the same target data set; Before performing target recognition on the first image in the same target data set, and using the recognition result of the first image as the recognition result of other frames in the same target data set, the method further includes: Statistical analysis is performed on the images in the same target dataset; When the proportion of images with the same first ranking result in the same target data set reaches a preset proportion, the operation of performing target recognition on the first image in the same target data set is performed, and the recognition result of the first image is used as the recognition result of other images in the same target data set. The first ranking result is the candidate recognition result with the highest similarity score among the candidate recognition results returned after a single frame image is sent for recognition; The statistical calculation of the proportion of images with the same first-order result in the same target dataset can be performed as follows: Select Q images from the same target dataset, and then calculate whether there are ⌈Q•h⌉ images with the same first-order result among these Q images. If there are ⌈Q•h⌉ images with the same first-order result, it indicates that the proportion of images with the same first-order result in the same target dataset has reached the preset proportion. Where h is the number of sheets threshold coefficient, and the value of h is set according to the preset ratio; the value of Q is set as: the total number of elements in the same target data set. 60% ~ the total number of elements in the same target dataset 90%.

Images