Re-recognition device and method
By using a processor to calculate the confidence score and visual score of the image in the re-identification device, and screening high-quality images, the problem of poor re-identification effect due to poor image quality in the prior art is solved, and a better re-identification effect is achieved.
Patent Information
- Application Number
- CN202311242614.6
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-09-25
- Publication Date
- 2025-03-25
AI Technical Summary
The existing re-identification technology cannot clearly present the character images due to factors such as monitor position, shooting angle, distance, imaging quality and object occlusion, resulting in poor re-identification effect.
A re-identification device is designed, including a processor and a storage, by capturing an object image from the input image, calculating confidence scores and visual scores, judging the quality of the image and deciding whether to store the image, so as to perform the re-identification step.
Effectively filter out high-quality images suitable for re-recognition, improve re-recognition effect, and enhance the recognition ability of character images.
Smart Images
Figure CN119693968A_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to an identification device and method, and more particularly to a re-identification device and method. Background Art
[0002] Re-identification technology is to classify images containing objects such as human bodies according to the appearance characteristics of the objects, so as to summarize the images corresponding to the same object (for example, a person). In order to realize re-identification technology, a large number of images containing objects need to be collected as recognition data.
[0003] For example, many re-identification technologies use images of people captured by surveillance cameras as identification data. However, due to factors such as the position of the surveillance camera, shooting angle, shooting distance, image quality, and object occlusion, the person is often blocked, only the back of the person is captured, the person is too small, or the image is blurred, so that the image of the person captured by the surveillance camera cannot clearly show all parts of the person, resulting in poor re-identification results.
[0004] In view of this, how to screen image data suitable for re-identification has become a goal that the industry urgently needs to work on. Summary of the invention
[0005] In order to solve the above problems, the present disclosure proposes a re-identification device, comprising a storage and a processor. The processor is coupled to the storage, and the processor is used to perform the following operations: capturing a first object image corresponding to an object from a first input image; based on the first object image, calculating a confidence score of the first object image corresponding to the object; based on the first object image, calculating multiple visibility scores of multiple key points corresponding to the first object image; based on the confidence score and the visibility scores of the first object image, determining whether to store a first image including the first object image in the storage as one of multiple images; and performing a re-identification step based on the images stored in the storage.
[0006] The present disclosure also provides a re-identification method applicable to an electronic device, wherein the electronic device includes a storage device, and the re-identification method includes: capturing a first object image corresponding to an object from a first input image; based on the first object image, calculating a confidence score of the first object image corresponding to the object; based on the first object image, calculating multiple visibility scores of multiple key points corresponding to the first object image; based on the confidence score and the visibility scores of the first object image, determining whether to store the first object image in the storage device as one of multiple images; and performing a re-identification step based on the images stored in the storage device.
[0007] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are intended to provide further explanation of the disclosure as claimed. BRIEF DESCRIPTION OF THE DRAWINGS
[0008] In order to make the above and other objects, features, advantages and embodiments of the present disclosure more clearly understood, the accompanying drawings are described as follows:
[0009] Figure 1 This is a schematic diagram of a re-identification device in the first embodiment of the present disclosure;
[0010] Figure 2 A schematic diagram of capturing an object image from an input image and calculating a confidence score in the first embodiment of the present disclosure;
[0011] Figure 3 A schematic diagram of key points in an object image in the first embodiment of the present disclosure; and
[0012] Figure 4 Flow chart of the re-identification method in the second embodiment of the present disclosure.
[0013]
Explanation of symbols
[0014] 1: Re-identification device
[0015] 12: Processor
[0016] 14: Storage
[0017] IMG: Input image
[0018] RS: Confidence score
[0019] OBJ: Object Image
[0020] KP0~KP16: Key Points
[0021] 200: Re-identification method
[0022] S201~S205: Steps DETAILED DESCRIPTION
[0023] In order to make the description of the present disclosure more detailed and complete, reference may be made to the attached drawings and various embodiments described below, in which the same numbers in the drawings represent the same or similar elements.
[0024] Please refer to Figure 1 , which is a schematic diagram of the re-identification device 1 in the first embodiment of the present disclosure. Figure 1As shown, the re-identification device 1 includes a processor 12 and a storage 14, wherein the processor 12 is coupled to the storage 14. The re-identification device 1 is used to select images suitable for re-identification data according to the imaging quality of objects in the input images. Please refer to the following description for the specific operation of the re-identification device 1.
[0025] In some embodiments, the processor 12 may include a central processing unit (CPU), a graphics processing unit (GPU), a multi-processor, a distributed processing system, an application specific integrated circuit (ASIC), and / or a suitable computing unit.
[0026] In some embodiments, storage 14 may include semiconductor or solid-state memory, magnetic tape, removable computer disk, random access memory (RAM), read-only memory (ROM), hard disk, and / or optical disk.
[0027] First, please refer to Figure 2 , the processor 12 of the re-identification device 1 extracts the object image OBJ corresponding to the object from the input image IMG. Figure 2 As shown, after the processor 12 recognizes a person in the input image IMG, it can extract the object image OBJ containing the person.
[0028] Furthermore, the processor 12 calculates a confidence score RS of the object corresponding to the object image OBJ based on the object image OBJ, wherein the confidence score RS is used to indicate a confidence level that the object image OBJ contains the object (ie, the human body).
[0029] In some embodiments, the processor 12 can use an image recognition algorithm to identify whether the input image IMG contains an object to capture the object image OBJ, and based on the object image OBJ, calculate a reliability score of the object contained in the object image OBJ as a confidence score RS. In some embodiments, the confidence score RS is a value between 0 and 1, and the higher the confidence score RS, the higher the probability that the object image OBJ contains the object. In some embodiments, the processor 12 can use the YOLOv7 (You Only Look Once version 7) algorithm to complete the operation of capturing the object image OBJ and calculating the confidence score RS.
[0030] Please refer to the following for further reference Figure 3The processor 12 calculates the visibility scores of the key points KP0-KP16 corresponding to the object image OBJ based on the object image OBJ. Figure 3 As shown, the processor 12 marks key points KP0-KP16 based on various parts of the object in the object image OBJ.
[0031] In this embodiment, 5 key points KP0-KP4 are key points of the face, and 12 key points KP5-KP16 are key points of the body, wherein the facial key points include the nose key point KP0, the left eye key point KP1, the right eye key point KP2, the left ear key point KP3, and the right ear key point KP4; the body key points include the left shoulder key point KP5, the right shoulder key point KP6, the left elbow key point KP7, the right elbow key point KP8, the left wrist key point KP9, the right wrist key point KP10, the left waist key point KP11, the right waist key point KP12, the left knee key point KP13, the right knee key point KP14, the left ankle key point KP15, and the right ankle key point KP16.
[0032] In addition, the processor 12 further calculates the visibility scores corresponding to the key points KP0 - KP16 , where the visibility scores are used to indicate the visibility of each of the key points KP0 - KP16 in the object image OBJ.
[0033] In some embodiments, the visibility score is a value between 0 and 1, and the higher the visibility score, the higher the probability that the object image OBJ contains key points. For example, if the right foot of the human body in the object image OBJ is blocked, the visibility scores of the corresponding right knee key point KP14 and right ankle key point KP16 are 0; if the object image OBJ only shows the back of the human body, the visibility score of the facial key point is 0.
[0034] In addition, if the human body in the object image OBJ is far away from the shooting equipment (e.g., monitor, camera) resulting in a low resolution of the object image OBJ, or if the image quality is poor resulting in a blurry object image OBJ, the processor 12 may also calculate a low visibility score for the corresponding key points KP0-KP16. On the other hand, if the front of the human body is shown in the object image OBJ, all parts are not blocked and / or the resolution is high, the processor 12 calculates a high visibility score for the corresponding key points KP0-KP16.
[0035] In some embodiments, the visibility scores corresponding to the aforementioned facial key points KP0-KP4 are facial visibility scores, and the visibility scores corresponding to the body key points KP5-KP16 are body visibility scores.
[0036] In some embodiments, the processor 12 can calculate the visibility scores of the corresponding key points KP0-KP16 by the following operations: determining the positions of the key points KP0-KP16 in the object image OBJ, wherein the key points KP0-KP16 correspond to the parts of the object; and calculating the visibility scores corresponding to the key points KP0-KP16 based on a plurality of pixels at the positions of the key points KP0-KP16 in the object image OBJ. In other words, after the processor 12 marks the positions of the key points KP0-KP16 in the object image OBJ, it further calculates the corresponding visibility scores through the pixels around the positions of the key points KP0-KP16.
[0037] In some embodiments, the processor 12 can use an image recognition algorithm to calculate the confidence scores of the various parts of the human body in the object image OBJ for the key points KP0-KP16 as the visibility scores of the key points KP0-KP16. In some embodiments, the processor 12 can use the YOLOv7-Pose algorithm to complete the operation of marking the key points KP0-KP16 and calculating the visibility scores corresponding to the key points KP0-KP16.
[0038] Next, the processor 12 determines whether to store the image including the object image OBJ in the storage 14 based on the confidence score RS and the visibility score of the object image OBJ. Finally, the processor 12 performs a re-identification step based on the image stored in the storage 14.
[0039] In some embodiments, the processor 12 can determine whether the confidence score RS and the visible score are greater than a threshold value (for example, 0.8). If the confidence score RS and the visible score are greater than the threshold value, the image containing the object image OBJ is stored in the storage 14 for a re-identification step. The image containing the object image OBJ can be determined according to actual needs, such as the process of the re-identification step, the effect of the re-identification, etc. It can be the object image OBJ itself, the input image IMG and / or an image block containing the object image OBJ cut out from the input image IMG.
[0040] In some embodiments, the processor 12 may further calculate a first total score based on the confidence score RS and the visibility score of the object image OBJ, and in response to the first total score being greater than a threshold, store the object image OBJ in the storage 14. For example, the processor 12 adds and averages the confidence score RS and the visibility score to calculate the first total score, and when the first total score is greater than a threshold (e.g., 0.8), stores the object image OBJ in the storage 14.
[0041] In some embodiments, the processor 12 may also calculate the first total score based on the confidence score RS of the object image OBJ, the visibility score of the object image OBJ, and multiple weights corresponding to the confidence score RS and the visibility score. Specifically, the confidence score RS and the visibility scores of different parts of the corresponding object each represent different attributes of the object image OBJ, so the re-identification device 1 may set corresponding weights according to the degree of emphasis of different attributes.
[0042] For example, if the recognition strength of facial features needs to be improved, the re-identification device 1 can set the weight corresponding to the visibility score of the facial key points to a higher value. In this way, compared with the confidence score RS of the object image OBJ and the visibility scores of other parts in the object image OBJ, the re-identification device 1 can select images with clearer facial features.
[0043] In some embodiments, the re-identification device 1 screens and stores the image data in the storage 14 for performing a re-identification step, wherein the re-identification step includes, after receiving the image to be identified, grouping the image to be identified into a first category of multiple identification categories; and based on the first category, outputting a recognized image corresponding to the first category.
[0044] Specifically, the re-identification step can group the input images and group the images corresponding to the same object (e.g., the same person) into the same category. Therefore, after the re-identification step receives and groups the image to be identified, it can further output other images (i.e., identified images) that are grouped into the same category as the image to be identified.
[0045] In this way, the re-identification step can be applied to track the time and location of the appearance of the same object. For example, after multiple surveillance camera images located in different locations are grouped using the re-identification step, it can be summarized on which surveillance camera screen the same person appears at each time point, and the person's whereabouts can be further obtained.
[0046] In some embodiments, the re-identification step further includes training a re-identification model based on the images stored in the storage 14. Specifically, the re-identification device 1 can use the images in the storage 14 as training data to train the re-identification model. The re-identification model can be a clustering model. During the training process, the re-identification model can be based on the features of the objects captured by the images in the training data, and the trained re-identification model can cluster the input images as in the re-identification step in the aforementioned embodiment, and classify the images corresponding to the same object (e.g., the same person) into the same category.
[0047] In some embodiments, the processor 12 of the re-ID apparatus 1 further tracks the object from a second input image to capture a second object image corresponding to the object, wherein a capture time of the second input image is later than the capture time of the first input image; based on the second object image, calculates the confidence score of the second object image corresponding to the object; based on the second object image, calculates the visibility scores of the second object image corresponding to the key points; and based on the confidence score and the visibility scores of the second object image, determines whether to store a second image including the second object image in the storage as one of the images.
[0048] Specifically, the re-identification device 1 may further receive an input image subsequent to the input image IMG (eg, the next frame in the video), and determine whether to use a second object image captured from the subsequent image for the re-identification step based on the same operation as determining whether to use the object image OBJ as an image.
[0049] In some embodiments, the processor 12 of the re-identification device 1 can track the position of the same object in different frames (i.e., the input image IMG and subsequent images) in the video, and further determine whether to store the frame image to the storage 14. In some embodiments, the processor 12 can use a target tracking algorithm (e.g., DeepSort algorithm) to track the position of the same object in multiple images.
[0050] In some embodiments, the re-identification device 1 may further calculate a first total score based on the confidence score RS and the visible score of the object image OBJ; calculate a second total score based on the confidence score RS and the visible score of the subsequent image; and determine to store the image in the storage 14 as the image for the re-identification step based on the first total score and the second total score.
[0051] Similar to the operation of calculating the first total score mentioned above, the processor 12 of the re-identification device 1 can calculate the second total score corresponding to the subsequent image through the same operation, and determine the image to be stored based on the first total score and the second total score, for example: storing both or either the object image OBJ and the second object image to the storage 14.
[0052] For example, the processor 12 can extract multiple images (i.e., object images) containing the same person from multiple frames in a video, then calculate the total score of each image based on the confidence score and the visibility score, and finally filter the images based on the total scores of each image. For example, the processor 12 can sort the multiple total scores and store one or more images with the highest total scores for the re-identification step, or filter the images with total scores exceeding the threshold value (e.g., 0.8) for the re-identification step. In this way, the re-identification device 1 can extract images containing the same person from the video, and filter out images with better imaging quality for the re-identification step.
[0053] It should be noted that the above embodiment is described using an object image OBJ included in the input image IMG as an example. However, in other embodiments, when the input image includes multiple objects, the re-identification device 1 can capture multiple object images through the same operation, and further determine whether to store the multiple object images in the storage 14 as a re-identification step through the same operation.
[0054] It should be noted that although the above embodiments use the human body as an example of an object to illustrate the operation of the re-identification device 1, the technology proposed in the present disclosure is not limited to this. In fact, the re-identification device 1 can also apply the same operation to the object images of other types of objects (such as animals).
[0055] In summary, the re-identification device 1 proposed in the present disclosure can capture an object image containing an object in an input image, and determine whether to use the object image for the re-identification step based on the imaging quality of the object image. In addition, the re-identification device 1 can also set weights for different parts of the object to filter out images with better imaging quality of specific parts. Furthermore, the re-identification device 1 can also capture multiple object images corresponding to the same object from multiple frames of a video, and filter out the object image with better imaging quality for the re-identification step.
[0056] The present disclosure also provides a re-identification method 200, which is applicable to an electronic device (e.g., re-identification device 1), wherein the electronic device includes a storage device (e.g., storage device 14). In some embodiments, the electronic device further includes a processor (e.g., processor 12) for executing the re-identification method 200. Figure 4 As shown, the re-identification method 200 includes steps S201 to S205.
[0057] In step S201, the electronic device captures a first object image corresponding to an object from a first input image. In step S202, the electronic device calculates a confidence score of the first object image corresponding to the object based on the first object image. In step S203, the electronic device calculates multiple visual scores of multiple key points corresponding to the first object image based on the first object image. In step S204, the electronic device determines whether to store the first object image in the storage as one of the multiple images based on the confidence score and the visual scores of the first object image. In step S205, the electronic device performs a re-recognition step based on the images stored in the storage.
[0058] In some embodiments, step S204 further comprises calculating a first total score based on the confidence score of the first object image and the visibility scores; and in response to the first total score being greater than a threshold, storing the first object image in the storage.
[0059] In some embodiments, the step of calculating the first total score further includes calculating the first total score based on the confidence score of the first object image, the visibility scores of the first object image, and a plurality of weights corresponding to the confidence score and the visibility scores.
[0060] In some embodiments, the re-identification method 200 further includes tracking the object from a second input image to capture a second object image corresponding to the object, wherein a capture time of the second input image is later than the capture time of the first input image; based on the second object image, calculating the confidence score that the second object image corresponds to the object; based on the second object image, calculating the visibility scores of the key points corresponding to the second object image; and based on the confidence score and the visibility scores of the second object image, determining whether to store a second image including the second object image in the storage as one of the images.
[0061] In some embodiments, step S204 further includes calculating a first total score based on the confidence score of the first object image and the visual scores; calculating a second total score based on the confidence score of the second object image and the visual scores; and determining to store in the storage as the content of the images based on the first total score and the second total score.
[0062] In some embodiments, step S203 further includes determining a position of each of the key points in the first object image, wherein each of the key points corresponds to a portion of the object; and calculating one of the visibility scores corresponding to each of the key points based on a plurality of pixels at the position of each of the key points in the first object image.
[0063] In some embodiments, the key points include a plurality of facial key points and a plurality of body key points, and the visibility scores include a plurality of facial visibility scores corresponding to the facial key points and a plurality of body visibility scores corresponding to the body key points.
[0064] In some embodiments, step S202 further includes calculating, based on the first object image, a confidence score that the first object image includes the object as the confidence score.
[0065] In some embodiments, the re-identification step further includes: receiving an image to be identified, grouping the image to be identified into a first category of a plurality of identification categories; and based on the first category, outputting a plurality of identified images corresponding to the first category.
[0066] In summary, the re-identification method 200 proposed in the present disclosure can capture an object image containing an object in an input image, and determine whether to use the object image for the re-identification step based on the imaging quality of the object image. In addition, the re-identification device 1 can also set weights for different parts of the object to select images with better imaging quality of specific parts. Furthermore, the re-identification device 1 can also capture multiple object images corresponding to the same object from multiple frames of a video, and select the object image with better imaging quality for the re-identification step.
[0067] Although several embodiments are described above as examples, the re-identification device and method proposed in the present disclosure may also be implemented by other systems, hardware, software, storage media or their combination. Therefore, the protection scope of the present disclosure should not be limited to the specific implementation methods described in the embodiments of the present disclosure, but should be based on the scope defined by the attached claims.
[0068] It is obvious to those with ordinary knowledge in the technical field to which the present disclosure belongs that various modifications and changes can be made to the structure of the present disclosure without departing from the scope or spirit of the present disclosure. In view of the foregoing, the protection scope of the present disclosure also covers the modifications and changes made within the attached claims.
Claims
1. A re-identification device, characterized in that: Include: a storage device; and A processor is coupled to the storage, and is used to perform the following operations: Capturing a first object image corresponding to an object from a first input image; Calculating a confidence score that the first object image corresponds to the object based on the first object image; Based on the first object image, calculating a plurality of visibility scores corresponding to a plurality of key points of the first object image; Based on the confidence score and the visibility scores of the first object image, determining whether to store a first image including the first object image in the storage as one of the plurality of images; and Based on the images stored in the memory, a recognition step is performed.
2. The re-identification device according to claim 1, characterized in that: The operation of determining to store in the storage as one of the images further includes: Calculating a first total score based on the confidence score of the first object image and the visibility scores; and In response to the first total score being greater than a threshold, the first object image is stored in the storage.
3. The re-identification device according to claim 2, characterized in that: The operation of calculating the first total score further includes calculating the first total score based on the confidence score of the first object image, the visibility scores of the first object image, and a plurality of weights corresponding to the confidence score and the visibility scores.
4. The re-identification device according to claim 1, characterized in that: The processor is further configured to perform the following operations: Tracking the object from a second input image to capture a second object image corresponding to the object, wherein a capture time of the second input image is later than the capture time of the first input image; Calculating the confidence score of the second object image corresponding to the object based on the second object image; Based on the second object image, calculating the visibility scores of the second object image corresponding to the key points; and Based on the confidence score of the second object image and the visibility scores, it is determined whether to store a second image including the second object image in the storage as one of the images.
5. The re-identification device according to claim 4, characterized in that: The operation of determining to store in the storage as one of the images includes: Calculate a first total score based on the confidence score of the first object image and the visibility scores; Calculating a second total score based on the confidence score and the visibility scores of the second object image; and The contents of the images to be stored in the storage are determined based on the first total score and the second total score.
6. The re-identification device according to claim 1, characterized in that: The operation of calculating the visibility scores of the key points of the first object image further includes: Determining a position of each of the key points in the first object image, wherein each of the key points corresponds to a portion of the object; and Based on a plurality of pixels at the position of each of the key points in the first object image, one of the visibility scores corresponding to each of the key points is calculated.
7. The re-identification device according to claim 1, characterized in that: The key points include a plurality of facial key points and a plurality of body key points, and the visibility scores include a plurality of facial visibility scores corresponding to the facial key points and a plurality of body visibility scores corresponding to the body key points.
8. The re-identification device according to claim 1, characterized in that: The operation of calculating the confidence score of the first object image corresponding to the object further includes: Based on the first object image, a confidence score of the object being included in the first object image is calculated as the confidence score.
9. The re-identification device according to claim 1, characterized in that: The re-identification step also includes: receiving an image to be recognized, and grouping the image to be recognized into a first category of a plurality of recognition categories; and Based on the first category, a plurality of recognized images corresponding to the first category are output.
10. A re-identification method, characterized in that: Applicable to an electronic device, wherein the electronic device includes a storage device, the re-identification method includes: Capturing a first object image corresponding to an object from a first input image; Calculating a confidence score that the first object image corresponds to the object based on the first object image; Based on the first object image, calculating a plurality of visibility scores corresponding to a plurality of key points of the first object image; Based on the confidence score and the visibility scores of the first object image, determining whether to store a first image including the first object image in the storage as one of the plurality of images; and Based on the images stored in the memory, a recognition step is performed.