Image processing method and device, electronic equipment and medium
By extracting the first feature in the image to generate a character box and generating a second feature for character recognition, the problem of low accuracy caused by the shared features of character detection and re-identification tasks in the prior art is solved, and multi-level in the feature pyramid is realized Feature prediction improves the accuracy of character recognition.
Patent Information
- Application Number
- CN202310531203.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-11
- Publication Date
- 2024-11-12
AI Technical Summary
In the prior art, the sharing of features between character detection and character recognition tasks leads to low detection accuracy, and it is necessary to ensure that the features are aligned on a scale, limiting the accuracy of object detection.
By extracting the first feature in the image, and generating the second feature based on the character box, distinguishing the features used for detection and re-identification, thereby predicting on the multi-level features of the feature pyramid and decoupling the features of the two tasks. , avoid scale alignment restrictions.
It improves the accuracy of character recognition, removes the optimization difficulties caused by feature coupling, and enhances the detection accuracy.
Smart Images

Figure CN118942108A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of image processing, and in particular to an image processing method, device, electronic equipment and medium. Background Art
[0002] Person recognition refers to finding the target person of interest from a series of image libraries. It has important applications in security, video editing, etc. Person recognition includes two subtasks: person detection and person re-identification.
[0003] In the related art, the two tasks of person detection and person re-identification are usually performed separately, but this method consumes a lot of time and resources. In addition, in the single-stage task recognition method in the related art, the person detection and person re-identification tasks share features, which will make it difficult to optimize the detection subtask and the re-identification subtask, resulting in low target detection accuracy. Summary of the invention
[0004] The present application provides an image processing method, device, electronic device and medium, which can improve the accuracy of image recognition.
[0005] The first aspect of the present application discloses an image processing method for an electronic device, the method comprising receiving a plurality of images, each of the plurality of images including at least one person; extracting a first feature from each of the images and generating a person frame based on the first feature; generating a second feature based on the person frame; and determining whether the images are the same person based on the second features of different images.
[0006] In a possible implementation of the first aspect above, the first feature is a point feature.
[0007] In a possible implementation of the first aspect, the second feature is a regional feature.
[0008] In a possible implementation of the first aspect above, generating the second feature based on the character frame includes dynamically sampling the character frame to generate a target frame; and generating the second feature based on the target frame.
[0009] In a possible implementation of the first aspect, extracting the first feature in each image includes extracting the first feature using a convolutional layer.
[0010] The second aspect of the present application discloses an image processing device, which includes a receiving module for receiving multiple images, each of the multiple images including at least one person; a first generating module for extracting a first feature from each of the images and generating a person frame based on the first feature; a second generating module for generating a second feature based on the person frame; and a verification module for determining whether the images are the same person based on the second features of different images.
[0011] In a possible implementation of the second aspect above, the device further includes a sampling module, which is used to dynamically sample the character frame to generate a target frame; and generate the second feature based on the target frame.
[0012] The third aspect of the present application discloses an electronic device, which includes a memory storing computer-executable instructions and a processor; when the instructions are executed by the processor, the device implements the method according to the first aspect of the present application.
[0013] The fourth aspect of the present application discloses a computer-readable medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the method described in the first aspect of the present application.
[0014] The fifth aspect of the present application discloses a computer program product, which implements the method of the first aspect of the present application when the computer program is executed by a processor.
[0015] The image processing method, device, electronic device and medium provided by the present application, after collecting multiple images, first extract the first feature to generate a character frame, then extract the second feature from the character frame, and finally determine whether the different images are the same person based on the second feature. Different from the related art that always extracts features first and then performs classification, regression and retrieval, the present application first performs classification and regression based on the first feature to generate a character frame, and then uses the character frame to extract retrieval features. The present application distinguishes between the first feature used to determine the person and the second feature used for re-identification, and can decouple the features used for the two tasks of person detection and person re-identification in space, without ensuring that the features used for person re-identification are aligned in scale. Therefore, it is not limited to only using the features of the third layer of the feature pyramid, and can make predictions on the multi-level features of the feature pyramid, thereby improving the accuracy of person recognition. BRIEF DESCRIPTION OF THE DRAWINGS
[0016] Figure 1 is a schematic diagram of a person recognition method 100 in the related art;
[0017] Figure 2 is a schematic diagram of a task identification method 200 in the related art;
[0018] Figure 3 A flowchart of an image processing method 300 according to an embodiment of the present application;
[0019] Figure 4 is a schematic diagram of each step in method 300;
[0020] Figure 5 A schematic diagram of a sampling method according to an embodiment of the present application;
[0021] Figure 6 A schematic diagram of an image processing device 600 according to an embodiment of the present application;
[0022] Figure 7 A block diagram of an electronic device 700 according to an embodiment of the present application. DETAILED DESCRIPTION
[0023] The present application is further described below in conjunction with specific embodiments and drawings. It is understood that the illustrative embodiments of the present disclosure include but are not limited to image processing methods, devices, electronic devices and media, and the specific embodiments described herein are only for the purpose of explaining the present application, rather than limiting the present application. In addition, for ease of description, only some, but not all, structures or processes related to the present application are shown in the drawings.
[0024] In order to make the objectives, technical solutions and advantages of the present application clearer, the implementation methods of the present application will be further described in detail below in conjunction with the accompanying drawings.
[0025] Person recognition refers to finding the target person of interest from a series of image libraries. It has important applications in intelligent security, video intelligent editing, etc. Person recognition includes two subtasks: person detection and person re-identification. Person detection locates all human targets in the scene, while person re-identification finds the person who is most similar to the person to be retrieved.
[0026] Person Re-identification (ReID) is a technology that uses computer vision technology to determine whether a specific pedestrian exists in an image or video sequence. Specifically, pedestrians are retrieved under a multi-camera network, and information such as gait movements and body features are used to identify people. ReID can be used alone or in combination with face recognition technology. ReID mainly relies on the overall posture of pedestrians for retrieval, that is, people are identified by analyzing the pedestrian's clothing and body shape. For example, people are identified by extracting static external features, such as clothing, hairstyle, and items carried with them (for example, backpacks, umbrellas), etc.
[0027] Figure 1 A schematic diagram of a person recognition method 100 in the related art is shown. Figure 1The person recognition method 100 in the embodiment performs two tasks, person detection and person re-identification, separately. That is, firstly, the input image 110 is cut out by the person detection model 120, and then the output of the person detection model 120 is input to the ReID network 130 for person re-identification, and the result 140 is output. The method 100 can achieve a better recognition effect, but the method consumes a lot of time and resources.
[0028] Figure 2 A schematic diagram of another task identification method 200 in the related art is shown, which can improve the efficiency of person identification. The method 200 provides an end-to-end single-stage task identification method, which uses the same deep network to implement two tasks of person detection and person re-identification.
[0029] The method 200 is based on the FCOS (Fully Convolutional One-Stage Object Detection) framework. After the image 210 is input, the classification branch 230 and the regression branch 240 are connected after the feature pyramid 220. Among them, the classification branch is used to determine whether it is a person, and the regression branch 240 is used to generate a person frame. In order to improve the accuracy of person recognition, the method 200 places the features 250 of person re-identification before detection and classification, and only uses the features of the third layer of the feature pyramid to ensure that the features of person re-identification are aligned in scale.
[0030] However, in method 200, both person detection 230 and person re-identification 240 use the features of the person's center point for prediction, and the person's center point is, for example, the waist of a person. However, person detection focuses on the commonalities of people, while person re-identification focuses on the differences of people. If the features of the shared center point will lead to coupling of features, it will bring difficulties to the optimization of the detection subtask and the re-identification subtask. In addition, since it is necessary to ensure that the features of person re-identification are aligned in scale, the target detection in method 200 can only be performed on the third layer of the feature pyramid, which leads to a decrease in the accuracy of target detection.
[0031] In order to solve the above problems, an embodiment of the present application provides an image processing method 300. Figure 3 . Figure 3 The steps are as follows.
[0032] In S310 , a plurality of images are received, each of the plurality of images including at least one person.
[0033] It can receive images captured by one or more cameras, or obtain images of different frames from a video, which include one or more people. For example, in a security scenario, it can obtain images of pedestrians in a network of multiple cameras located in different locations. In a video editing scenario, it can obtain images of people in different frames.
[0034] In S320 , a first feature is extracted from each image, and a person frame is generated based on the first feature.
[0035] A first feature is extracted from each acquired picture to identify a person. In some embodiments, the first feature is a point feature. For example, the first feature may be a feature of a center point of a person, and the center point of a person may be a waist. After obtaining the first feature, a person frame may be generated based on the first feature.
[0036] In some implementations, a convolutional layer may be used to extract the first feature.
[0037] At S330 , a second feature is generated based on the character frame.
[0038] A second feature different from the first feature is obtained based on the character frame. The second feature is a character feature. In some implementations, the second feature is a region feature.
[0039] In S340, it is determined whether the images are of the same person based on the second features of the different images.
[0040] Whether the characters in different images are the same person can be determined based on the second features from different images. In some embodiments, a retrieval feature, i.e., a Re-ID feature, can be generated based on the second feature, and then whether the characters are the same person can be determined based on the Re-ID feature.
[0041] Figure 3 In the image processing method 300, after collecting multiple images, the first feature is first extracted to generate a character frame, and then the second feature is extracted from the character frame, and finally whether the same person is in different images is determined based on the second feature. Different from the related art that always extracts features first and then performs classification, regression, and retrieval, the method 300 first performs classification and regression based on the first feature to generate a character frame, and then uses the character frame to extract retrieval features. The method 300 distinguishes between the first feature used to determine the person and the second feature used for re-identification, and can decouple the features used for the two tasks of person detection and person re-identification in space. There is no need to ensure that the features used for person re-identification are aligned in scale. Therefore, it is not limited to only using the features of the third layer of the feature pyramid, and can make predictions on the multi-level features of the feature pyramid, thereby improving the accuracy of person recognition.
[0042] To better illustrate method 300, Figure 4The corresponding method schematic diagram is shown. Figure 4 In the example, the image 410 is subjected to feature extraction in the convolution layer 420 and connected to the classification branch 430 and the regression branch 440. The classification branch 430 is used to determine whether it is a person according to the first feature, and the regression branch 440 is used to generate a person frame according to the first feature. Based on the person frame 460, the ReID feature 450 is obtained. Figure 4 It is shown in the figure that the features used for the two tasks of person detection and person re-identification in method 300 are spatially decoupled, and method 300 first performs classification and regression to generate a person frame, and then uses the person frame to extract retrieval features.
[0043] In some implementations, generating the second feature based on the character frame includes dynamically sampling the character frame to generate a target frame; and generating the second feature based on the target frame.
[0044] Usually, the four coordinates of the target output by the regression branch are b = (x, y, w, h). In this application, two scaling factors f can be used. w ,f h Generate a target frame. That is, determine the target frame b related to the person according to the scaling factor * =(x,y,w×f w ,h×f h ). Retrieval features can be generated based on the target box.
[0045] For example, Figure 5 A sampling example of an embodiment of the present application is shown. Figure 5 In the input image 510, there are two people 511 and 512. The frame line part in 510 is a person frame 520 represented by (x, y, w, h). Usually, the person frame is the feature extraction area in the target detection method. By setting, two additional scaling factors are generated to obtain the dotted frame (x, y, w×f w ,h×f h )530.
[0046] When comparing with image 540, if the full frame sampling method is used, that is, person frame 520 is used, it is easy to determine the person in person frame 520 as person 541 in image 540. If the dynamic sampling method is used, the person in person frame 530 can be correctly determined as person 542 in image 540. Figure 5 It can be seen that the dynamic sampling method is closer to the character boundary and can avoid the interference caused by the character background.
[0047] Figure 6 FIG. 6 is a schematic diagram of an image processing device 600 according to an embodiment of the present application. The device 600 includes the following modules.
[0048] A receiving module 610 is configured to receive a plurality of images, each of the plurality of images including at least one person;
[0049] A first generating module 620, configured to extract a first feature from each image and generate a character frame based on the first feature;
[0050] A second generating module 630, configured to generate a second feature based on the character frame;
[0051] The verification module 640 is used to determine whether the images are of the same person according to the second features of different images.
[0052] In some implementations, the device 600 further includes a sampling module, which is used to dynamically sample the character frame to generate a target frame; and generate a second feature based on the target frame.
[0053] Reference now Figure 7 , which is a block diagram of an electronic device 700 according to an embodiment of the present application. The device 700 may include one or more processors 702, a system control logic 708 connected to at least one of the processors 702, a system memory 704 connected to the system control logic 708, a non-volatile memory (NVM) 706 connected to the system control logic 708, and a network interface 710 connected to the system control logic 708.
[0054] The processor 702 may include one or more single-core or multi-core processors. The processor 702 may include any combination of general-purpose processors and special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In the embodiments of the present invention, the processor 702 may be configured to execute the following operations: Figure 3-6 One or more of the various embodiments shown.
[0055] In some embodiments, system control logic 708 may include any suitable interface controller to provide any suitable interface to at least one of processors 702 and / or any suitable device or component in communication with system control logic 708 .
[0056] In some embodiments, the system control logic 708 may include one or more memory controllers to provide an interface to the system memory 704. The system memory 704 may be used to load and store data and / or instructions. In some embodiments, the memory 704 of the device 700 may include any suitable volatile memory, such as a suitable dynamic random access memory (DRAM).
[0057] NVM / memory 706 may include one or more tangible, non-transitory computer-readable media for storing data and / or instructions. In some embodiments, NVM / memory 706 may include any suitable non-volatile memory such as flash memory and / or any suitable non-volatile storage device, such as at least one of a HDD (Hard Disk Drive), a CD (Compact Disc) drive, and a DVD (Digital Versatile Disc) drive.
[0058] NVM / storage 706 may include a portion of storage resources installed on a device of apparatus 700, or it may be accessible to the apparatus but not necessarily a part of the apparatus. For example, NVM / storage 706 may be accessed over a network via network interface 710.
[0059] In particular, system memory 704 and NVM / storage 706 may include, respectively, a temporary copy and a permanent copy of instructions 720. Instructions 720 may include instructions that, when executed by at least one of processors 702, cause device 700 to perform the following operations: Figure 3-5 In some embodiments, instructions 720, hardware, firmware and / or software components thereof may additionally / alternatively be placed in system control logic 708, network interface 710 and / or processor 702.
[0060] The network interface 710 may include a transceiver for providing a radio interface for the device 700 to communicate with any other suitable device (such as a front-end module, an antenna, etc.) through one or more networks. In some embodiments, the network interface 710 may be integrated with other components of the device 700. For example, the network interface 710 may be integrated with at least one of the processor 702, the system memory 704, the NVM / storage 706, and a firmware device (not shown) having instructions. When at least one of the processors 702 executes the instructions, the device 700 implements Figure 3-6 One or more of the various embodiments shown.
[0061] The network interface 710 may further include any suitable hardware and / or firmware to provide a multiple-input multiple-output radio interface. For example, the network interface 710 may be a network adapter, a wireless network adapter, a telephone modem and / or a wireless modem.
[0062] In one embodiment, at least one of the processors 702 may be packaged together with logic for one or more controllers of the system control logic 708 to form a system in package (SiP). In one embodiment, at least one of the processors 702 may be integrated on the same die with logic for one or more controllers of the system control logic 708 to form a system on a chip (SoC).
[0063] The device 700 may further include an input / output (I / O) device 712. The I / O device 712 may include a user interface to enable a user to interact with the device 700; the design of the peripheral component interface enables the peripheral components to interact with the device 700. In some embodiments, the device 700 further includes a sensor for determining at least one of an environmental condition and location information related to the device 700.
[0064] In some embodiments, the user interface may include, but is not limited to, a display (e.g., an LCD display, a touch screen display, etc.), a speaker, a microphone, one or more cameras (e.g., a still image camera and / or a video camera), a flashlight (e.g., an LED flash), and a keyboard.
[0065] In some embodiments, the peripheral component interface may include, but is not limited to, a non-volatile memory port, an audio jack, and a power interface.
[0066] In some embodiments, the sensors may include, but are not limited to, gyroscope sensors, accelerometers, proximity sensors, ambient light sensors, and positioning units. The positioning unit may also be part of or interact with the network interface 710 to communicate with components of a positioning network (e.g., global positioning system (GPS) satellites).
[0067] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the Bluetooth communication device 700. In other embodiments of the present application, the Bluetooth communication device 700 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0068] Program code can be applied to input instructions to perform the functions described herein and generate output information. The output information can be applied to one or more output devices in a known manner. For purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.
[0069] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described herein is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.
[0070] One or more aspects of at least one embodiment may be implemented by representative instructions stored on a computer-readable storage medium, which represent various logic in a processor, and when the instructions are read by a machine, cause the machine to fabricate logic for performing the techniques described herein. These representations, known as "IP cores," may be stored on a tangible computer-readable storage medium and supplied to multiple customers or manufacturing facilities to load into a manufacturing machine that actually manufactures the logic or processor.
[0071] An embodiment of the present application discloses a computer-readable medium storing one or more programs executable by one or more processors to implement the method of the present application.
[0072] An embodiment of the present application discloses a computer program product, including a computer program, which implements the method of the present application when executed by a processor.
[0073] The above is a description of the implementation mode of the present application by specific specific embodiments. Those skilled in the art can easily understand other advantages and effects of the present application from the contents disclosed in this specification. Although the description of the present application will be introduced in conjunction with the preferred embodiment, this does not mean that the features of this invention are limited to this implementation mode. On the contrary, the purpose of introducing the invention in conjunction with the implementation mode is to cover other options or modifications that may be extended based on the claims of the present application. In order to provide an in-depth understanding of the present application, the following description will include many specific details. The present application can also be implemented without using these details. In addition, in order to avoid confusion or blurring the focus of the present application, some specific details will be omitted in the description. It should be noted that the embodiments in the present application and the features in the embodiments can be combined with each other without conflict.
[0074] Furthermore, various operations will be described as multiple discrete operations in a manner that is most helpful for understanding the illustrative embodiments; however, the order of description should not be construed as implying that these operations are necessarily order dependent. In particular, these operations do not need to be performed in the order presented.
[0075] Unless the context dictates otherwise, the terms "comprising," "having," and "including" are synonymous. The phrase "A / B" means "A or B." The phrase "A and / or B" means "(A and B) or (A or B)."
[0076] As used herein, the term "module" or "unit" may refer to, be or include: an application specific integrated circuit (ASIC), an electronic circuit, a (shared, dedicated or group) processor and / or memory that executes one or more software or firmware programs, a combinational logic circuit and / or other suitable components that provide the described functionality.
[0077] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or by other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, but not limited to, floppy disks, optical disks, optical disks, read-only memories (CD-ROMs), magneto-optical disks, read-only memories (ROMs), random access memories (RAMs), erasable programmable read-only memories (EPROMs), electrically erasable programmable read-only memories (EEPROMs), magnetic or optical cards, flash memory, or a tangible machine-readable memory for transmitting information through the Internet via electrical, optical, acoustic, or other forms of propagation signals (e.g., carrier waves, infrared signals, digital signals, etc.). Therefore, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a machine (e.g., computer) readable form.
[0078] In the accompanying drawings, some structural or method features are shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. In some embodiments, these features may be arranged in a manner and / or order different from that shown in the illustrative drawings. In addition, the inclusion of structural or method features in a specific figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.
[0079] It should be understood that although the terms "first", "second", etc. may be used here to describe various units or data, these units or data should not be limited by these terms. These terms are used only to distinguish one feature from another. For example, without departing from the scope of the exemplary embodiment, a first feature may be referred to as a second feature, and similarly, a second feature may be referred to as a first feature.
[0080] It should be noted that in this specification, similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, it does not need to be further defined and explained in the subsequent drawings.
[0081] While the present invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention.
Claims
1. An image processing method, used in electronic equipment, characterized in that: The method comprises, receiving a plurality of images, each of the plurality of images including at least one person; Extracting a first feature from each of the images, and generating a character frame based on the first feature; generating a second feature based on the character frame; Determine whether the images are the same person based on the second feature of the different images.
2. The method according to claim 1, characterized in that: The first feature is a point feature.
3. The method according to claim 2, characterized in that The second feature is a regional feature.
4. The method according to claim 1, characterized in that Generating a second feature based on the character frame includes: Dynamically sampling the character frame to generate a target frame; as well as The second feature is generated based on the target box.
5. The method according to claim 1, characterized in that Extracting the first feature in each of the images includes extracting the first feature using a convolutional layer.
6. An image processing device, characterized in that: The device comprises, A receiving module, configured to receive a plurality of images, each of the plurality of images including at least one person; A first generating module, configured to extract a first feature from each of the images and generate a character frame based on the first feature; A second generating module, used for generating a second feature based on the character frame; A verification module is used to determine whether the different images are of the same person based on the second feature of the different images.
7. The image processing device according to claim 6, characterized in that: The device further comprises: A sampling module is used to dynamically sample the character frame to generate a target frame; and generate the second feature based on the target frame.
8. An electronic device, characterized in that: The device comprises a memory storing computer executable instructions and a processor; when the instructions are executed by the processor, the device implements the method according to any one of claims 1 to 5.
9. A computer-readable medium, characterized in that The storage medium stores one or more programs, and the one or more programs can be executed by one or more processors to implement the method according to any one of claims 1 to 5.
10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method according to any one of claims 1 to 5 is implemented.
