Fusion deduplication method, device and readable medium based on multi-camera captured images

Abstract

The present invention discloses a fusion deduplication method, device and readable medium based on images captured by multiple cameras. The method comprises the following steps: acquiring images captured by multiple cameras respectively arranged around a channel, respectively identifying a head image, a face image and a body image from the images captured by each camera, and obtaining a quality score corresponding to the face image; performing head tracking based on the head image acquired by each camera to obtain a head trajectory, and determining a main person passing through the channel according to the order in which the same person in the head trajectory appears in a personnel candidate area and a personnel confirmation area of ​​the channel; obtaining the best body image and the best face image of the main person under each camera based on the main person and his head trajectory; deduplicating and merging the best body images of the main person captured by all cameras through a ReID algorithm, determining a target person, and determining a target face image based on the quality score of the best face image of the target person, thereby improving the accuracy of capturing.

Description

Technical Field

[0001] The present invention relates to the field of computer vision, and in particular to a method, device and readable medium for fusion deduplication of images captured by multiple cameras. Background Art

[0002] Security checkpoints at airports, high-speed rail stations, and other locations; security checkpoints at fixed venues like museums, libraries, and concert halls; and security checkpoints at temporary venues like lantern festivals and music festivals. When people pass through these checkpoints, facial or body images are required for security purposes. As security levels increase, passengers in airports, railway stations, and other locations are required to verify their identities. Each passenger's identity must be verified as they pass through baggage security checkpoints, and special attention must be paid to those who display warning signs.

[0003] Passengers unload their luggage at the baggage inspection lane and pass through the security gate. During this process, each passenger's face must be captured and any key individuals identified. A single camera can capture these images in two ways, each with its own challenges:

[0004] 1. Take a bird's-eye view shot:

[0005] A single camera, mounted atop the security gate, is forced to use a bird's-eye view to capture people passing through. However, when people are of an inappropriate height or bow their heads as they pass through the gate, this bird's-eye view makes it difficult to easily identify faces, resulting in a high number of missed detections. Furthermore, the top-mounted camera compromises aesthetics and can create negative emotions for passengers.

[0006] 2. Side view snapshot:

[0007] A single camera is placed on the side of the aisle, behind the unloading area, facing passengers unloading their luggage and capturing them as they do so. However, passengers often lower their heads when unloading their luggage, so current side-loading solutions miss many instances. Capturing passengers of both short and tall stature is also ineffective.

[0008] In the existing technology, face detection is mostly done in a segmented manner, first detecting the face, then estimating the face angle, and selecting the front face for recognition. Therefore, this process is cumbersome and inefficient. Summary of the Invention

[0009] In response to the technical problems mentioned above such as missed recognition or poor capture effect caused by a single camera, the embodiment of this application aims to propose a method, device and readable medium for fusion deduplication based on multi-camera captured images to solve the technical problems mentioned in the background technology section above.

[0010] In a first aspect, an embodiment of the present application provides a method for deduplication based on fusion of images captured by multiple cameras, comprising the following steps:

[0011] S1, acquiring images captured by multiple cameras set up around the channel, identifying the head image, face image, and body image from the images captured by each camera, and obtaining the quality score corresponding to the face image, which is determined based on the pitch angle, roll angle, and yaw angle of the face;

[0012] S2: Based on the head images captured by each camera, head tracking is performed to obtain a head trajectory, and the main person passing through the passage is determined based on the order in which the same person in the head trajectory appears in the candidate area and the confirmation area of ​​the passage;

[0013] S3, based on the subject and its head trajectory, obtaining the best human body image and the best face image of the subject captured by each camera;

[0014] S4, using the ReID algorithm to de-duplicate and merge the best human body images of the subject person captured by all cameras, determine the target person, and determine the target face image based on the quality score of the best face image corresponding to the target person.

[0015] In a specific embodiment, step S1 further includes:

[0016] A standard 3D facial key point model is constructed and deflected to obtain the pitch angle, roll angle and yaw angle and their corresponding image 2D plane coordinates of the facial key points, and a regression model based on the image 2D plane coordinates to the pitch angle, roll angle and yaw angle is established.

[0017] Establish a training set. In the training set, the 2D plane coordinates of the facial key points in the image are detected by the standard 3D facial key point model, and the pitch angle, roll angle and yaw angle corresponding to the 2D plane coordinates of the facial key point coordinates in the training set are calculated. After the pitch angle, roll angle and yaw angle are known, the labels of the facial images with angles less than the preset threshold are recorded as high-quality face labels, and the labels of the facial images with angles greater than the threshold are recorded as low-quality face labels. All face frames in the training set are added with high-quality face labels or low-quality face labels, and the regression model is trained using the training set. , and obtain a trained regression model; the training set also contains head frames, body frames, and face frames corresponding to head images, body images, and face images respectively, and establishes an image detection and evaluation model. The image detection and evaluation model includes an input layer, a trained regression model, and a detector. The image detection and evaluation model is trained using the training set. The image detection and evaluation model is used to detect head images, body images, and face images and evaluate to obtain the quality scores corresponding to the face images. The quality scores are directly regressed and output from the image detection and evaluation model, and the quality scores are calculated from the pitch angle, roll angle, and yaw angle of the face image.

[0018] In a specific embodiment, step S1 identifies a head image, a face image, and a body image from the images captured by each camera, and obtains a quality score corresponding to the face image, specifically including:

[0019] The images captured by each camera are input into the image detection and evaluation model, which outputs the head image, face image, and body image, and displays the corresponding quality score on the face image.

[0020] In a specific embodiment, step S2 performs head tracking based on the head image acquired by each camera to obtain the head trajectory, specifically including:

[0021] Track all heads in the head images captured by each camera to obtain all head trajectories of each camera;

[0022] Cache the human body images and face images of all points on all head trajectories of each camera.

[0023] In a specific embodiment, step S2 determines the main person passing through the passage based on the order in which the same person in the head trajectory appears in the personnel candidate area and the personnel confirmation area of ​​the passage, specifically including:

[0024] Select the personnel candidate area and personnel confirmation area of ​​the channel from the head image captured by each camera;

[0025] Based on the fact that a person in the head trajectory appears in the personnel candidate area and the personnel confirmation area successively within the target time period, the person is determined to be the main person.

[0026] In a specific embodiment, step S3 specifically includes:

[0027] The face image with the best quality score is obtained from the face images of each point on the head trajectory of the subject person cached by each camera as the best face image;

[0028] The best human image is the human image with the most complete human features of similar size and position among the human images at each point on the head trajectory of the subject person cached by each camera.

[0029] In a specific embodiment, step S4 specifically includes:

[0030] Summarize the best human body image and best face image of the subject corresponding to each camera;

[0031] The ReID algorithm is used to extract human features from the best human images. The similarity of human features in the best human images of the subject captured by each two cameras is calculated to obtain a similarity matrix, and the mean of the similarity matrix is ​​calculated.

[0032] The human features of the best human image of the subject person corresponding to each camera are used as anchor points. In response to the similarity between one of the anchor points and the other anchor points being greater than the mean value of the similarity matrix, the subject person captured by one of the cameras and the other cameras is determined to be in the same cluster, thereby obtaining multiple clusters.

[0033] Recursively traverse all clusters and classify them to obtain multiple cluster sets;

[0034] The principal person in the cluster set with the largest number of clusters among the multiple cluster sets is taken as the target person;

[0035] The best face image with the best quality score among the best face images corresponding to the target person is used as the target face image.

[0036] In a specific embodiment, the plurality of cameras include six cameras located at the upper, middle and lower parts of the left and right sides of the channel respectively.

[0037] In a second aspect, an embodiment of the present application provides a fusion deduplication device based on multi-camera captured images, comprising:

[0038] a quality score calculation module configured to acquire images captured by multiple cameras disposed around the channel, identify a head image, a face image, and a body image from the images captured by each camera, and obtain a quality score corresponding to the face image, the quality score being determined based on the pitch angle, roll angle, and yaw angle of the face;

[0039] The main person identification module is configured to track the head based on the head image obtained by each camera to obtain the head trajectory, and determine the main person passing through the passage according to the order in which the same person in the head trajectory appears in the candidate area and the confirmation area of ​​the passage;

[0040] An optimal image acquisition module is configured to acquire an optimal human body image and an optimal face image of the subject person captured by each camera based on the subject person and the head trajectory;

[0041] The target image acquisition module is configured to de-duplicate and merge the best human body images of the subject person captured by all cameras through the ReID algorithm, determine the target person, and determine the target face image based on the quality score of the best face image corresponding to the target person.

[0042] In a third aspect, an embodiment of the present application provides an electronic device comprising one or more processors; a storage device for storing one or more programs, wherein when the one or more programs are executed by one or more processors, the one or more processors implement the method described in any implementation manner in the first aspect.

[0043] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any implementation manner in the first aspect.

[0044] Compared with the prior art, the present invention has the following beneficial effects:

[0045] (1) The present invention's method for deduplication based on fusion of multi-camera captured images detects faces from camera captured images and directly outputs a face quality score. The quality score can be used to determine whether the face is easily recognizable. Compared with existing face detection technologies, this method can directly determine whether the face is easily recognizable.

[0046] (2) The present invention's fusion deduplication method based on multi-camera captured images can identify the main person passing through the security gate from multiple people in the captured image, and then send the main person's best human image and best person image for identification. Compared with existing captured image technology, it will not confuse the passenger passing through the security gate with other passengers; and by capturing the front face image from the facial images cached in the trajectory, the most frontal face of the entire process can be obtained; the selection of human body images ensures that all human body images have similar sizes and interception locations.

[0047] (3) The present invention can remove duplicates through the ReID algorithm and subject merging logic, obtain the most positive face image in images captured by multiple cameras, expand the selection effect, and at the same time, remove duplicate images from different cameras and eliminate duplicate reporting. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the technical solutions in the embodiments of the present invention, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0049] Figure 1 is a diagram of an exemplary device architecture to which an embodiment of the present application may be applied;

[0050] Figure 2 Schematic diagram of the process of a method for fusion deduplication based on multi-camera captured images according to an embodiment of the present invention;

[0051] Figure 3 A schematic diagram of camera installation according to a method for fusion deduplication of images captured by multiple cameras according to an embodiment of the present invention;

[0052] Figure 4 Schematic diagram of a standard 3D facial key point model based on a fusion deduplication method for multi-camera captured images according to an embodiment of the present invention;

[0053] Figure 5 A schematic diagram of a candidate personnel area according to a method for fusion and deduplication of images captured by multiple cameras according to an embodiment of the present invention;

[0054] Figure 6 A schematic diagram of a personnel confirmation area based on a fusion and deduplication method of images captured by multiple cameras according to an embodiment of the present invention;

[0055] Figure 7 Schematic diagram of a fusion deduplication device based on multi-camera captured images according to an embodiment of the present invention;

[0056] Figure 8 It is a structural diagram of a computer device suitable for implementing the electronic device of the embodiment of the present application. DETAILED DESCRIPTION

[0057] To make the objectives, technical solutions, and advantages of the present invention more apparent, the present invention will be further described in detail below with reference to the accompanying drawings. It is apparent that the embodiments described are only some, not all, of the present invention. All other embodiments derived by persons of ordinary skill in the art based on the embodiments of the present invention without creative effort are intended to fall within the scope of protection of the present invention.

[0058] Figure 1 An exemplary device architecture 100 is shown to which a method for fusion deduplication based on images captured by multiple cameras or a device for fusion deduplication based on images captured by multiple cameras according to an embodiment of the present application can be applied.

[0059] like Figure 1 As shown, the device architecture 100 may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0060] Users can use terminal devices 101, 102, 103 to interact with server 105 via network 104 to receive or send messages, etc. Various applications, such as data processing applications and file processing applications, can be installed on terminal devices 101, 102, 103.

[0061] Terminal devices 101, 102, and 103 can be hardware or software. When terminal devices 101, 102, and 103 are hardware, they can be various electronic devices, including but not limited to smartphones, tablet computers, laptop computers, and desktop computers. When terminal devices 101, 102, and 103 are software, they can be installed in the electronic devices listed above. They can be implemented as multiple software or software modules (for example, software or software modules used to provide distributed services), or they can be implemented as a single software or software module. No specific limitations are given here.

[0062] The server 105 may be a server that provides various services, such as a background data processing server that processes files or data uploaded by the terminal devices 101, 102, and 103. The background data processing server may process the acquired files or data and generate processing results.

[0063] It should be noted that the fusion deduplication method based on multi-camera captured images provided in the embodiment of the present application can be executed by the server 105, or by the terminal devices 101, 102, and 103. Accordingly, the fusion deduplication device based on multi-camera captured images can be set in the server 105, or in the terminal devices 101, 102, and 103.

[0064] It should be understood that Figure 1 The number of terminal devices, networks, and servers in the above description is merely illustrative. Any number of terminal devices, networks, and servers may be provided as needed. If the processed data does not need to be acquired remotely, the above-described apparatus architecture may not include a network, but only require servers or terminal devices.

[0065] Figure 2 The embodiment of the present application provides a method for fusion deduplication based on multi-camera captured images, including the following steps:

[0066] S1, obtains and collects images captured by multiple cameras set up around the channel, identifies the head image, face image and body image from the images captured by each camera, and obtains the quality score corresponding to the face image. The quality score is determined based on the pitch angle, roll angle and yaw angle of the face.

[0067] Specifically, the channels mentioned in the embodiments of the present application include but are not limited to: security channels in front of security gates such as airports / high-speed rail stations, security channels in fixed gathering places such as museums / libraries / concert halls, and security gates in temporary venues such as lantern festivals and music festivals. When people pass through these security channels, if there is a security need, it is required to record facial or body images. The camera of the present invention is placed outside the security gate. When someone passes by, the face is captured for security identification. If the face cannot be recognized, the image of the person will be transmitted to prompt the security inspector. In addition, the embodiments of the present application can also be used in other security and missed capture scenarios to solve the problem of missed capture under the non-sensing capture of a single camera. In a preferred embodiment, if Figure 3 As shown, the multiple cameras installed around the passageway include six cameras located at the top, middle, and bottom of the left and right sides of the passageway, respectively. These cameras are assembled vertically, with three cameras on each side, enabling capture of people of varying heights. In other optional embodiments, the cameras can be located at other suitable locations near the passageway, and the number of cameras can be other, such as four or eight. The following example uses six cameras installed at the top, middle, and bottom of the left and right sides of the security gate as an example.

[0068] In a specific embodiment, step S1 further includes:

[0069] A standard 3D facial key point model is constructed and deflected to obtain the pitch angle, roll angle and yaw angle and their corresponding image 2D plane coordinates of the facial key points, and a regression model based on the image 2D plane coordinates to the pitch angle, roll angle and yaw angle is established.

[0070] Establish a training set. In the training set, the 2D plane coordinates of the facial key points in the image are detected by the standard 3D facial key point model, and the pitch angle, roll angle and yaw angle corresponding to the 2D plane coordinates of the facial key point coordinates in the training set are calculated. After the pitch angle, roll angle and yaw angle are known, the labels of the facial images with angles less than the preset threshold are recorded as high-quality face labels, and the labels of the facial images with angles greater than the threshold are recorded as low-quality face labels. All face frames in the training set are added with high-quality face labels or low-quality face labels, and the regression model is trained using the training set. , and obtain a trained regression model; the training set also contains head frames, body frames, and face frames corresponding to head images, body images, and face images respectively, and establishes an image detection and evaluation model. The image detection and evaluation model includes an input layer, a trained regression model, and a detector. The image detection and evaluation model is trained using the training set. The image detection and evaluation model is used to detect head images, body images, and face images and evaluate to obtain the quality scores corresponding to the face images. The quality scores are directly regressed and output from the image detection and evaluation model, and the quality scores are calculated from the pitch angle, roll angle, and yaw angle of the face image.

[0071] Specifically, the standard 3D face key point model is as follows Figure 4 As shown, by deflecting a standard 3D facial landmark model, different pitch, roll, and yaw angles are obtained, along with the corresponding 2D image coordinates (p, r, y) of the facial landmarks. The 2D image coordinates of the facial landmarks correspond one-to-one with the pitch, roll, and yaw angles of the facial model. The pitch, roll, and yaw angles of the facial model are compared with a preset threshold to obtain a face quality label. Faces above the preset threshold are labeled with a high-quality face label, while faces below the preset threshold are labeled with a low-quality face label. This results in a training set for a regression model based on the image 2D plane coordinates to pitch, roll, and yaw angles. The trained regression model takes the 2D image coordinates of the facial landmarks as input and outputs either a high-quality face label or a low-quality face label.

[0072] Furthermore, the training set also includes head frames, body frames, and face frames, corresponding to head images, body images, and face images, respectively. Based on the face quality labels and the head, body, and face frames in the training set, an image detection and evaluation model is trained to detect head frames, body frames, and face frames and output a quality score for the face image. The image detection and evaluation model consists of an input layer, a trained regression model, and a detector. The quality score is directly regressed from the image detection and evaluation model.

[0073] In a specific embodiment, step S1 identifies a head image, a face image, and a body image from the images captured by each camera, and obtains a quality score corresponding to the face image, specifically including:

[0074] The images captured by each camera are input into the image detection and evaluation model, and the images captured by each camera are input into the image detection and evaluation model, and the head image, face image, and body image are output respectively, and the quality score corresponding to the face image is obtained, and then the corresponding quality score is displayed on the face image.

[0075] Specifically, the image detection and evaluation model in the embodiment of the present application can not only detect head images, face images and body images, but also use the self-labeling method to add quality scores to all faces in the image and directly estimate the recognizability of the face. Due to the existence of the trained regression model and the combination of the detector, the quality score corresponding to the face image can be detected. If the quality score is Figure 4 The values ​​0.1, 0.5, and 1.0 on the image shown are quality scores. When the quality score is 1.0, the face in the image is the marked front face.

[0076] S2, based on the head image obtained by each camera, head tracking is performed to obtain a head trajectory, and the main person passing through the channel is determined according to the order in which the same person in the head trajectory appears in the personnel candidate area and the personnel confirmation area of ​​the channel.

[0077] In a specific embodiment, step S2 performs head tracking based on the acquired head image to obtain a head trajectory, specifically including:

[0078] Track all heads in the head images captured by each camera to obtain all head trajectories of each camera;

[0079] Cache the human body images and face images of all points on all head trajectories of each camera.

[0080] In a specific embodiment, step S2 determines the main person passing through the passage based on the order in which the same person in the head trajectory appears in the personnel candidate area and the personnel confirmation area of ​​the passage, specifically including:

[0081] Select the personnel candidate area and personnel confirmation area of ​​the channel from the head image captured by each camera;

[0082] Based on the fact that a person in the head trajectory appears in the personnel candidate area and the personnel confirmation area successively within the target time period, the person is determined to be the main person.

[0083] Specifically, in step S2, there is no order between the judgment of the subject person and the caching of the human body image and the face image in each head track of each camera. The two can be performed simultaneously or sequentially, and there is no restriction. The head track is obtained by tracking the head image. The head image contains face information. Therefore, face information can be obtained while using the head image for track tracking, which is more accurate. Among them, the judgment of the subject person is mainly based on the selection of the subject person in the field of view captured by the camera according to the head track. The subject person is the person who passes through the security gate. Reference Figure 5 and Figure 6 First, we define regions of interest (ROIs) in the images captured by the six cameras. ROI1 is the candidate area for passengers passing through the security checkpoint, and ROI2 is the confirmation area for passengers passing through the security checkpoint.

[0084] When a head track first appears in ROI1, it is set as a candidate for the passenger passing through the security gate.

[0085] When a head track appears in ROI2 for the first time, if it has already appeared in ROI1, it is determined that it is the subject person, that is, the person who has passed the security gate.

[0086] Therefore, the six cameras correspond to six main persons, and these six main persons may be the same person or different persons.

[0087] After obtaining the head trajectory, the corresponding human body image and facial image can be cached based on each head trajectory of each camera. Then, based on the subject person identified in each camera, the human body image and facial image of the subject person can be determined from all cached human body images and facial images, thereby obtaining the best human body image and facial image of the subject person. Alternatively, the human body image and facial image of the subject person can be directly cached after the subject person is determined.

[0088] S3, based on the subject person and the head trajectory, obtaining the best body image and the best face image of the subject person captured by each camera.

[0089] In a specific embodiment, step S3 specifically includes:

[0090] The face image with the best quality score is obtained from the face images of each point on the head trajectory of the subject person cached by each camera as the best face image;

[0091] The best human image is the one that contains the most complete human features of similar size and position among the human images at each point on the head trajectory of the subject person cached by each camera.

[0092] Specifically, the best facial image can be the facial image with the best quality score among the facial images of the subject person found in step S2. For example, taking the frontal face score as an example, the facial image with the highest frontal face score is selected as the best facial image. The best human body image selected must include complete human body features for human body feature similarity calculation and comparison in step S4. When selecting human body images, ensure that all human body images have similar sizes and cropped parts. In a preferred embodiment, taking the capture of the upper body as an example, the upper body image in which the upper edge of the human head frame of the human body image coincides with the upper edge of the human body image or the lower edge of the human body frame of the human body image coincides with the lower edge of the human body image is selected as the best human body image. The best facial images and the best human body images obtained from the 6 cameras are fused and deduplicated to obtain the target person and the target facial image.

[0093] S4, using the ReID algorithm to de-duplicate and merge the best human body images of the subject person captured by all cameras, determine the target person, and determine the target face image based on the quality score of the best face image corresponding to the target person.

[0094] In a specific embodiment, step S4 specifically includes:

[0095] Summarize the best human body image and best face image of the subject corresponding to each camera;

[0096] The ReID algorithm is used to extract human features from the best human images. The similarity of human features in the best human images of the subject captured by each two cameras is calculated to obtain a similarity matrix, and the mean of the similarity matrix is ​​calculated.

[0097] The human features of the best human image of the subject person corresponding to each camera are used as anchor points. In response to the similarity between one of the anchor points and the other anchor points being greater than the mean value of the similarity matrix, the subject person captured by one of the cameras and the other cameras is determined to be in the same cluster, thereby obtaining multiple clusters.

[0098] Recursively traverse all clusters and classify them to obtain multiple cluster sets;

[0099] The principal person in the cluster set with the largest number of clusters among the multiple cluster sets is taken as the target person;

[0100] The best face image with the best quality score among the best face images corresponding to the target person is used as the target face image.

[0101] Specifically, human images are subjected to a ReID algorithm for data fusion to extract human features. The ReID algorithm can handle variations in human features across different cameras. The ReID algorithm, also known as a person re-identification algorithm (REID), fuses images captured by different cameras to remove duplicates. The REID algorithm fuses images of the same person across six cameras and selects a target face image or optimal human image for identification. The human features of the extracted images are then similarly calculated to form a similarity matrix. A 6x6 matrix is ​​generated for each of the six cameras, and the mean of the similarity matrix is ​​calculated. This is used as a priori condition for class identity. Each human feature of the subject is then used as an anchor point. If its similarity with any other point is greater than the mean of the similarity matrix, the two points are considered to be of the same class and form a cluster. All clusters are recursively traversed. If a cluster shares feature points with another cluster, it is considered to be of the same class. Clusters are combined until no more clusters can be combined, resulting in multiple cluster sets. The human images contained in each cluster set correspond to the same person. The subject person corresponding to the cluster set with the largest number of clusters among all cluster sets is selected and merged to determine the target person passing through multiple cameras. The best face image with the best quality score among the face images of the subject person corresponding to the cluster set with the largest number of clusters among all cluster sets is used as the target face image.

[0102] Further references Figure 7 As an implementation of the methods shown in the above figures, this application provides an embodiment of a fusion deduplication device based on multi-camera captured images. Figure 2 Corresponding to the method embodiment shown, the device can be specifically applied to various electronic devices.

[0103] The embodiment of the present application provides a device for fusion deduplication based on images captured by multiple cameras, comprising:

[0104] The quality score calculation module 1 is configured to obtain images captured by multiple cameras respectively arranged around the channel, identify the head image, face image and body image from the images captured by each camera, and obtain the quality score corresponding to the face image, where the quality score is determined based on the pitch angle, roll angle and yaw angle of the face;

[0105] The main person identification module 2 is configured to track the head based on the head image captured by each camera to obtain a head trajectory, and determine the main person passing through the passage based on the order in which the same person in the head trajectory appears in the candidate area and the confirmation area of ​​the passage;

[0106] The best image acquisition module 3 is configured to acquire the best human body image and the best face image of the subject person captured by each camera based on the subject person and the head trajectory;

[0107] The target image acquisition module 4 is configured to de-duplicate and merge the best human body images of the subject person captured by all cameras through the ReID algorithm, determine the target person, and determine the target face image based on the quality score of the best face image corresponding to the target person.

[0108] Reference below Figure 8 , which shows an electronic device (eg Figure 1 A schematic structural diagram of a computer device 800 (a server or terminal device as shown). Figure 8 The electronic device shown is merely an example and should not limit the functions and scope of use of the embodiments of the present application.

[0109] like Figure 8As shown, the computer device 800 includes a central processing unit (CPU) 801 and a graphics processing unit (GPU) 802, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 803 or the program loaded from the storage part 809 to the random access memory (RAM) 804. In the RAM 804, various programs and data required for the operation of the device 800 are also stored. The CPU 801, GPU 802, ROM 803 and RAM 804 are connected to each other through a bus 805. An input / output (I / O) interface 806 is also connected to the bus 805.

[0110] The following components are connected to the I / O interface 806: an input section 807 including a keyboard, a mouse, and the like; an output section 808 including a display such as a liquid crystal display (LCD), a speaker, and the like; a storage section 809 including a hard disk and the like; and a communication section 810 including a network interface card such as a LAN card or a modem. The communication section 810 performs communication processing via a network such as the Internet. A drive 811 may also be connected to the I / O interface 806 as needed. A removable medium 812, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, and the like, is installed in the drive 811 as needed, so that a computer program read therefrom can be installed into the storage section 809 as needed.

[0111] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network through the communication part 810, and / or installed from the removable medium 812. When the computer program is executed by the central processing unit (CPU) 801 and the graphics processing unit (GPU) 802, the above-mentioned functions defined in the method of the present application are executed.

[0112] It should be noted that the computer-readable medium of the present application may be a computer-readable signal medium or a computer-readable medium, or any combination of the two. The computer-readable medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor device, apparatus, or component, or any combination of the above. More specific examples of computer-readable media may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution device, apparatus, or component. In the present application, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution apparatus, device, or device. Program code embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wireline, optical cable, RF, or any suitable combination thereof.

[0113] Computer program code for performing the operations of the present application can be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on the remote computer or server. In cases involving a remote computer, the remote computer can be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computer (e.g., through the Internet using an Internet service provider).

[0114] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the devices, methods and computer program products according to various embodiments of the present application. In this regard, each box in the flowchart or block diagram can represent a module, program segment or a part of code, and the module, program segment or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart can be implemented with a dedicated hardware-based device that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0115] The modules involved in the embodiments described in this application may be implemented in software or hardware, and may also be set in a processor.

[0116] As another aspect, the present application also provides a computer-readable medium, which may be included in the electronic device described in the above embodiment; or it may exist independently and not be assembled into the electronic device. The above computer-readable medium carries one or more programs, and when the above one or more programs are executed by the electronic device, the electronic device: acquires and collects images captured by multiple cameras respectively arranged around the channel, identifies a head image, a face image, and a body image from the images captured by each camera, and obtains a quality score corresponding to the face image, the quality score being determined based on the pitch angle, roll angle, and yaw angle of the face; performs head tracking based on the head image acquired by each camera to obtain a head trajectory, determines the main person passing through the channel according to the order in which the same person in the head trajectory appears in the personnel candidate area and the personnel confirmation area of ​​the channel; obtains the best body image and the best face image of the main person captured by each camera based on the main person and his head trajectory; de-duplicates and merges the best body images of the main person captured by all cameras using a ReID algorithm to determine the target person, and determines the target face image based on the quality score of the best face image corresponding to the target person.

[0117] The above description is merely a preferred embodiment of the present application and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in this application is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also encompasses other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned inventive concept. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this application.

Claims

1. A fusion deduplication method based on multi-camera captured images, characterized in that: The following steps are involved: S1, acquiring images captured by multiple cameras respectively arranged around the channel, identifying a head image, a face image and a body image from the images captured by each camera, and obtaining a quality score corresponding to the face image, wherein the quality score is determined based on a pitch angle, a roll angle and a yaw angle of the face; S2, performing head tracking based on the head image acquired by each camera to obtain a head trajectory, and determining the main person passing through the channel according to the order in which the same person in the head trajectory appears in the personnel candidate area and the personnel confirmation area of ​​the channel; S3, obtaining the best human body image and the best face image of the subject person captured by each camera based on the subject person and the head trajectory of the subject person; S4, de-duplication and merging the best human body images of the subject person captured by all cameras through the ReID algorithm to determine the target person, and determine the target face image based on the quality score of the best face image corresponding to the target person.

2. The method for deduplication based on fusion of images captured by multiple cameras according to claim 1 is characterized in that: The step S1 also includes: Constructing a standard 3D facial key point model, deflecting the standard 3D facial key point model to obtain the pitch angle, roll angle and yaw angle and the image 2D plane coordinates of the corresponding facial key points, and establishing a regression model based on the image 2D plane coordinates to the pitch angle, roll angle and yaw angle; A training set is established. In the training set, after the 2D plane coordinates of the image of the facial key points are detected by the standard 3D facial key point model, the pitch angle, roll angle and yaw angle corresponding to the 2D plane coordinates of the facial key point coordinates in the training set are calculated. After the pitch angle, roll angle and yaw angle are known, the labels of the facial images with angles less than a preset threshold are recorded as high-quality face labels, and the labels of the facial images with angles greater than the threshold are recorded as low-quality face labels. High-quality face labels or low-quality face labels are added to all face frames in the training set. The regression model is trained using the training set to obtain a trained face image. A trained regression model; the training set also includes a head frame, a body frame, and a face frame corresponding to a head image, a body image, and a face image, respectively, to establish an image detection and evaluation model, the image detection and evaluation model includes an input layer, a trained regression model and a detector, the image detection and evaluation model is trained by the training set, the image detection and evaluation model is used to detect a head image, a body image, and a face image and evaluate and obtain the quality score corresponding to the face image, the quality score is directly regressed and output from the image detection and evaluation model, and the quality score is calculated by the pitch angle, roll angle, and yaw angle of the face image.

3. The method for fusion deduplication based on multi-camera captured images according to claim 2 is characterized in that: The step S1 identifies a head image, a face image and a body image from the images captured by each camera, and obtains a quality score corresponding to the face image, specifically including: The images captured by each camera are input into the image detection and evaluation model, and a head image, a face image, and a body image are output, and corresponding quality scores are obtained on the face images, wherein the quality scores include frontal face scores.

4. The method for fusion deduplication based on multi-camera captured images according to claim 3 is characterized in that: In step S2, the head tracking is performed based on the head image acquired by each camera to obtain the head trajectory, which specifically includes: Tracking all heads in the head image captured by each camera to obtain all head trajectories of each camera; Cache the human body images and face images of all points on all head trajectories of each camera.

5. The method for fusion deduplication based on multi-camera captured images according to claim 4 is characterized in that: The step S2 determines the main personnel passing through the passage according to the order in which the same person in the head trajectory appears in the personnel candidate area and the personnel confirmation area of ​​the passage, specifically including: Selecting a personnel candidate area and a personnel confirmation area of ​​the channel from the head images captured by each camera; Based on the fact that the person in the head trajectory appears in the person candidate area and the person confirmation area successively within the target time period, the person is determined to be the main person.

6. The method for fusion deduplication based on multi-camera captured images according to claim 5 is characterized in that: The step S3 specifically includes: Obtaining a facial image with the best quality score from facial images of each point on the head trajectory of the subject person cached by each camera as the best facial image; The human body images at each point on the head trajectory of the subject person cached by each camera include the human body image with the most complete human body features of similar size and position as the best human body image.

7. The method for fusion deduplication based on multi-camera captured images according to claim 1 is characterized in that: The step S4 specifically includes: Summarize the best human body image and the best face image of the subject person corresponding to each camera; The ReID algorithm is used to extract human features from the best human image, and the similarity of human features in the best human images of the subject captured by every two cameras is calculated to obtain a similarity matrix, and the mean of the similarity matrix is ​​calculated; Taking the human features of the best human image of the subject person corresponding to each camera as anchor points, in response to the similarity between one of the anchor points and the other anchor points being greater than the mean value of the similarity matrix, determining that the subject person captured by one of the cameras and the other cameras are in the same cluster, thereby obtaining multiple clusters; Recursively traverse all clusters and classify them to obtain multiple cluster sets; Taking the subject person of the cluster set with the largest number of clusters among the multiple cluster sets as the target person; The best facial image with the best quality score among the best facial images corresponding to the target person is used as the target facial image.

8. The method for fusion deduplication based on multi-camera captured images according to claim 1, characterized in that: The multiple cameras include 6 cameras located at the upper, middle and lower parts of the left and right sides of the channel respectively.

9. A fusion deduplication device based on multi-camera captured images, characterized in that: include: The quality score calculation module is configured to acquire images captured by a plurality of cameras respectively arranged around the channel, respectively identify a head image, a face image and a body image from the images captured by each camera, and obtain a quality score corresponding to the face image, wherein the quality score is determined based on a pitch angle, a roll angle and a yaw angle of the face; A main person determination module is configured to perform head tracking based on the head image acquired by each camera to obtain a head trajectory, and determine the main person passing through the channel according to the order in which the same person in the head trajectory appears in the personnel candidate area and the personnel confirmation area of ​​the channel; An optimal image acquisition module is configured to acquire the optimal human body image and the optimal face image of the subject person captured by each camera based on the subject person and the head trajectory of the subject person; The target image acquisition module is configured to remove overlaps and merge the best human body images of the subject person captured by all cameras through the ReID algorithm, determine the target person, and determine the target face image based on the quality score of the best face image corresponding to the target person.

10. An electronic device, comprising: one or more processors; a storage device for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 8.

11. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the program is executed by a processor, the method according to any one of claims 1 to 8 is implemented.

Images