Cross-lens re-identification method and system for target human body in closed environment and electronic equipment
By using twin models and random patching strategies in a closed environment, the similarity score value of the target image and the current frame image is calculated, and combined with pixel coordinates and label information, the recognition problem after the target is lost in closed complex scenes is solved, which improves the Accuracy and performance of pedestrian tracking.
Patent Information
- Application Number
- CN202410990606.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2024-07-23
- Publication Date
- 2024-11-29
- Estimated Expiration
- 2044-07-23
AI Technical Summary
In closed and complex scenarios, the pedestrian background is complex and changeable. When the computer tracks pedestrians, it is easy to lose people due to problems such as occlusion, resulting in the loss of the target and cannot be re-identified. The existing technology lacks effective solutions.
Using pre-trained twin models, the model is trained by randomly adding patch strategies to calculate the feature vector similarity score of the target image and the current frame image, and judge the spatial distance based on pixel coordinates and label information, quickly find the lost target, and improve the Accuracy and performance of target tracking methods.
It improves the accuracy and tracking performance of the target re-identification method, and can quickly find lost targets in closed and complex scenarios, solving the problem that targets cannot be re-identified after they are lost during cross-lens re-identification.
Smart Images

Figure CN119049078A_ABST
Abstract
Description
Technical Field
[0001] The present invention relates to the field of cross-shot re-identification technology, and more specifically, to a method, system and electronic device for cross-shot re-identification of a target human body in a closed environment. Background Art
[0002] Cross-shot re-identification refers to a method of retrieving and tracking pedestrians that appear in two non-overlapping cameras at different time points. For example, camera A and camera B are two cameras without any intersection. The motion video data or single-frame image of a pedestrian is captured by camera A, and then the pedestrian captured by camera A is retrieved and identified in the picture captured by camera B to observe whether the pedestrian captured by camera A appears in the data collected by camera B. Camera A and camera B are independent of each other. This retrieval and identification method is called cross-shot re-identification.
[0003] For example, invention application CN112001251A discloses a pedestrian re-identification method and system based on the combination of human body analysis and clothing color, including: obtaining a target pedestrian image to be identified; inputting the target pedestrian image to be identified into a pre-built human body analysis model, and outputting the body part category of the target pedestrian to be identified; extracting color features of each category of body parts from the target pedestrian image to be identified; and identifying the target pedestrian from candidate images based on the extracted color features of each category of body parts.
[0004] However, in closed and complex scenes, pedestrians have complex and changeable backgrounds. When computers track pedestrians, they can easily lose track of people due to occlusion and other issues, and cannot re-identify after the target is lost. There is currently no corresponding solution to the problem of being unable to re-identify after the target is lost during cross-border navigation in closed and complex scenes. Summary of the invention
[0005] In view of the deficiencies in the prior art, the present invention provides a method, system and electronic device for re-identifying a target person across shots in a closed environment.
[0006] According to one aspect of the present invention, a method for cross-shot re-identification of a target person in a closed environment is provided, comprising:
[0007] Acquire a target image and a first video data stream to be re-identified across shots; wherein the first video data stream is acquired by a first image acquisition device in a closed environment, the target image is any frame of an image containing a target object in a second video data stream, the second video data stream is acquired by a second image acquisition device in a closed environment, and the first image acquisition device and the second image acquisition device are independent of each other;
[0008] The target image is used as a template image, and the current frame image of the first video data stream is used as a search image. The template image and the search image are input into a pre-trained twin model, and a similarity score value between a feature vector of the template image and a feature vector of the search image is calculated; wherein the twin model has two branches with the same architecture and weight set, one of which is used to learn the feature representation of the target, and the other is used to search the area; and the twin model is trained using a random patch addition strategy; the twin model uses the following function to calculate the similarity score value between the feature vector of the template image and the feature vector of the search image:
[0009]
[0010] In the formula, f θ (z,x) represents the similarity response function between the feature vector of the template image z and the feature vector of the search image x in the twin model with parameter θ; represents the feature vector of the template image z in the twin model with parameter θ, represents the search for the feature vector of image x in the twin model with parameter θ, and b·l represents the deviation term at each position b;
[0011] Compare the similarity score value with the preset similarity threshold value to determine whether there is a target human body with the same identity as the target object in the current frame image; when there is a target human body with the same identity as the target object in the current frame image, determine the label of the target human body in the current frame image according to the label of the target object, and record the pixel coordinates of the target human body in the current frame image; when it is recognized that there are other human bodies except the target human body in the current frame image, determine the pixel coordinates and label information of the other human bodies;
[0012] Based on the pixel coordinates and label information of each human body in each frame image, determine whether the spatial distance between the current frame image and the human body with the same label in the previous frame image is less than a preset distance threshold; when the spatial distance between the human body with the same label in the current frame image and the previous frame image is less than the preset distance threshold, determine that there is a target human body with the same identity as the target object in the first video data stream.
[0013] Optionally, acquiring a target image containing a target object includes:
[0014] Identify the second video data stream and determine each frame image of the target object in the second video data stream;
[0015] Building a template image library based on each frame image containing the target object;
[0016] When cross-shot re-identification of the target person is required, a frame of image is randomly selected from the constructed template image library as the target image.
[0017] Optionally, the method further comprises:
[0018] Before training the twin model, black patches of preset areas are randomly placed in the two sample images to cover up the most distinctive parts of the two sample images; the first sample image is the original image collected, and the second sample image is an image containing the human body area obtained by cropping the first sample image;
[0019] During the training process, two sample images including black patches are input into each branch of the twin model, and the feature calculation results output by each branch are all linked. Then, they are input into the linear fully connected layer of the twin model to output the training results.
[0020] Optionally, the method also includes: when the spatial distance between the human body with the same label in the current frame image and the previous frame image is not less than a preset distance threshold, re-determining the label of the corresponding human body in the current frame image, and deducting the image of the corresponding human body and storing it in the template image library.
[0021] Optionally, after determining that a target human body having the same identity as the target object exists in the video data stream, the method further comprises: extracting an image of the target human body and incorporating it into a template image library.
[0022] According to another aspect of the present invention, a system for re-identifying a target person across shots in a closed environment is provided, comprising:
[0023] An acquisition module is used to acquire a target image and a first video data stream to be re-identified across shots; wherein the first video data stream is acquired by a first image acquisition device in a closed environment, the target image is any frame image of a target object in a second video data stream, the second video data stream is acquired by a second image acquisition device in a closed environment, and the first image acquisition device and the second image acquisition device are independent of each other;
[0024] A similarity calculation module is used to use the target image as a template image, the current frame image of the first video data stream as a search image, input the template image and the search image into a pre-trained twin model, and calculate the similarity score value between the feature vector of the template image and the feature vector of the search image; wherein the twin model has two branches with the same architecture and weight set, one branch is used to learn the feature representation of the target, and the other branch is used to search the area; and the twin model is trained using a random patch addition strategy; the twin model uses the following function to calculate the similarity score value between the feature vector of the template image and the feature vector of the search image:
[0025]
[0026] In the formula, fθ (Z,x) represents the similarity response function between the feature vector of the template image z and the feature vector of the search image x in the twin model with parameter θ; represents the feature vector of the template image z in the twin model with parameter θ, represents the search for the feature vector of image x in the twin model with parameter θ, and b·l represents the deviation term at each position b;
[0027] The first judgment module is used to compare the similarity score value with a preset similarity threshold value to determine whether there is a target human body with the same identity as the target object in the current frame image; when there is a target human body with the same identity as the target object in the current frame image, determine the label of the target human body in the current frame image according to the label of the target object, and record the pixel coordinates of the target human body in the current frame image; when it is recognized that there are other human bodies except the target human body in the current frame image, determine the pixel coordinates and label information of the other human bodies;
[0028] The second judgment module is used to judge whether the spatial distance between the human body with the same label in the current frame image and the previous frame image is less than a preset distance threshold based on the pixel coordinates and label information of each human body in each frame image; when the spatial distance between the human body with the same label in the current frame image and the previous frame image is less than the preset distance threshold, it is determined that there is a target human body with the same identity as the target object in the first video data stream.
[0029] Optionally, the acquisition module is specifically used to:
[0030] Identify the second video data stream and determine each frame image of the target object in the second video data stream;
[0031] Building a template image library based on each frame image containing the target object;
[0032] When cross-shot re-identification of the target person is required, a frame of image is randomly selected from the constructed template image library as the target image.
[0033] Optionally, the system further comprises a training module for:
[0034] Before training the twin model, black patches of preset areas are randomly placed in the two sample images to cover up the most distinctive parts of the two sample images; the first sample image is the original image collected, and the second sample image is an image containing the human body area obtained by cropping the first sample image;
[0035] During the training process, two sample images including black patches are input into each branch of the twin model, and the feature calculation results output by each branch are all linked. Then, they are input into the linear fully connected layer of the twin model to output the training results.
[0036] According to another aspect of the present invention, a computer-readable storage medium is provided, wherein the storage medium stores a computer program, and the computer program is used to execute the method described in any one of the above aspects of the present invention.
[0037] According to another aspect of the present invention, an electronic device is provided, comprising: a processor; a memory for storing instructions executable by the processor; the processor is configured to read the executable instructions from the memory and execute the executable instructions to implement the method described in any one of the above aspects of the present invention.
[0038] Based on the existing cross-border head recognition technology, the present invention uses any frame image containing the target object in the second video data stream as a template image, uses the current frame image of the first video data stream as a search image, inputs the template image and the search image into a pre-trained twin model, calculates the similarity score between the feature vector of the template image and the feature vector of the search image, and mines the time information in the video according to the similarity score to obtain the trajectory fragment of the human body, so that the accuracy of the target re-recognition method is improved. Furthermore, based on the pixel coordinates and label information of each human body in each frame image, the spatial information in the video is mined to obtain the position information of each human body, and the spatial distance between the current frame image and the human body with the same label in the previous frame image is determined according to the position information. It is less than a preset distance threshold, so as to quickly find the lost target and improve the performance of the target tracking method. It solves the technical problem that the target cannot be re-identified after being lost during the cross-border head recognition process in closed and complex scenes. BRIEF DESCRIPTION OF THE DRAWINGS
[0039] A more complete understanding of exemplary embodiments of the present invention may be obtained by referring to the following drawings:
[0040] Figure 1 is a flow chart of a method for re-identifying a target person across shots in a closed environment provided by an exemplary embodiment of the present invention;
[0041] Figure 2 A schematic diagram of the structure of a system for re-identifying a target person across shots in a closed environment provided by an embodiment of the present invention;
[0042] Figure 3 This is a structure of an electronic device provided by an exemplary embodiment of the present invention. DETAILED DESCRIPTION
[0043] Below, the exemplary embodiments according to the present invention will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present invention, rather than all the embodiments of the present invention, and it should be understood that the present invention is not limited to the exemplary embodiments described here.
[0044] It should be noted that the relative arrangement of components and steps, numerical expressions and numerical values set forth in these embodiments do not limit the scope of the present invention unless specifically stated otherwise.
[0045] Those skilled in the art can understand that the terms "first" and "second" in the embodiments of the present invention are only used to distinguish different steps, devices or modules, etc., and neither represent any specific technical meaning nor indicate the necessary logical order between them.
[0046] It should also be understood that, in the embodiments of the present invention, “plurality” may refer to two or more than two, and “at least one” may refer to one, two or more than two.
[0047] It should also be understood that any component, data or structure mentioned in the embodiments of the present invention can generally be understood as one or more, unless explicitly limited or otherwise indicated in the context.
[0048] In addition, the term "and / or" in the present invention is only a description of the association relationship of the associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in the present invention generally indicates that the associated objects before and after are in an "or" relationship.
[0049] It should also be understood that the description of the various embodiments of the present invention focuses on the differences between the various embodiments, and the same or similar aspects thereof can be referenced to each other, and for the sake of brevity, they will not be described one by one.
[0050] At the same time, it should be understood that for the convenience of description, the sizes of the various parts shown in the drawings are not drawn according to the actual proportional relationship.
[0051] The following description of at least one exemplary embodiment is merely illustrative in nature and is in no way intended to limit the invention, its application, or uses.
[0052] Technologies, methods, and equipment known to ordinary technicians in the relevant art may not be discussed in detail, but where appropriate, the technologies, methods, and equipment should be considered part of the specification.
[0053] It should be noted that like reference numerals and letters refer to similar items in the following figures, and therefore, once an item is defined in one figure, it need not be further discussed in subsequent figures.
[0054] Embodiments of the present invention can be applied to electronic devices such as communication terminals, computer systems, servers, etc., which can operate with many other general or special computing system environments or configurations. Examples of well-known communication terminals, computing systems, environments and / or configurations suitable for use with electronic devices such as communication terminals, computer systems, servers, etc. include, but are not limited to: personal computer systems, server computer systems, thin clients, thick clients, handheld or laptop devices, microprocessor-based systems, set-top boxes, programmable consumer electronics, network personal computers, small computer systems, large computer systems, and distributed cloud computing technology environments including any of the above systems, etc.
[0055] Electronic devices such as communication terminals, computer systems, and servers can be described in the general context of computer system executable instructions (such as program modules) executed by a computer system. In general, program modules can include routines, programs, object programs, components, logic, data structures, etc., which perform specific tasks or implement specific abstract data types. Computer systems / servers can be implemented in a distributed cloud computing environment, where tasks are performed by remote processing devices linked through a communication network. In a distributed cloud computing environment, program modules can be located on local or remote computing system storage media including storage devices.
[0056] The present invention provides a method, system and electronic equipment for re-identifying a target human body across shots in a closed environment. Figure 1 FIG. 1 is a flow chart of a method for re-identifying a target person across shots in a closed environment provided by an exemplary embodiment of the present invention. Figure 1 As shown in the figure, the cross-shot re-identification method of the target person in a closed environment includes:
[0057] Step S1: obtaining a target image and a first video data stream to be re-identified across shots; wherein the first video data stream is acquired by a first image acquisition device in a closed environment, the target image is any frame image of a target object in a second video data stream, the second video data stream is acquired by a second image acquisition device in a closed environment, and the first image acquisition device and the second image acquisition device are independent of each other;
[0058] Optionally, obtaining the target image containing the target object includes: identifying the second video data stream to determine each frame image in the second video data stream that contains the target object; constructing a template image library based on each frame image containing the target object; and when cross-shot re-identification of the target human body is required, randomly extracting a frame image from the constructed template image library as the target image.
[0059] In an embodiment of the present invention, it is first necessary to obtain a target image containing a target object and a first video data stream to be re-identified across shots. Specifically, the first video data stream to be re-identified across shots is acquired by a first image acquisition device, and the second video data stream is acquired by a second image acquisition device, and then the second video data stream is recognized, and each frame image containing the target object in the second video data stream is determined, and a template image library is constructed based on each frame image containing the target object. When it is necessary to re-identify the target human body across shots, a frame image is randomly extracted from the constructed template image library as the target image.
[0060] Step S2: The current frame image of the first video data stream is used as the search image, and the template image and the search image are input into the pre-trained twin model to calculate the similarity score between the feature vector of the template image and the feature vector of the search image; wherein the twin model has two branches with the same architecture and weight set, one of which is used to learn the feature representation of the target, and the other is used to search the area; and the twin model is trained using a random patch addition strategy; the twin model uses the following function to calculate the similarity score between the feature vector of the template image and the feature vector of the search image:
[0061]
[0062] In the formula, f θ (Z,x) represents the similarity response function between the feature vector of the template image z and the feature vector of the search image x in the twin model with parameter θ; represents the feature vector of the template image z in the twin model with parameter θ, represents the search for the feature vector of image x in the twin model with parameter θ, and b·l represents the deviation term at each position b;
[0063] Optionally, the method also includes: before training the twin model, randomly placing black patches of preset areas in the two sample images to mask the most distinctive parts of the two sample images; wherein the first sample image is the collected original image, and the second sample image is an image containing a human body area obtained by cropping the first sample image; during the training process, the two sample images including the black patches are input into each branch of the twin model, the feature calculation results output by each branch are all linked, and then input into the linear fully connected layer of the twin model to output the training results.
[0064] In an embodiment of the present invention, a tracking similarity graph is referenced from a deep model with a twin network structure, one branch is used to learn the feature representation of the target, and the other branch is used to search the area. There are two branches in the network that share exactly the same architecture and the same set of weights. During the training process, after the video stream is input into the twin network, black patches of a preset area are randomly placed in the image area to ensure that the most distinctive part of an object is not always visible to the network, thereby forcing it to also pay attention to other related parts of the object. In this way, the limitations of existing semi-supervised methods, that is, only focusing on the most distinctive part of the object, can be overcome, and the generalization ability of the model can be improved.
[0065] During the training process, each training branch takes the two images containing patches as input and then performs a series of convolution operations. The feature calculation results output by the branches are all linked and input into the top network consisting of a linear fully connected layer. The training process repeats the above process and finally obtains the optimal convolutional neural network with parameter θ, thus obtaining a trained twin model.
[0066] Furthermore, the target image is used as the template image z and the current video stream is used as the search image x. The two image inputs are processed by a convolutional neural network with parameter θ to generate two interrelated feature maps:
[0067]
[0068] Among them, f θ (z,x) represents the similarity response function between the feature vector of the template image z and the feature vector of the search image x in the convolutional neural network with parameter θ; represents the feature vector of the template image z in the convolutional neural network with parameter θ, Represents searching for the feature vector of image x in a convolutional neural network with parameters θ, b·l represents the bias term of the value of b at each position, and formula (1) is equivalent to an exhaustive search of the region z of x on the image, with the goal of matching the maximum value in the response map f to the target position. To achieve this goal, the network obtains the parameters θ of the convolutional neural network by minimizing the logistic loss of the training set through random image pairs (z, x) and the corresponding true position labels y:
[0069]
[0070] This simple matching function has two inherent limitations when designing a tracker. These are: the feature extractor used in the tracker has an intrinsic constraint of strict translation invariance f(z,x[Δτ j ])=f(z,x)[Δτ j], where f(z,x) represents the similarity response function between the feature vector of template image z and the feature vector of search image x, and Δτj is the translation subwindow operator, which ensures effective training and inference; the inherent restriction on structural symmetry is f(z,x')=f(x',z).
[0071] Among them, z represents the template image, and the algorithm uses the ground truth of the first frame; x represents the search area, which indicates the candidate search area in the subsequent frames to be tracked; Represents the feature mapping operation, which maps the original image to a specific feature space through the convolution layer and pooling layer in the neural network. 6×6×128 represents z through the program The feature obtained after this is a 128-channel feature with a size of 6×6. Similarly, 22×22×128 is the search area x through The subsequent feature size after 22×22×128 is convolved by a 6×6×128 convolution kernel, and then a 17×17 score map is obtained, which is the similarity value of each image pair in the search area. The neural network itself compares the difference between the target image search area and the template image to obtain the score value of the search area.
[0072] Step S3: Compare the similarity score value with the preset similarity threshold value to determine whether there is a target human body with the same identity as the target object in the current frame image; when there is a target human body with the same identity as the target object in the current frame image, determine the label of the target human body in the current frame image according to the label of the target object, and record the pixel coordinates of the target human body in the current frame image; when it is identified that there are other human bodies except the target human body in the current frame image, determine the pixel coordinates and label information of the other human bodies;
[0073] Optionally, after determining that a target human body having the same identity as the target object exists in the video data stream, the method further comprises: extracting an image of the target human body and incorporating it into a template image library.
[0074] In an embodiment of the present invention, by presetting a similarity score threshold, when the score value is less than the threshold, it is determined that there is no target with the same identity as the template image in the current search image, and the target of interest in the current video stream is cut out (using common target detection methods) and stored in the template image library; if the score is not less than the threshold, it is preliminarily determined that there is a target with the same identity as the template image in the current search image.
[0075] Step S4: Based on the pixel coordinates and label information of each human body in each frame image, determine whether the spatial distance between the current frame image and the human body with the same label in the previous frame image is less than a preset distance threshold; when the spatial distance between the human body with the same label in the current frame image and the previous frame image is less than the preset distance threshold, determine that there is a target human body with the same identity as the target object in the first video data stream.
[0076] Optionally, the method also includes: when the spatial distance between the human body with the same label in the current frame image and the previous frame image is not less than a preset distance threshold, re-determining the label of the corresponding human body in the current frame image, and deducting the image of the corresponding human body and storing it in the template image library.
[0077] In the embodiment of the present invention, although the random patching method is adopted when training the model to effectively reduce the number of times that a person cannot be identified or misidentified due to partial occlusion of the person, when the person is severely occluded, the above method still cannot accurately identify the identity information of the person. Therefore, a correction method based on spatial information is subsequently added, and the specific process is as follows:
[0078] 1) Save the pixel coordinates and label information of each target correctly identified in the current search image x for correction with the next frame.
[0079] 2) Determine whether the spatial distance between the target with the same label in the current frame and the previous frame is less than a threshold. This distance is calculated by the Euclidean distance between the coordinates of the center points of the two detection frames.
[0080] 3) If the distance is greater than or equal to the threshold, it is preliminarily determined that the same target in the previous frame and the current frame is not the same person, the target in the current frame is reset to a new label, and the target image is deducted and stored in the template image library; if the distance is less than the threshold, the target label information is not corrected, and it is considered that the preliminary judgment through the twin network is correct, the result is output, and the current target is deducted and stored in the template image library.
[0081] 4) The expanded template image library has template images with different labels in different time series, and calculates f of the same type of labels in different time series. θ (z,x), dynamically remove template images with lower scores in the same category of labels in real time to improve the accuracy of re-identification.
[0082] The present invention uses a twin network as the basic network for extracting position information features, and considering the "person occlusion" problem, random mask blocks are connected to the target objects in the training set so that certain areas of the target person with distinguishing features will not affect the training of the overall features, thereby improving the accuracy of pedestrian tracking. The present invention introduces a stable twin network tracking method based on semi-supervised learning, and adopts a position-enhanced twin network model and a random occlusion attack strategy to solve the problem of pedestrian sample occlusion; the present invention explores a progressive pedestrian re-identification network update method based on reinforcement learning, and adopts a network collaborative training strategy based on reinforcement learning and a video sequence frame-to-frame information mining model to cope with the situation of insufficient information mining between video streams.
[0083] The present invention aims to improve the problem of complex and changeable pedestrian backgrounds. Based on the existing cross-border head re-identification technology, the time information in the video is mined to obtain the trajectory fragments of pedestrians, and the pseudo-label video fragment data set is expanded to improve the accuracy of the pedestrian re-identification method; the pedestrian re-identification technology is used as the target tracking method to mine the spatial information in the video to obtain the position fragments of pedestrians. Due to the characteristics of pedestrian re-identification, the lost targets can be quickly found and such targets can be recorded as difficult samples, so that the performance of the target tracking method is improved. Relying on this system optimization method of combining time-space information, the target tracking network and the pedestrian re-identification network can promote each other, forming a virtuous circle, and further improving the overall performance of the network.
[0084] Therefore, the present invention can be used as a common visual matching algorithm to provide auxiliary solutions for other computer vision tasks. For example, when using face recognition technology for identity recognition, there may be a problem that the face image of the pedestrian captured in the video surveillance system across time and space is not high-definition, and the identity cannot be correctly recognized at this time. The cross-lens re-identification technology can be used as an auxiliary solution to improve the accuracy of the pedestrian identification system in complex interference scenes based on the appearance, clothing and other information of the pedestrian.
[0085] The application of the technology of the present invention can fully identify the trajectory of the target human body's movement and stay, track and analyze the target human body's behavior. For example, when a computer tracks pedestrians, it may lose the person due to occlusion and other problems. The cross-lens re-identification method can relocate the target person, allowing the computer to continue the tracking task.
[0086] The algorithm research ideas of the technology of the present invention can inspire the research and development of other visual retrieval and matching tasks, such as hot technologies such as video image retrieval, image classification and face matching, thereby promoting the development and progress of research related to computer vision tasks.
[0087] In summary, based on the existing cross-border head re-identification technology, the present invention uses any frame image containing the target object in the second video data stream as a template image, uses the current frame image of the first video data stream as a search image, inputs the template image and the search image into a pre-trained twin model, calculates the similarity score between the feature vector of the template image and the feature vector of the search image, and mines the time information in the video according to the similarity score to obtain the trajectory fragment of the human body, so that the accuracy of the target re-identification method is improved. Furthermore, based on the pixel coordinates and label information of each human body in each frame image, the spatial information in the video is mined to obtain the position information of each human body, and the spatial distance between the current frame image and the human body with the same label in the previous frame image is determined according to the position information. Is less than a preset distance threshold, so as to quickly find the lost target and improve the performance of the target tracking method. It solves the technical problem that the target cannot be re-identified after being lost during the cross-border head tracking process in closed and complex scenes.
[0088] Exemplary Systems
[0089] Figure 2 FIG. 1 is a schematic diagram of a system for re-identifying a target person across shots in a closed environment provided by an exemplary embodiment of the present invention. Figure 2 As shown, the system includes:
[0090] The acquisition module 210 is used to acquire a target image and a first video data stream to be re-identified across shots; wherein the first video data stream is acquired by a first image acquisition device in a closed environment, the target image is any frame image of a target object in a second video data stream, and the second video data stream is acquired by a second image acquisition device in a closed environment, and the first image acquisition device and the second image acquisition device are independent of each other;
[0091] The similarity calculation module 220 is used to use the target image as a template image, the current frame image of the first video data stream as a search image, input the template image and the search image into a pre-trained twin model, and calculate the similarity score between the feature vector of the template image and the feature vector of the search image; wherein the twin model has two branches with the same architecture and weight set, one of which is used to learn the feature representation of the target, and the other branch is used to search the area; and the twin model is trained using a random patch addition strategy; the twin model uses the following function to calculate the similarity score between the feature vector of the template image and the feature vector of the search image:
[0092]
[0093] In the formula, f θ (z,x) represents the similarity response function between the feature vector of the template image z and the feature vector of the search image x in the twin model with parameter θ; represents the feature vector of the template image z in the twin model with parameter θ, represents the search for the feature vector of image x in the twin model with parameter θ, and b·l represents the deviation term at each position b;
[0094] The first judgment module 230 is used to compare the similarity score value with a preset similarity threshold value to determine whether there is a target human body with the same identity as the target object in the current frame image; when there is a target human body with the same identity as the target object in the current frame image, determine the label of the target human body in the current frame image according to the label of the target object, and record the pixel coordinates of the target human body in the current frame image; when it is recognized that there are other human bodies except the target human body in the current frame image, determine the pixel coordinates and label information of the other human bodies;
[0095] The second judgment module 240 is used to judge whether the spatial distance between the human body with the same label in the current frame image and the previous frame image is less than a preset distance threshold based on the pixel coordinates and label information of each human body in each frame image; when the spatial distance between the human body with the same label in the current frame image and the previous frame image is less than the preset distance threshold, it is determined that there is a target human body with the same identity as the target object in the first video data stream.
[0096] Optionally, the acquisition module 210 is specifically configured to:
[0097] Identify the second video data stream and determine each frame image of the target object in the second video data stream;
[0098] Building a template image library based on each frame image containing the target object;
[0099] When cross-shot re-identification of the target person is required, a frame of image is randomly selected from the constructed template image library as the target image.
[0100] Optionally, the system further comprises a training module for:
[0101] Before training the twin model, black patches of preset areas are randomly placed in the two sample images to cover up the most distinctive parts of the two sample images; the first sample image is the original image collected, and the second sample image is an image containing the human body area obtained by cropping the first sample image;
[0102] During the training process, two sample images including black patches are input into each branch of the twin model, and the feature calculation results output by each branch are all linked. Then, they are input into the linear fully connected layer of the twin model to output the training results.
[0103] The apparatus for re-identifying a target person across shots in a closed environment in an embodiment of the present invention corresponds to a method for re-identifying a target person across shots in a closed environment in another embodiment of the present invention, and will not be described in detail herein.
[0104] Exemplary Electronic Devices
[0105] Figure 3 This is a structure of an electronic device provided by an exemplary embodiment of the present invention. Figure 3 As shown, the electronic device 30 includes one or more processors 31 and a memory 32 .
[0106] The processor 31 may be a central processing unit (CPU) or other forms of processing units having data processing capabilities and / or instruction execution capabilities, and may control other components in the electronic device to perform desired functions.
[0107] The memory 32 may include one or more computer program products, and the computer program product may include various forms of computer-readable storage media, such as volatile memory and / or non-volatile memory. The volatile memory may include, for example, random access memory (RAM) and / or cache memory (cache), etc. The non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc. One or more computer program instructions may be stored on the computer-readable storage medium, and the processor 31 may run the program instructions to implement the methods of the software programs of the various embodiments of the present invention described above and / or other desired functions. In one example, the electronic device may also include: an input device 33 and an output device 34, which are interconnected via a bus system and / or other forms of connection mechanisms (not shown).
[0108] In addition, the input device 33 may also include, for example, a keyboard, a mouse, and the like.
[0109] The output device 34 can output various information to the outside. The output device 34 can include, for example, a display, a speaker, a printer, a communication network and a remote output device connected thereto.
[0110] Of course, to simplify, Figure 3 Only some of the components related to the present invention in the electronic device are shown, and components such as a bus, an input / output interface, etc. are omitted. In addition, according to specific application conditions, the electronic device may also include any other appropriate components.
[0111] Exemplary computer program products and computer-readable storage media
[0112] In addition to the above-mentioned methods and devices, an embodiment of the present invention may also be a computer program product, which includes computer program instructions, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present invention described in the above-mentioned "Exemplary Method" section of this specification.
[0113] The computer program product may be written in any combination of one or more programming languages to write program code for performing the operations of the embodiments of the present invention, including object-oriented programming languages such as Java, C++, etc., and conventional procedural programming languages such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server.
[0114] In addition, an embodiment of the present invention may also be a computer-readable storage medium having computer program instructions stored thereon, which, when executed by a processor, enable the processor to execute the steps of the method according to various embodiments of the present invention described in the above “Exemplary Method” section of this specification.
[0115] The computer readable storage medium can adopt any combination of one or more readable media. The readable medium can be a readable signal medium or a readable storage medium. The readable storage medium can include, for example, but is not limited to, a system, system or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of readable storage media include: an electrical connection with one or more wires, a portable disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above.
[0116] The basic principle of the present invention is described above in conjunction with specific embodiments. However, it should be pointed out that the advantages, strengths, effects, etc. mentioned in the present invention are only examples and not limitations, and it cannot be considered that these advantages, strengths, effects, etc. must be possessed by each embodiment of the present invention. In addition, the specific details disclosed above are only for the purpose of illustration and facilitation of understanding, rather than limitation, and the above details do not limit the present invention to being implemented by adopting the above specific details.
[0117] Each embodiment in this specification is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the system embodiment, since it basically corresponds to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.
[0118] The block diagrams of the devices, systems, equipment, and systems involved in the present invention are only illustrative examples and are not intended to require or imply that they must be connected, arranged, and configured in the manner shown in the block diagram. As will be appreciated by those skilled in the art, these devices, systems, equipment, and systems can be connected, arranged, and configured in any manner. Words such as "including", "comprising", "having", etc. are open words, referring to "including but not limited to", and can be used interchangeably with them. The words "or" and "and" used here refer to the words "and / or" and can be used interchangeably with them, unless the context clearly indicates otherwise. The word "such as" used here refers to the phrase "such as but not limited to", and can be used interchangeably with it.
[0119] The method and system of the present invention may be implemented in many ways. For example, the method and system of the present invention may be implemented by software, hardware, firmware or any combination of software, hardware, firmware. The above order of steps for the method is only for illustration, and the steps of the method of the present invention are not limited to the order specifically described above, unless otherwise specifically stated. In addition, in some embodiments, the present invention may also be implemented as a program recorded in a recording medium, which includes machine-readable instructions for implementing the method according to the present invention. Thus, the present invention also covers a recording medium storing a program for executing the method according to the present invention.
[0120] It should also be noted that in the system, device and method of the present invention, each component or each step can be decomposed and / or recombined. These decompositions and / or recombinations should be regarded as equivalent schemes of the present invention. The above description of the disclosed aspects is provided to enable any technician in the field to make or use the present invention. Various modifications to these aspects are very obvious to those skilled in the art, and the general principles defined here can be applied to other aspects without departing from the scope of the present invention. Therefore, the present invention is not intended to be limited to the aspects shown here, but in accordance with the widest range consistent with the principles and novel features disclosed here.
[0121] The above description has been given for the purpose of illustration and description. In addition, this description is not intended to limit the embodiments of the present invention to the forms disclosed herein. Although a number of example aspects and embodiments have been discussed above, those skilled in the art will recognize certain variations, modifications, changes, additions and sub-combinations thereof.
Claims
1. A cross-shot re-identification method for human targets in a closed environment. It is characterized in that include: Acquire a target image and a first video data stream to be re-identified across shots; wherein the first video data stream is acquired by a first image acquisition device in a closed environment, the target image is any frame of an image containing a target object in a second video data stream, the second video data stream is acquired by a second image acquisition device in a closed environment, and the first image acquisition device and the second image acquisition device are independent of each other; The target image is used as a template image, and the current frame image of the first video data stream is used as a search image. The template image and the search image are input into a pre-trained twin model, and a similarity score value between a feature vector of the template image and a feature vector of the search image is calculated; wherein the twin model has two branches with the same architecture and weight set, one of which is used to learn the feature representation of the target, and the other is used to search the area; and the twin model is trained using a random patch addition strategy; the twin model uses the following function to calculate the similarity score value between the feature vector of the template image and the feature vector of the search image: In the formula, f θ (z,x) represents the similarity response function between the feature vector of the template image z and the feature vector of the search image x in the twin model with parameter θ; represents the feature vector of the template image z in the twin model with parameter θ, represents the search for the feature vector of image x in the twin model with parameter θ, and b·l represents the deviation term at each position b; Compare the similarity score value with the preset similarity threshold value to determine whether there is a target human body with the same identity as the target object in the current frame image; when there is a target human body with the same identity as the target object in the current frame image, determine the label of the target human body in the current frame image according to the label of the target object, and record the pixel coordinates of the target human body in the current frame image; when it is recognized that there are other human bodies except the target human body in the current frame image, determine the pixel coordinates and label information of the other human bodies; Based on the pixel coordinates and label information of each human body in each frame of image, determine whether the spatial distance between the current frame of image and the human body with the same label in the previous frame of image is less than a preset distance threshold; when the spatial distance between the human body with the same label in the current frame of image and the previous frame of image is less than the preset distance threshold, determine that there is a target human body with the same identity as the target object in the first video data stream; Furthermore, the method further comprises: Before training the twin model, black patches of preset areas are randomly placed in the two sample images to cover up the most distinctive parts of the two sample images; the first sample image is the original image collected, and the second sample image is an image containing the human body area obtained by cropping the first sample image; During the training process, two sample images including black patches are input into each branch of the twin model, and the feature calculation results output by each branch are all linked. Then, they are input into the linear fully connected layer of the twin model to output the training results.
2. The method according to claim 1, It is characterized in that The step of acquiring a target image containing a target object comprises: Identify the second video data stream and determine each frame image of the target object in the second video data stream; Building a template image library based on each frame image containing the target object; When cross-shot re-identification of the target person is required, a frame of image is randomly selected from the constructed template image library as the target image.
3. The method according to claim 2, It is characterized in that Also includes: When the spatial distance between the human body with the same label in the current frame image and the previous frame image is not less than the preset distance threshold, the label of the corresponding human body in the current frame image is re-determined, and the image of the corresponding human body is deducted and stored in the template image library.
4. The method according to claim 2, It is characterized in that After determining that a target human body with the same identity as the target object exists in the video data stream, the method further includes: extracting an image of the target human body and merging it into a template image library.
5. A cross-lens human target re-identification system in a closed environment. It is characterized in that include: An acquisition module is used to acquire a target image and a first video data stream to be re-identified across shots; wherein the first video data stream is acquired by a first image acquisition device in a closed environment, the target image is any frame image of a target object in a second video data stream, the second video data stream is acquired by a second image acquisition device in a closed environment, and the first image acquisition device and the second image acquisition device are independent of each other; A similarity calculation module is used to use the target image as a template image, the current frame image of the first video data stream as a search image, input the template image and the search image into a pre-trained twin model, and calculate the similarity score value between the feature vector of the template image and the feature vector of the search image; wherein the twin model has two branches with the same architecture and weight set, one branch is used to learn the feature representation of the target, and the other branch is used to search the area; and the twin model is trained using a random patch addition strategy; the twin model uses the following function to calculate the similarity score value between the feature vector of the template image and the feature vector of the search image: In the formula, f θ (z,x) represents the similarity response function between the feature vector of the template image z and the feature vector of the search image x in the twin model with parameter θ; represents the feature vector of the template image z in the twin model with parameter θ, represents the search for the feature vector of image x in the twin model with parameter θ, and b·l represents the deviation term at each position b; The first judgment module is used to compare the similarity score value with a preset similarity threshold value to determine whether there is a target human body with the same identity as the target object in the current frame image; when there is a target human body with the same identity as the target object in the current frame image, determine the label of the target human body in the current frame image according to the label of the target object, and record the pixel coordinates of the target human body in the current frame image; when it is recognized that there are other human bodies except the target human body in the current frame image, determine the pixel coordinates and label information of the other human bodies; A second judgment module is used to judge whether the spatial distance between the current frame image and the human body with the same label in the previous frame image is less than a preset distance threshold based on the pixel coordinates and label information of each human body in each frame image; when the spatial distance between the human body with the same label in the current frame image and the previous frame image is less than the preset distance threshold, it is determined that there is a target human body with the same identity as the target object in the first video data stream; Furthermore, the system also includes a training module for: Before training the twin model, black patches of preset areas are randomly placed in the two sample images to cover up the most distinctive parts of the two sample images; the first sample image is the original image collected, and the second sample image is an image containing the human body area obtained by cropping the first sample image; During the training process, two sample images including black patches are input into each branch of the twin model, and the feature calculation results output by each branch are all linked. Then, they are input into the linear fully connected layer of the twin model to output the training results.
6. The system according to claim 5, It is characterized in that The acquisition module is specifically used for: Identify the second video data stream and determine each frame image of the target object in the second video data stream; Building a template image library based on each frame image containing the target object; When cross-shot re-identification of the target person is required, a frame of image is randomly selected from the constructed template image library as the target image.
7. A computer-readable storage medium, wherein the storage medium stores a computer program, wherein the computer program is used to execute the method according to any one of claims 1 to 4.
8. An electronic device, It is characterized in that The electronic device comprises: processor; a memory for storing instructions executable by the processor; The processor is used to read the executable instructions from the memory and execute the executable instructions to implement the method described in any one of claims 1 to 4 above.
Citation Information
Patent Citations
Cross-border specific pedestrian tracking method and device
CN114639117A
Cross-video target tracking method and system, and electronic device and storage medium
WO2023216572A1
