A target relocation method, device, computer device and storage medium

Abstract

The application discloses a target repositioning method and device, computer equipment and a storage medium, relates to the technical field of computer vision, and comprises the following steps: acquiring a plurality of groups of image sample sets collected by at least two cameras and position information of each camera, and extracting sample prompt information of a target to be positioned from a first image sample set and the position information of the camera; then inputting the sample prompt information and a second image sample set into a pre-trained detection model, so that the detection model detects the second image sample set under the prompt of the sample prompt information to obtain a detection frame; calculating a frame loss, updating weight parameters of the detection model according to the frame loss, and obtaining a target detection model, which is used for target repositioning. The application solves the technical problems of high model training cost, inaccurate target repositioning in the case of environmental changes and multi-target occlusion in the prior art.

Description

Technical Field

[0001] This application relates to the field of computer vision technology, and in particular to a target relocation method, apparatus, computer device, and storage medium. Background Technology

[0002] Target relocation (or visual target tracking) is an important research direction in the field of computer vision. The core goal of target relocation technology is to accurately and quickly identify targets in images or video streams and relocate targets in other videos.

[0003] Current deep learning-based target relocalization techniques obtain feature maps of video frames using deep convolutional neural networks, then predict the target location on the feature maps and mark it with a bounding box (anchor). In subsequent video frames, the target is tracked by calculating the feature similarity within the bounding box.

[0004] However, traditional target relocalization methods have the following problems: First, they rely on a large amount of training data, which requires a lot of manpower to do data annotation tasks; second, the detection accuracy is greatly affected by the environment, such as changes in lighting, changes in surrounding obstacles, and changes in the surrounding scenery, which will affect the recognition results; in addition, there is a high target relocalization error rate when multiple targets appear in the video.

[0005] In summary, although target relocation technology has made great progress in terms of accuracy and speed, there are still problems such as the large amount of training data required for training models, high training costs, and inaccurate target relocation under environmental changes and multi-target occlusion. Summary of the Invention

[0006] This application provides a target relocation method, apparatus, computer device, and storage medium to solve the technical problems of high model training cost, inaccurate target relocation under environmental changes and multi-target occlusion scenarios in the prior art.

[0007] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0008] In a first aspect, embodiments of this application provide a target relocation method, the method comprising:

[0009] Acquire multiple sets of pre-annotated image samples captured by at least two cameras, and the position information of each camera, wherein the annotation includes annotating the position of the target to be located in the image sample set, and the image sample set includes a first image sample set and a second image sample set;

[0010] Extract sample prompt information of the target to be located from the first image sample set and the location information of the camera;

[0011] The sample prompt information and the second image sample set are input into a pre-trained detection model, so that the detection model performs bounding box detection on the second image sample set under the prompt of the sample prompt information to obtain the detected bounding box;

[0012] Calculate the bounding box loss and update the weight parameters of the detection model based on the bounding box loss to obtain the target detection model. The target detection model is used to detect the bounding box of the target to be located in the first video to be detected, obtain the target predicted bounding box and target prompt information, and combine the target prompt information to relocate the target to be located in the second video to be detected.

[0013] In conjunction with the first aspect, in one possible design approach, extracting sample cue information of the target to be located from the first image sample set and the camera's location information includes:

[0014] Semantic extraction is performed on the first image sample set and the location information of the camera respectively to obtain the visual semantic information and location semantic information of the target to be located.

[0015] The visual semantic information and location semantic information are used as the sample prompt information.

[0016] In conjunction with the first aspect, in one possible design approach, the semantic extraction of the first image sample set and the camera's position information to obtain the visual semantic information and positional semantic information of the target to be located includes:

[0017] Feature extraction is performed on the target to be located in the first image sample set to obtain the visual features corresponding to the target to be located, and the visual features are semantically transformed to obtain the visual semantic information of the target to be located;

[0018] Based on the visual features, target detection is performed on the target to be located in the first image sample set to obtain the detection bounding box corresponding to the target to be located; the position information of the camera corresponding to the image sample containing the detection bounding box in the first image sample set is semantically transformed to obtain the position semantic information of the target to be located.

[0019] In conjunction with the first aspect, in one possible design approach, the step of semantically transforming the position information of the camera corresponding to the image sample containing the detection bounding box in the first image sample set to obtain the position semantic information of the target to be located includes:

[0020] Obtain the position information of the camera corresponding to the image sample containing the detection bounding box in the first image sample set;

[0021] According to the time sequence in which the image samples containing the detection borders are captured by the camera, the position information of the camera corresponding to the image samples containing the detection borders is sorted.

[0022] The motion trajectory of the target to be located is generated based on the sorted position information of the cameras, wherein the position semantic information includes the motion trajectory of the target to be located.

[0023] In conjunction with the first aspect, in one possible design approach, the step of performing target detection on the target to be located in the first image sample set based on the visual features to obtain the detection bounding box corresponding to the target to be located includes:

[0024] Based on the visual features, the bounding box of the target to be located is predicted to obtain the bounding box information of multiple predicted bounding boxes corresponding to the target to be located, and the detection bounding box is selected from the multiple predicted bounding boxes.

[0025] In conjunction with the first aspect, in one possible design approach, acquiring a set of multiple image samples captured by at least two cameras includes:

[0026] Acquire multiple video samples captured by at least two cameras;

[0027] Frame extraction is performed on the multiple video samples to obtain the image sample set corresponding to the video sample captured by each camera;

[0028] The number of samples in the second image sample set is less than the number of samples in the first image sample set.

[0029] In conjunction with the first aspect, in one possible design approach, the pre-training includes:

[0030] The first image sample set is used as input, and the labeled bounding box of the target to be located in the first image sample is used as the label. The detection model is trained with the goal of minimizing the loss between the detection bounding box output by the detection model and the labeled bounding box.

[0031] Secondly, embodiments of this application provide a target relocation device, comprising:

[0032] The data acquisition module is used to acquire a set of multiple pre-annotated image samples collected by at least two cameras, as well as the position information of each camera. The annotation includes annotating the position of the target to be located in the set of image samples. The set of image samples includes a first set of image samples and a second set of image samples.

[0033] The information extraction module is used to extract sample prompt information of the target to be located from the first image sample set and the position information of the camera;

[0034] The model training module is used to input the sample prompt information and the second image sample set into a pre-trained detection model, so that the detection model performs bounding box detection on the second image sample set under the prompt of the sample prompt information to obtain the detected bounding box; calculates the bounding box loss, and updates the weight parameters of the detection model according to the bounding box loss to obtain a target detection model. The target detection model is used to perform bounding box detection on the target to be located in the first video to be detected to obtain the target predicted bounding box and the target prompt information, and combines the target prompt information to relocate the target to be located in the second video to be detected.

[0035] Thirdly, embodiments of this application provide a computer device including a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to perform the method of the first aspect and its possible design.

[0036] Fourthly, embodiments of this application provide a storage medium storing a computer program, wherein the computer program is configured to execute the method of the first aspect and its possible design at runtime, and the display screen is configured to display a second audio-visual signal processed by the processor.

[0037] Compared with the prior art, the target relocalization method, apparatus, computer device and storage medium provided in this application first acquire multiple sets of pre-annotated image samples collected by at least two cameras, as well as the position information of each camera. The annotation includes annotating the position of the target to be located in the image sample set, which includes a first image sample set and a second image sample set. Then, sample prompt information of the target to be located is extracted from the first image sample set and the position information of the cameras. The sample prompt information and the second image sample set are input into a pre-trained detection model, so that the detection model performs bounding box detection on the second image sample set under the prompt of the sample prompt information to obtain the detected bounding box. The bounding box loss is calculated, and the weight parameters of the detection model are updated according to the bounding box loss to obtain the target detection model. The target detection model is used to perform bounding box detection on the target to be located in the first video to be detected to obtain the target predicted bounding box and the target prompt information. The target prompt information is then combined to relocalize the target to be located in the second video to be detected. This application fine-tunes the detection model using sample prompts, enabling the target detection model to more accurately detect the target in the second image sample set by combining sample prompts. The model is trained to achieve high recognition accuracy even when the image samples in the first and second image sample sets have different shooting angles and shooting locations, with the help of sample prompts. This method improves the target detection model's ability to recognize targets in changing environments and multi-target occlusion scenarios. In addition, the number of image samples required for fine-tuning is small and the number of training rounds is small, thus reducing the training cost.

[0038] Details of one or more embodiments of this application are set forth in the following drawings and description to make other features, objects and advantages of this application more readily apparent. Attached Figure Description

[0039] The accompanying drawings, which are included to provide a further understanding of this application and form part of this application, illustrate exemplary embodiments and are used to explain this application, but do not constitute an undue limitation of this application. In the drawings:

[0040] Figure 1 This illustration shows a hardware structure block diagram of a computer device according to an embodiment of this application;

[0041] Figure 2 A flowchart of a target relocation method provided in an embodiment of this application is shown;

[0042] Figure 3 A schematic diagram of a prompt word generation model provided in an embodiment of this application is shown;

[0043] Figure 4A schematic diagram of a target relocation method provided in an embodiment of this application is shown;

[0044] Figure 5 A schematic diagram illustrating a fine-tuning process for a target model provided in an embodiment of this application is shown;

[0045] Figure 6 A schematic diagram illustrating the reasoning process of a target model provided in an embodiment of this application is shown;

[0046] Figure 7 A structural block diagram of a target relocation device provided in an embodiment of this application is shown. Detailed Implementation

[0047] To better understand the purpose, technical solution, and advantages of this application, the application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0048] Unless otherwise defined, the technical or scientific terms used in this application shall have the general meaning understood by one of ordinary skill in the art to which this application pertains. Words such as “a,” “an,” “an,” “the,” “the,” and “these,” used in this application, do not indicate quantitative limitation and may be singular or plural. The terms “comprising,” “including,” “having,” and any variations thereof used in this application are intended to cover non-exclusive inclusion; for example, a process, method, apparatus, product, or device that comprises a series of steps or modules (units) is not limited to the listed steps or modules (units) but may include steps or modules (units) not listed, or may include other steps or modules (units) inherent to such processes, methods, products, or devices. The terms “connected,” “linked,” and “coupled,” used in this application, are not limited to physical or mechanical connections but may include electrical connections, whether direct or indirect. The term “multiple” used in this application refers to two or more. The "and / or" operator describes the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: A alone, A and B simultaneously, and B alone. Typically, the character " / " indicates that the objects before and after it are in an "or" relationship. The terms "first," "second," and "third," etc., used in this application are merely for distinguishing similar objects and do not represent a specific ordering of the objects.

[0049] This application provides a target relocalization method that extracts sample cue information of the target to be located from a first image sample set and the camera's position information. This sample cue information helps the detection model more accurately detect the target in the next video segment, reducing interference from the environment and crowds. It achieves better detection accuracy during training, and this training is essentially a fine-tuning of the detection model, requiring significantly less data and fewer training iterations, thus lowering training costs. Therefore, this application solves the technical problems of high model training costs and inaccurate target relocalization in scenarios with environmental changes and multi-target occlusion.

[0050] This method can be executed on a terminal, computer, or similar testing system. Taking running on a computer as an example... Figure 1 A hardware structure block diagram of a computer device provided in an embodiment of this application is shown. Figure 1 As shown, a computer device may include one or more ( Figure 1 Only one is shown in the diagram. A processor 102 and a memory 104 for storing data are also included. The processor 102 may be, but is not limited to, a microprocessor (MCU) or a programmable logic device (FPGA). The aforementioned computer device may also include a transmission device 106 for communication functions and an input / output device 108. Those skilled in the art will understand that… Figure 1 The structure shown is for illustrative purposes only and does not limit the structure of the computer device described above. For example, the computer device may also include components that are more... Figure 1 The more or fewer components shown, or having the same Figure 1 The different configurations shown.

[0051] Memory 104 can be used to store computer programs, such as application software programs and modules. Processor 102 executes various functional applications and data processing by running the computer programs stored in memory 104, thereby implementing the methods described above. Memory 104 can be used to store data, such as image samples, videos to be detected, etc. Memory 104 may include high-speed random access memory and non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, memory 104 may further include memory remotely located relative to processor 102, and these remote memories can be connected to computer devices via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0052] The transmission device 106 is used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by a communication provider for the computer equipment. In one example, the transmission device 106 includes a Network Interface Controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the transmission device 106 may be a Radio Frequency (RF) module used for wireless communication with the Internet.

[0053] This application's embodiments can be applied to scenarios such as case investigation, autonomous driving, traffic monitoring, and human-computer interaction, where a target to be located is extracted from one part of video and relocated to another part of the video. Taking case investigation as an example, video data is collected from various camera locations along the road. From one part of the video data, visual descriptions (such as clothing color and physical appearance) and the movement trajectory (such as movement route) of the target of interest are extracted. Using the visual descriptions and movement trajectories as prompts, the target of interest is relocated in another part of the video data, thereby finding the target of interest at other camera locations. By repeating this process, the target of interest can be continuously tracked and found from new locations, thus assisting in case investigation.

[0054] The target to be located can be a person, vehicle, traffic sign, license plate, etc. A target detection model is trained to relocate the target in the video to be detected, and the category and location of the target are labeled in the video. The location of the target can be labeled using detection bounding boxes, with the target positioned within the detection bounding boxes. The following mainly uses a person as an example to illustrate the target relocation method provided in this application.

[0055] Figure 2 A flowchart of a target relocation method provided in an embodiment of this application is shown, as follows: Figure 2 As shown, the method includes steps S201 to S204.

[0056] Step S201: Obtain a set of multiple pre-annotated image samples collected by at least two cameras, and the position information of each camera. The annotation includes annotating the position of the target to be located in the image sample set. The image sample set includes a first image sample set and a second image sample set.

[0057] The image sample set includes multiple image samples. The annotation can be done by software in the image samples. The annotation includes the category and the first position, which is represented by coordinates.

[0058] The category refers to the type of object, usually a label such as "target person" or "person". The coordinates are the position of the target person in the image sample, which is usually represented by a bounding box, containing the pixel coordinates of the top left and bottom right corners of the rectangle containing the target person. These two pixel coordinates determine the exact position and size of the target person in the image sample.

[0059] In this step, the image sample set is divided into at least two parts: a first image sample set and a second image sample set. The first image sample set is used to pre-train the detection model, enabling it to extract feature information of the target person. The first image sample set is also used to determine sample prompts. The difference between the first and second image sample sets can be that the areas captured by the cameras are different, or that the cameras have different viewing angles. For example, some samples in the first image sample set are captured by camera A at the entrance of a shopping mall from a first-person perspective, while others are captured by camera B at the entrance of the shopping mall from a second-person perspective; cameras A and B have different monitoring angles at the shopping mall entrance. Similarly, some samples in the second image sample set are captured by camera C at a traffic light intersection from a third-person perspective, while others are captured by camera D inside a supermarket from a fourth-person perspective; cameras C and D monitor different areas and have different monitoring angles. The model trained using the first and second image sample sets then possesses the ability to track targets from different perspectives, with different camera positions, and in different monitoring areas. In this embodiment, the sample prompt information is text information, such as prompt words or prompt text. The sample prompt information can also be in other forms such as voice or commands. The second image sample set is used to fine-tune the pre-trained detection model, enabling it to achieve better recognition performance in complex and multi-target environments.

[0060] In some embodiments, pre-training includes: taking a first set of image samples as input, using labeled bounding boxes that annotate the target to be located in the first set of image samples as labels, and training the detection model with the objective of minimizing the loss between the detected bounding boxes output by the detection model and the labeled bounding boxes. The pre-trained detection model has the ability to extract feature information of the target person. Therefore, when a set of image samples is input into the detection model, the detection model can identify the target person in the set of image samples and label the target person in the image samples by means of detected bounding boxes.

[0061] Image samples can be a set of consecutive images from a video, or they can be obtained by extracting frames from the video. Specifically, acquiring multiple sets of image samples captured by at least two cameras includes: acquiring multiple video samples captured by at least two cameras; and performing frame extraction on the multiple video samples to obtain the image sample set corresponding to the video sample captured by each camera. Frame extraction reduces redundant video frames when adjacent frames in the video contain highly similar information, thus reducing the amount of computation.

[0062] In some embodiments, the number of samples in the second image sample set is less than the number of samples in the first image sample set. It is understood that the detection model is fine-tuned using the second image sample set under the guidance of sample cue information; therefore, the second image sample set has a smaller data volume, and fewer training epochs are required for fine-tuning the model, resulting in lower training costs.

[0063] Step S202: Extract sample prompt information of the target to be located from the first image sample set and the camera's location information.

[0064] In this step, a prompt word generation model can be trained to extract prompt information from samples. Specifically... Figure 3 The diagram illustrates a prompt word generation model provided in an embodiment of this application, such as... Figure 3 As shown, the prompt word generation model comprises two main modules: a visual text representation module and a prompt word output module. In the visual text representation module, multiple video frames (equivalent to the first set of image samples) are input into an image encoder for feature extraction, and the camera's position information is input into a text encoder for feature extraction. The features output from these two encoders are then input into a Q-Former for multimodal feature fusion. In the prompt word output module, a large language model performs semantic transformation on the input multimodal features and outputs prompt words describing the target person.

[0065] Large Language Models (LLMs) are a type of artificial intelligence model that learns to understand and generate human language by studying large amounts of text data. These models are typically based on a Transformer architecture, learning basic language patterns and structures through pre-training on extensive text data, thereby understanding the syntax, semantics, and contextual relationships of language. After pre-training, large language models can be fine-tuned to adapt to specific tasks or applications, such as text generation, text classification, and sentiment analysis. As an example, a large language model can be a language model based on VisualGLM. VisualGLM-6B is an open-source, multimodal dialogue language model supporting images, Chinese, and English. The image part uses BLIP2-Qformer to bridge the visual and language models. BLIP-2 (Bootstrapping Language-Image Pre-training) introduces a new visual language pre-training paradigm to arbitrarily combine and fully utilize two pre-trained visual encoders and an LLM, without requiring end-to-end pre-training of the entire architecture. This achieves accurate outputs on multiple visual language tasks while significantly reducing the number of training parameters and pre-training costs.

[0066] In some embodiments, step S202 includes: performing semantic extraction on the first image sample set and the camera's location information respectively to obtain the visual semantic information and location semantic information of the target to be located; and using the visual semantic information and location semantic information as sample prompt information.

[0067] In this embodiment, the sample prompt information includes two parts. The first is visual semantic information extracted from the first image sample set. This visual semantic information refers to the characteristics of the target person that can be seen from the image, such as the target person's gender, whether they are wearing glasses, their clothing, and their age. The second is location semantic information extracted by combining the camera's location information and the first image sample set. The location semantic information is used to describe the location of the camera that captured the appearance of the target. The following explains how to extract the location semantic information: Camera A collects image sample set a, camera B collects image sample set b, and camera C collects image sample set c. The target person is detected in image sample set a and image sample set b, respectively. Combining the location information 1 of camera A corresponding to image sample set a and the location information 3 of camera C corresponding to image sample set c, the location semantic information can be extracted as follows: the target person has appeared in location information 1 and location information 3.

[0068] In some embodiments, the location semantic information is obtained by semantically transforming the visual features after feature extraction from the first image sample set, and by semantically transforming the location information of the detected target person. Specifically, the above-mentioned semantic extraction of the first image sample set and the camera's location information to obtain the visual semantic information and location semantic information of the target to be located includes: extracting features from the target to be located in the first image sample set to obtain the visual features corresponding to the target to be located, and semantically transforming the visual features to obtain the visual semantic information of the target to be located; performing target detection on the target to be located in the first image sample set based on the visual features to obtain the detection bounding box corresponding to the target to be located; and semantically transforming the camera's location information corresponding to the image sample containing the detection bounding box in the first image sample set to obtain the location semantic information of the target to be located.

[0069] In this embodiment, visual features of the target person are extracted using a feature extraction network. Prior to this, the feature extraction network can be pre-trained using labeled visual feature samples, enabling it to extract visual features. After extraction, the visual features are converted into visual semantic information using a large language model, such as: the target is wearing a red top, black pants, and sunglasses. Then, the location information of all cameras is input into the large language model, which can extract location semantic information from the location information of the cameras that detected the target person, such as: the target person passed by a supermarket and A Elementary School. This combination of visual and location-based cues provides a more comprehensive description of the target person, offering more accurate information and aiding in training the detection model to achieve more accurate target relocalization capabilities, while being less susceptible to interference from the environment and multiple target occlusions.

[0070] In one embodiment, the target detection of the target to be located in the first image sample set based on visual features and the resulting detection bounding box of the target to be located includes: predicting the bounding box of the target to be located based on visual features, obtaining the bounding box information of multiple predicted bounding boxes of the target to be located, and filtering out the detection bounding box from the multiple predicted bounding boxes.

[0071] In some embodiments, the movement trajectory of a target person can be determined from multiple camera locations by combining the time when the sample images were acquired. Specifically, the above-mentioned semantic transformation of the position information of the cameras corresponding to the image samples containing detection borders in the first image sample set to obtain the position semantic information of the target to be located includes: obtaining the position information of the cameras corresponding to the image samples containing detection borders in the first image sample set; sorting the position information of the cameras corresponding to the image samples containing detection borders according to the time order in which the image samples containing detection borders were acquired by the cameras; and generating the movement trajectory of the target to be located based on the sorted position information of the cameras, wherein the position semantic information includes the movement trajectory of the target to be located.

[0072] In this embodiment, the location semantic information includes movement trajectories, thus describing the sequence of the target person's movements at various points. For example, the location semantic information could be: the target person first appeared in a supermarket, passed by Elementary School A, and last appeared in a shopping district. Movement trajectories provide a more comprehensive description of the target person, thus allowing for better fine-tuning of the detection model.

[0073] In other embodiments, location semantic information includes movement trajectory and time of appearance. Specifically, by combining the capture timestamp of the image sample containing the target person, the time of appearance of the target person can be determined. This time of appearance can be used as part of the location semantic information to optimize the detection model. For example, the location semantic information is: the target person appeared in a supermarket at 16:32 on November 26, 2024, passed by Elementary School A at 16:38 on November 26, 2024, and last appeared in a shopping area at 16:57 on November 26, 2024. Combining the movement trajectory with the timestamp provides a more comprehensive description of the target person's location, thus providing richer cue features for the detection model.

[0074] Step 203: Input the sample prompt information and the second image sample set into the pre-trained detection model, so that the detection model can perform border detection on the second image sample set under the prompt of the sample prompt information and obtain the detection border.

[0075] In traditional target relocalization techniques, a target person may be located in one video segment but missed in the next, leading to the target being lost. This step inputs sample cue information and a second set of image samples into the detection model. This is equivalent to inputting the descriptive features of the target person from the previous video along with those from the next video into the model for recognition. This allows the model to identify the target person based on the descriptive features and annotate them in the image samples as bounding boxes. The bounding boxes can be understood as the target person identified by the pre-trained detection model based on the sample cue information and the second set of image samples, given the original model parameters. Subsequent training of the detection model by minimizing the loss function updates the model parameters, giving them better relocalization capabilities. For details on the training process, please refer to step S204.

[0076] Step S204: Calculate the bounding box loss and update the weight parameters of the detection model according to the bounding box loss to obtain the target detection model. The target detection model is used to detect the bounding box of the target to be located in the first video to be detected, obtain the target predicted bounding box and target prompt information, and combine the target prompt information to relocate the target to be located in the second video to be detected.

[0077] Boundary loss refers to the difference between the detected bounding box and the labeled bounding box. The loss function of the detection model is constructed using the boundary loss, and the detection model is further optimized (or fine-tuned) with the goal of minimizing the loss function to obtain the object detection model. Compared with the detection model, the object detection model learns the relationship between the sample prompt information and the second video sample set, so it has a higher accuracy in object detection in complex scenes. Therefore, it is not easy to lose the target person, that is, it has a stronger target relocalization ability.

[0078] In the embodiments described in steps S201 to S204 above, firstly, multiple sets of pre-annotated image samples collected by at least two cameras are acquired, along with the position information of each camera. The annotation includes annotating the position of the target to be located in the image sample set, which includes a first image sample set and a second image sample set. Then, sample prompt information of the target to be located is extracted from the first image sample set and the position information of the cameras. Next, the sample prompt information and the second image sample set are input into a pre-trained detection model, so that the detection model performs bounding box detection on the second image sample set under the prompt of the sample prompt information to obtain the detected bounding box. The bounding box loss is calculated, and the weight parameters of the detection model are updated according to the bounding box loss to obtain the target detection model. The target detection model is used to perform bounding box detection on the target to be located in the first video to be detected, to obtain the target predicted bounding box and the target prompt information, and to relocate the target to be located in the second video to be detected in combination with the target prompt information. This application fine-tunes the detection model by using sample prompts, enabling the target detection model to better detect the target to be located in the second image sample set by combining sample prompts. In this way, the model's ability to recognize targets in environmental changes and multi-target occlusion scenarios is improved. In addition, the number of image samples required for fine-tuning is small and the number of training rounds is small, so the training cost is low.

[0079] The method provided in this application embodiment will be further described below using case investigation as an example. This method is applied to a server. Figure 4 The diagram illustrates a target relocation method provided in an embodiment of this application, as shown below. Figure 4 As shown, in this method, video data is first acquired. The server extracts frames from the video data, using a portion as labeled data to train the model and another portion as test data to test the model. The server deploys the target model on a GPU (Graphics Processing Unit), using labeled and test data to train and optimize the target model to enable it to relocalize the target, outputting a prediction of the target's position in the image. The target model first needs to undergo extensive image pre-training to obtain feature representation capabilities, and then train through an object detection task to obtain the ability to mark and detect bounding boxes. Then, a prompt word generation module generates prompt words from a small number of fine-tuned samples for prompt word fine-tuning. The fine-tuned target model can then relocalize the target person in video frames captured by other camera devices.

[0080] Figure 5 The diagram illustrates a fine-tuning process for a target model according to an embodiment of this application, as shown below. Figure 5As shown, the process of fine-tuning the target model using sample prompts includes: inputting video frames and prompt words into the backbone network, where the backbone network extracts features and the detection head is responsible for category prediction and detection regression to obtain the category and detection bounding box. The loss is calculated, including bounding box loss and category loss, and the weight parameters are updated based on the bounding box loss and category loss, thereby achieving fine-tuning of the target model.

[0081] After obtaining the target model, inference can be performed using the target model. Figure 6 The diagram illustrates the reasoning process of a target model provided in an embodiment of this application, as shown below. Figure 6 As shown, the video frame and the previous video prompt are input into the backbone network, which extracts features and outputs the prediction result through the detection head.

[0082] The backbone network can use an image encoder, and the detection head is composed of an MLP (Multilayer Perceptron). The features output by the backbone network are processed by the MLP and then organized into multiple predicted bounding boxes. The predicted bounding box with the highest confidence is then selected as the final output of the detection bounding box.

[0083] Figure 7 This paper illustrates a structural block diagram of a target relocation device provided in an embodiment of this application, as shown below. Figure 7 As shown, the device includes:

[0084] The data acquisition module 71 is used to acquire multiple sets of pre-annotated image samples collected by at least two cameras, as well as the position information of each camera. The annotation includes annotating the position of the target to be located in the image sample set. The image sample set includes a first image sample set and a second image sample set.

[0085] The information extraction module 72 is used to extract sample prompt information of the target to be located from the first image sample set and the position information of the camera;

[0086] The model training module 73 is used to input the sample prompt information and the second image sample set into the pre-trained detection model, so that the detection model can perform bounding box detection on the second image sample set under the prompt of the sample prompt information to obtain the detected bounding box; calculate the bounding box loss, and update the weight parameters of the detection model according to the bounding box loss to obtain the target detection model. The target detection model is used to perform bounding box detection on the target to be located in the first video to be detected, obtain the target predicted bounding box and the target prompt information, and combine the target prompt information to relocate the target to be located in the second video to be detected.

[0087] In some embodiments, the data acquisition module 71 is further configured to acquire multiple video samples captured by at least two cameras; perform frame extraction processing on the multiple video samples to obtain an image sample set corresponding to the video sample captured by each camera; wherein the number of samples in the second image sample set is less than the number of samples in the first image sample set.

[0088] In some embodiments, the information extraction module 72 is further configured to perform semantic extraction on the first image sample set and the camera position information respectively, and obtain the visual semantic information and position semantic information of the target to be located; and use the visual semantic information and position semantic information as sample prompt information.

[0089] In some embodiments, the information extraction module 72 is further configured to extract features from the target to be located in the first image sample set to obtain visual features corresponding to the target to be located, and to perform semantic transformation on the visual features to obtain visual semantic information of the target to be located; to perform target detection on the target to be located in the first image sample set based on the visual features to obtain detection bounding boxes corresponding to the target to be located; and to perform semantic transformation on the position information of the camera corresponding to the image sample containing the detection bounding box in the first image sample set to obtain position semantic information of the target to be located.

[0090] In some embodiments, the information extraction module 72 is further configured to obtain the position information of the camera corresponding to the image sample containing the detection border in the first image sample set; sort the position information of the camera corresponding to the image sample containing the detection border according to the time order in which the image sample containing the detection border was acquired by the camera; and generate the motion trajectory of the target to be located based on the sorted position information of the camera, wherein the position semantic information includes the motion trajectory of the target to be located.

[0091] In some embodiments, the model training module 73 is further configured to predict the bounding box of the target to be located based on visual features, obtain the bounding box information of multiple predicted bounding boxes corresponding to the target to be located, and filter out the detection bounding box from the multiple predicted bounding boxes.

[0092] In some embodiments, the model training module 73 is further configured to take the first image sample set as input, use the labeled bounding boxes that annotate the target to be located in the first image sample as labels, and train the detection model with the goal of minimizing the loss between the detection bounding boxes output by the detection model and the labeled bounding boxes.

[0093] It should be noted that the above modules can be functional modules or program modules, and can be implemented through software or hardware. For modules implemented through hardware, the above modules can reside in the same processor; or the above modules can be located in different processors in any combination.

[0094] This embodiment also provides a computer device, including a memory and a processor, wherein the memory stores a computer program and the processor is configured to run the computer program to perform the steps in any of the above method embodiments.

[0095] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor and the input / output device is connected to the processor.

[0096] Optionally, in this embodiment, the processor can be configured to execute steps S1 to S4 via a computer program.

[0097] S1. Obtain a set of multiple pre-annotated image samples collected by at least two cameras, and the position information of each camera. The annotation includes annotating the position of the target to be located in the image sample set. The image sample set includes a first image sample set and a second image sample set.

[0098] S2. Extract sample prompt information of the target to be located from the first image sample set and the camera's position information.

[0099] S3. Input the sample prompt information and the second image sample set into the pre-trained detection model so that the detection model can perform bounding box detection on the second image sample set under the prompt of the sample prompt information and obtain the detection bounding box.

[0100] S4. Calculate the bounding box loss and update the weight parameters of the detection model based on the bounding box loss to obtain the target detection model. The target detection model is used to detect the bounding box of the target to be located in the first video to be detected, obtain the target predicted bounding box and target prompt information, and combine the target prompt information to relocate the target to be located in the second video to be detected.

[0101] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementations, and will not be repeated in this embodiment.

[0102] Furthermore, in conjunction with the methods provided in the above embodiments, this embodiment can also provide a storage medium for implementation. This storage medium stores a computer program; when executed by a processor, the computer program implements any of the target relocation methods in the above embodiments.

[0103] This application also provides a computer program product that, when run on a computer, causes the computer to perform various functions or steps executed by the processor in the above method embodiments.

[0104] It should be understood that the specific embodiments described herein are merely illustrative of the application and not intended to limit it. All other embodiments derived by those skilled in the art based on the embodiments provided in this application without inventive effort are within the scope of protection of this application.

[0105] Obviously, the accompanying drawings are merely some examples or embodiments of this application. Those skilled in the art can apply this application to other similar situations based on these drawings without any creative effort. Furthermore, it is understood that although the work done in this development process may be complex and lengthy, for those skilled in the art, certain design, manufacturing, or production modifications made based on the technical content disclosed in this application are merely conventional technical means and should not be considered as insufficient disclosure of this application.

[0106] The term "embodiment" in this application refers to a specific feature, structure, or characteristic described in connection with an embodiment that may be included in at least one embodiment of this application. The appearance of this phrase in various places in the specification does not necessarily imply the same embodiment, nor does it imply that it is mutually exclusive with or independent of other embodiments. It will be clearly or implicitly understood by those skilled in the art that the embodiments described in this application may be combined with other embodiments without conflict.

[0107] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of patent protection. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the appended claims.

Claims

1. A target relocation method, characterized by, The method comprises: acquiring a plurality of sets of pre-labeled image samples collected by at least two cameras and position information of each camera, wherein the labeling comprises labeling the position of a target to be positioned in the image sample set, and the image sample set comprises a first image sample set and a second image sample set; extracting sample prompt information of the target to be positioned from the first image sample set and the position information of the camera; the extraction of the sample prompt information of the target to be positioned from the first image sample set and the position information of the camera comprises: respectively performing semantic extraction on the first image sample set and the position information of the camera to obtain visual semantic information and position semantic information of the target to be positioned; and taking the visual semantic information and the position semantic information as the sample prompt information; inputting the sample prompt information and the second image sample set into a pre-trained detection model to enable the detection model to detect a bounding box under the prompting of the sample prompt information, and obtain a detection bounding box; calculating a bounding box loss, updating weight parameters of the detection model according to the bounding box loss, and obtaining a target detection model, which is used to detect a bounding box of the target to be positioned in a first video to be detected, obtain a target prediction bounding box and target prompt information, and reposition the target to be positioned in a second video to be detected in combination with the target prompt information.

2. The target relocation method according to claim 1, wherein The respective semantic extraction of the first image sample set and the position information of the camera to obtain the visual semantic information and the position semantic information of the target to be positioned comprises: performing feature extraction on the target to be positioned in the first image sample set to obtain visual features corresponding to the target to be positioned, and performing semantic conversion on the visual features to obtain visual semantic information of the target to be positioned; performing target detection on the target to be positioned in the first image sample set based on the visual features to obtain a detection bounding box corresponding to the target to be positioned; and performing semantic conversion on position information of the camera corresponding to the image sample containing the detection bounding box in the first image sample set to obtain position semantic information of the target to be positioned.

3. The target relocation method according to claim 2, wherein The semantic conversion of the position information of the camera corresponding to the image sample containing the detection bounding box in the first image sample set to obtain the position semantic information of the target to be positioned comprises: acquiring the position information of the camera corresponding to the image sample containing the detection bounding box in the first image sample set; sorting the position information of the camera corresponding to the image sample containing the detection bounding box in the order of time when the image sample is collected by the camera; generating a motion trajectory of the target to be positioned based on the sorted position information of the camera, wherein the position semantic information comprises the motion trajectory of the target to be positioned.

4. The target relocation method according to claim 2, wherein The target detection on the to-be-positioned target in the first image sample set based on the visual feature comprises: The to-be-positioned target is subjected to frame prediction based on the visual feature, frame information of a plurality of predicted frames of the to-be-positioned target is obtained, and a detection frame is screened out from the plurality of predicted frames.

5. The target relocation method according to any one of claims 1 to 4, characterized in that, The plurality of groups of image samples collected by the at least two cameras comprises: A plurality of video samples collected by the at least two cameras are obtained. Frame extraction processing is performed on the plurality of video samples, and an image sample set corresponding to a video sample collected by each camera is obtained. The number of samples in the second image sample set is less than the number of samples in the first image sample set.

6. The target relocation method of claim 1, wherein, The pre-training comprises: The first image sample set is taken as input, a labeled frame of the to-be-positioned target in the first image sample is taken as a label, and the detection model is trained with the objective of minimizing the loss between the detection frame output by the detection model and the labeled frame.

7. A target repositioning device, comprising: Comprise: A data acquisition module is configured to acquire a plurality of groups of pre-labeled image samples collected by at least two cameras and position information of each camera, wherein the labeling comprises labeling the position of a to-be-positioned target in the image sample set, and the image sample set comprises a first image sample set and a second image sample set. An information extraction module is configured to extract sample prompt information of the to-be-positioned target from the first image sample set and the position information of the camera, wherein the extraction of the sample prompt information of the to-be-positioned target from the first image sample set and the position information of the camera comprises: performing semantic extraction on the first image sample set and the position information of the camera respectively, and correspondingly obtaining visual semantic information and position semantic information of the to-be-positioned target; and taking the visual semantic information and the position semantic information as the sample prompt information. A model training module is configured to input the sample prompt information and the second image sample set into a pre-trained detection model, so that the detection model performs frame detection on the second image sample set under the prompting of the sample prompt information to obtain a detection frame; calculate a frame loss, and update the weight parameters of the detection model according to the frame loss to obtain a target detection model, wherein the target detection model is used for performing frame detection on the to-be-positioned target in a first to-be-detected video to obtain a target predicted frame and target prompt information, and repositioning the to-be-positioned target in a second to-be-detected video in combination with the target prompt information. 8.A computer device, comprising a memory and a processor, and characterized in that, The memory stores a computer program, and the processor is configured to run the computer program to execute the target repositioning method in any one of claims 1 to 6.

9. A storage medium, characterized by The storage medium stores a computer program, wherein the computer program is configured to execute the target repositioning method in any one of claims 1 to 6 when running.

Images