Target repositioning method and device, computer equipment and storage medium

Abstract

The invention discloses a target repositioning method and device, computer equipment and a storage medium, and relates to the technical field of computer vision, and the method comprises the steps: obtaining a plurality of image sample sets which are collected by at least two cameras and are labeled in advance, and the position information of each camera, extracting sample prompt information of the to-be-positioned target from the first image sample set and the position information of the camera; inputting the sample prompt information and the second image sample set into a pre-trained detection model, so that the detection model performs frame detection on the second image sample set under the prompt of the sample prompt information to obtain a detection frame; and frame loss is calculated, weight parameters of the detection model are updated according to the frame loss, a target detection model is obtained, and the target detection model is used for target relocation. The technical problems of high model training cost, environment change and inaccurate target relocation in a multi-target shielding scene in the prior art are solved.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a target relocation method, device, computer equipment and storage medium. Background Art

[0002] Target relocalization (or visual target tracking) is an important research direction in the field of computer vision. The core goal of target relocalization technology is to accurately and quickly identify targets in images or video streams, and relocate targets in other videos.

[0003] At present, the target relocalization technology based on deep learning obtains the feature map of the video frame through the deep convolutional neural network, and then predicts the target position on the feature map to mark the detection frame (anchor). In the subsequent video frames, the target is tracked by calculating the feature similarity in the detection frame.

[0004] However, traditional target relocalization methods have the following problems: First, they rely on a large amount of training data, which requires a lot of manpower to do data labeling tasks; second, the detection accuracy is greatly affected by the environment. For example, changes in lighting, changes in surrounding obstacles, and changes in surrounding scenery will affect the recognition results; in addition, when multiple targets appear in the video, there is a high error rate in target relocalization.

[0005] In summary, although the target relocalization technology has been greatly improved in accuracy and speed, the existing problems of large amount of training data required for training models and high training cost, as well as the problem of inaccurate target relocalization in the face of environmental changes and multi-target occlusion have not yet been effectively solved. Summary of the invention

[0006] The embodiments of the present application provide a target relocation method, apparatus, computer device and storage medium to solve the technical problems in the prior art such as high model training cost, environmental changes and inaccurate target relocation in multi-target occlusion scenarios.

[0007] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0008] In a first aspect, an embodiment of the present application provides a target relocation method, the method comprising:

[0009] Acquire a plurality of sets of pre-annotated image sample sets collected by at least two cameras, and position information of each of the cameras, wherein the annotation includes annotating the position of the target to be located in the image sample set, and the image sample set includes a first image sample set and a second image sample set;

[0010] Extracting sample prompt information of the target to be located from the first image sample set and the position information of the camera;

[0011] Inputting the sample prompt information and the second image sample set into a pre-trained detection model, so that the detection model performs border detection on the second image sample set under the prompt of the sample prompt information to obtain a detection border;

[0012] The border loss is calculated, and the weight parameters of the detection model are updated according to the border loss to obtain a target detection model. The target detection model is used to perform border detection on the target to be located in the first video to be detected, obtain a target predicted border and target prompt information, and relocate the target to be located in the second video to be detected in combination with the target prompt information.

[0013] In combination with the first aspect, in a possible design manner, extracting sample prompt information of the target to be located from the first image sample set and the position information of the camera includes:

[0014] Semantically extracting the first image sample set and the position information of the camera respectively, and correspondingly obtaining visual semantic information and position semantic information of the target to be located;

[0015] The visual semantic information and the position semantic information are used as the sample prompt information.

[0016] In combination with the first aspect, in a possible design manner, the semantic extraction is performed on the first image sample set and the position information of the camera respectively, and the visual semantic information and the position semantic information of the target to be located are correspondingly obtained, including:

[0017] Extracting features of the target to be located in the first image sample set to obtain visual features corresponding to the target to be located, and performing semantic conversion on the visual features to obtain visual semantic information of the target to be located;

[0018] Based on the visual features, target detection is performed on the target to be located in the first image sample set to obtain a detection bounding box corresponding to the target to be located; and the position information of the camera corresponding to the image sample containing the detection bounding box in the first image sample set is semantically converted to obtain the position semantic information of the target to be located.

[0019] In combination with the first aspect, in a possible design manner, performing semantic conversion on the position information of the camera corresponding to the image sample containing the detection border in the first image sample set to obtain the position semantic information of the target to be located includes:

[0020] Obtaining position information of the camera corresponding to the image sample containing the detection border in the first image sample set;

[0021] Sort the position information of the camera corresponding to the image samples including the detection frame according to the time sequence in which the image samples including the detection frame are collected by the camera;

[0022] The motion trajectory of the target to be located is generated based on the sorted position information of the cameras, wherein the position semantic information includes the motion trajectory of the target to be located.

[0023] In combination with the first aspect, in a possible design manner, performing target detection on the target to be located in the first image sample set based on the visual feature to obtain a detection bounding box corresponding to the target to be located includes:

[0024] Based on the visual features, a border prediction is performed on the target to be located, border information of multiple predicted borders corresponding to the target to be located is obtained, and a detection border is screened out from the multiple predicted borders.

[0025] In combination with the first aspect, in a possible design manner, obtaining a plurality of sets of image sample sets collected by at least two cameras includes:

[0026] Acquire a plurality of video samples collected by at least two cameras;

[0027] Performing frame extraction processing on the multiple video samples to obtain a set of image samples corresponding to the video samples collected by each camera;

[0028] The number of samples in the second image sample set is smaller than the number of samples in the first image sample set.

[0029] In combination with the first aspect, in a possible design manner, the pre-training includes:

[0030] The first image sample set is taken as input, and the annotated bounding box of the target to be located in the first image sample is taken as a label. The detection model is trained with the goal of minimizing the loss between the detection bounding box output by the detection model and the annotated bounding box.

[0031] In a second aspect, an embodiment of the present application provides a target relocation device, comprising:

[0032] A data acquisition module, used to acquire a plurality of sets of image samples collected by at least two cameras and labeled in advance, and the position information of each of the cameras, wherein the labeling includes labeling the position of the target to be located in the image sample set, and the image sample set includes a first image sample set and a second image sample set;

[0033] An information extraction module, used to extract sample prompt information of the target to be located from the first image sample set and the position information of the camera;

[0034] A model training module is used to input the sample prompt information and the second image sample set into a pre-trained detection model, so that the detection model performs border detection on the second image sample set under the prompt of the sample prompt information to obtain a detection border; calculates the border loss, and updates the weight parameters of the detection model according to the border loss to obtain a target detection model, wherein the target detection model is used to perform border detection on the target to be located in the first video to be detected, obtain a target predicted border and target prompt information, and relocate the target to be located in the second video to be detected in combination with the target prompt information.

[0035] In a third aspect, an embodiment of the present application provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor is configured to run the computer program to execute the method of the first aspect and possible design methods thereof.

[0036] In a fourth aspect, an embodiment of the present application provides a storage medium storing a computer program, wherein the computer program is configured to execute the method of the first aspect and its possible design method when running, and the display screen is configured to display the second video and audio processed by the processor.

[0037] Compared with the prior art, the embodiments of the present application provide a target relocation method, device, computer equipment and storage medium, which first obtains a plurality of sets of pre-annotated image sample sets collected by at least two cameras, and the position information of each camera, wherein the annotation includes annotating the position of the target to be located in the image sample set, and the image sample set includes a first image sample set and a second image sample set; then, sample prompt information of the target to be located is extracted from the first image sample set and the position information of the camera, and the sample prompt information and the second image sample set are input into a pre-trained detection model, so that the detection model performs border detection on the second image sample set under the prompt of the sample prompt information to obtain a detection border; calculates the border loss, and updates the weight parameters of the detection model according to the border loss to obtain a target detection model, the target detection model is used to perform border detection on the target to be located in the first video to be detected, obtains a target predicted border and target prompt information, and relocates the target to be located in the second video to be detected in combination with the target prompt information. The present application fine-tunes the detection model through sample prompt information, so that the obtained target detection model has the ability to detect the target to be located in the second image sample set more accurately in combination with the sample prompt information. The model is trained with the purpose of having high recognition accuracy under the prompt of the sample prompt information even if the image samples in the first image sample set and the second image sample set have different shooting angles and shooting addresses. In this way, the trained target detection model is improved in its ability to recognize targets in scenes with environmental changes and multi-target occlusion. In addition, the number of image samples required for fine-tuning is small and the number of training rounds is small, so the training cost is low.

[0038] Details of one or more embodiments of the present application are set forth in the following drawings and description to make other features, objects, and advantages of the present application more readily apparent. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings:

[0040] Figure 1 A hardware structure block diagram of a computer device provided in an embodiment of the present application is shown;

[0041] Figure 2 A flow chart of a target relocation method provided by an embodiment of the present application is shown;

[0042] Figure 3 A schematic diagram of a prompt word generation model provided in an embodiment of the present application is shown;

[0043] Figure 4A schematic diagram of a target relocation method provided by an embodiment of the present application is shown;

[0044] Figure 5 A schematic diagram showing a fine-tuning process of a target model provided in an embodiment of the present application is shown;

[0045] Figure 6 A schematic diagram showing a reasoning process of a target model provided in an embodiment of the present application is shown;

[0046] Figure 7 A structural block diagram of a target relocation device provided in an embodiment of the present application is shown. DETAILED DESCRIPTION

[0047] In order to more clearly understand the purpose, technical solutions and advantages of the present application, the present application is described and illustrated below in conjunction with the accompanying drawings and embodiments.

[0048] Unless otherwise defined, the technical terms or scientific terms involved in this application shall have the general meaning understood by people with ordinary skills in the technical field to which this application belongs. The words "one", "a", "a", "the", "these" and the like in this application do not indicate quantitative restrictions, and they can be singular or plural. The terms "include", "comprise", "have" and any variants thereof involved in this application are intended to cover non-exclusive inclusions; for example, a process, method and apparatus, product or equipment comprising a series of steps or modules (units) is not limited to the listed steps or modules (units), but may include unlisted steps or modules (units), or may include other steps or modules (units) inherent to these processes, methods, products or equipment. The words "connect", "connected", "coupled" and the like involved in this application are not limited to physical or mechanical connections, but may include electrical connections, whether directly or indirectly. The "multiple" involved in this application refers to two or more. "And / or" describes the association relationship of associated objects, indicating that there can be three relationships. For example, "A and / or B" can mean: A exists alone, A and B exist at the same time, and B exists alone. Usually, the character " / " indicates that the objects associated with each other are in an "or" relationship. The terms "first", "second", "third", etc. involved in this application are only used to distinguish similar objects and do not represent a specific ordering of the objects.

[0049] The embodiment of the present application provides a target relocation method, which extracts sample prompt information of the target to be located from the first image sample set and the position information of the camera, and helps the detection model to detect the target to be located in the next video more accurately through the sample prompt information, reduces the interference caused by the environment and the crowd, and can achieve better detection accuracy in training. In addition, this training is actually a fine-tuning of the detection model, and the amount of data required will be much less, and the number of rounds required for training is small, so the training cost is low. Therefore, the embodiment of the present application solves the current technical problems of high model training cost, inaccurate target relocation in environmental changes and multi-target occlusion scenes.

[0050] The method can be executed in a terminal, a computer or a similar test system. For example, running on a computer device, Figure 1 FIG. 1 shows a hardware structure block diagram of a computer device provided in an embodiment of the present application. Figure 1 As shown, the computer device may include one or more ( Figure 1 The computer device may further include a transmission device 106 and an input / output device 108 for communication functions. It will be appreciated by those skilled in the art that Figure 1 The structure shown is only for illustration and does not limit the structure of the above-mentioned computer device. Figure 1 More or fewer components as shown, or with Figure 1 Different configurations are shown.

[0051] The memory 104 can be used to store computer programs, for example, software programs and modules of application software. The processor 102 executes various functional applications and data processing by running the computer programs stored in the memory 104, that is, realizing the above method. The memory 104 can be used to store data, such as image samples, videos to be detected, etc. The memory 104 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some instances, the memory 104 may further include a memory remotely arranged relative to the processor 102, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and a combination thereof.

[0052] The transmission device 106 is used to receive or send data via a network. The specific example of the above network may include a wireless network provided by a communication provider of the computer device. In one example, the transmission device 106 includes a network adapter (Network Interface Controller, referred to as NIC), which can be connected to other network devices through a base station so as to communicate with the Internet. In one example, the transmission device 106 can be a radio frequency (RF) module, which is used to communicate with the Internet wirelessly.

[0053] The embodiments of the present application can be applied to scenarios such as case investigation, autonomous driving, traffic monitoring, and human-computer interaction to relocate the target to be located extracted from one part of the video to another part of the video. Taking case investigation as an example, video data is collected from various camera points on the road, and the visual description (such as clothing color, body shape and appearance) and the movement trajectory (such as the movement route) of the key focus object are extracted from a part of the video data. The visual description and movement trajectory are used as prompt information to relocate the key focus object in another part of the video data, so that the key focus object can be found in other camera points. By repeating this process, the key focus object can be continuously tracked and found from new points to assist in case investigation.

[0054] The target to be located may be a target person, a vehicle, a traffic sign, a license plate, etc. The target to be located is relocated in the video to be detected by training the target detection model, and the category and position of the target to be located are marked in the video to be detected, wherein the position of the target to be located can be marked by detecting a frame, and the target to be located is located within the detection frame. The target relocation method provided in the embodiment of the present application is mainly described below by taking the target to be located as a target person as an example.

[0055] Figure 2 A flow chart of a target relocation method provided by an embodiment of the present application is shown. Figure 2 As shown, the method includes steps S201 to S204.

[0056] Step S201, obtain a plurality of sets of pre-annotated image sample sets collected by at least two cameras, and the position information of each camera, wherein the annotation includes annotating the position of the target to be located in the image sample set, and the image sample set includes a first image sample set and a second image sample set.

[0057] The image sample set includes a plurality of image samples, and the annotation may be annotated in the image samples by software, and the annotation content includes annotation of the category and the first position, and the first position is represented by coordinates.

[0058] The category refers to the type of object, usually a label such as "target person" or "person". The coordinates are the position of the target person in the image sample, which is usually represented by a bounding box, which contains the pixel coordinates of the upper left corner and lower right corner of the rectangular box where the target person is located. These two pixel coordinates determine the exact position and size of the target person in the image sample.

[0059] In this step, the image sample set is divided into at least two parts, one part is the first image sample set, and the other part is the second image sample set, wherein the first image sample set is used to pre-train the detection model so that it has the ability to extract the characteristic information of the target person, and the first image sample set is also used to determine the sample prompt information. The difference between the first image sample set and the second image sample set can be that the camera captures different areas or the camera captures different viewing angles. For example, some samples in the first image sample set are captured by camera A at the entrance of the mall with a first viewing angle, and other samples are captured by camera B at the entrance of the mall with a second viewing angle. Camera A and camera B have different monitoring angles for the entrance of the mall. Some samples in the second image sample set are captured by camera C at a traffic light intersection with a third viewing angle, and other samples are captured by camera D in a supermarket with a fourth viewing angle. Camera C and camera D have different monitoring areas and different monitoring angles. The model trained by the first image sample set and the second image sample set has the ability to track targets with different viewing angles, different camera positions, and different monitoring areas. In this embodiment, the sample prompt information is text information, such as a prompt word or prompt text, and the sample prompt information may also be in other forms such as voice and instructions. The second image sample set is used to fine-tune the pre-trained detection model to achieve better recognition effect in complex environments and multi-target environments.

[0060] In some embodiments, pre-training includes: taking a first set of image samples as input, using an annotated bounding box that annotates the target to be located in the first image sample as a label, and training the detection model with the goal of minimizing the loss between the detection bounding box output by the detection model and the annotated bounding box. The pre-trained detection model has the ability to extract feature information of the target person, so when a set of image samples is input into the detection model, the target person in the set of image samples can be identified by the detection model, and the target person is marked in the image samples in the form of a detection bounding box.

[0061] The image sample may be a set of continuous images in a video, or may be obtained by extracting frames from a video. Specifically, obtaining a plurality of sets of image samples collected by at least two cameras includes: obtaining a plurality of video samples collected by at least two cameras; extracting frames from the plurality of video samples to obtain a set of image samples corresponding to the video samples collected by each camera. When the frame extraction process is used, when adjacent frames in a video contain highly similar information, redundant video frames are reduced, thereby reducing the amount of calculation.

[0062] In some embodiments, the number of samples in the second image sample set is less than the number of samples in the first image sample set. It can be understood that the second image sample set fine-tunes the detection model under the guidance of the sample prompt information, so the data volume of the second image sample set is small, and the number of training rounds required for the fine-tuning model is small, and the training cost is low.

[0063] Step S202: extracting sample prompt information of the target to be located from the first image sample set and the position information of the camera.

[0064] In this step, the prompt word generation model can be trained to extract sample prompt information. Specifically, Figure 3 A schematic diagram of a prompt word generation model provided in an embodiment of the present application is shown. Figure 3 As shown, the prompt word generation model includes two modules, namely the visual text representation module and the prompt word output module. In the visual text representation module, multiple video frames (equivalent to the first image sample set) are input into the image encoder for feature extraction, the camera position information is input into the text encoder for feature extraction, and the features output by the two encoders are input into the Q-Former for multimodal feature fusion. In the prompt word output module, the large language model performs semantic transformation on the input multimodal features and outputs prompt words describing the target person.

[0065] Among them, the Large Language Model (LLM) is an artificial intelligence model that understands and generates human language by learning a large amount of text data. This model is usually based on the Transformer structure. By pre-training on a large amount of text data, it learns the basic patterns and structures of the language, thereby understanding the grammar, semantics and contextual relationships of the language. After pre-training, the large language model can be fine-tuned to adapt to specific tasks or applications, such as text generation, text classification, sentiment analysis, etc. As an example, the large language model can be a language model based on VisualGLM. VisualGLM-6B is an open source, multimodal dialogue language model that supports images, Chinese and English. The image part builds a bridge between the visual model and the language model by training BLIP2-Qformer. BLIP-2 (BootstrappingLanguage-Image Pre-training) introduces a new visual language pre-training paradigm to arbitrarily combine and fully utilize two pre-trained visual encoders and LLMs without end-to-end pre-training of the entire architecture, thereby achieving accurate results on multiple visual language tasks while significantly reducing the amount of training parameters and pre-training costs.

[0066] In some of the embodiments, step S202 includes: performing semantic extraction on the first image sample set and the position information of the camera respectively, and correspondingly obtaining visual semantic information and position semantic information of the target to be located; and using the visual semantic information and position semantic information as sample prompt information.

[0067] In this embodiment, the sample prompt information includes two parts, one of which is the visual semantic information extracted from the first image sample set. It can be understood that the visual semantic information refers to the characteristics of the target person that can be seen from the image, such as the target person's gender, whether he wears glasses, clothing, age, etc. The second is the location semantic information extracted in combination with the camera's location information and the first image sample set. The location semantic information is used to describe the point of the camera that captured the target to be located. The following is an explanation of how to extract location semantic information: Camera A collects image sample set a, camera B collects image sample set b, and camera C collects image sample set c. The target person is detected in image sample set a and image sample set b, respectively. Combined with the location information 1 of camera A corresponding to image sample set a, and the location information 3 of camera C corresponding to image sample set c, the location semantic information that can be extracted is: the target person has appeared in location information 1 and location information 3.

[0068] In some embodiments, the position semantic information is obtained after feature extraction of the first image sample set and semantic conversion of the visual features, and the position semantic information is obtained after semantic conversion of the position information of the point where the target person is detected. Specifically, the semantic extraction of the first image sample set and the position information of the camera is performed respectively, and the corresponding visual semantic information and position semantic information of the target to be located include: feature extraction of the target to be located in the first image sample set to obtain the visual features corresponding to the target to be located, and semantic conversion of the visual features to obtain the visual semantic information of the target to be located; target detection is performed on the target to be located in the first image sample set based on the visual features to obtain the detection bounding box corresponding to the target to be located; semantic conversion of the position information of the camera corresponding to the image sample containing the detection bounding box in the first image sample set is performed to obtain the position semantic information of the target to be located.

[0069] In this embodiment, the visual features of the target person are extracted by a feature extraction network. Prior to this, the feature extraction network can be pre-trained by using samples with labeled visual features so that the network has the ability to extract visual features. After the visual features are extracted, they are converted into visual semantic information through a large language model, such as: the target is wearing a red top, black pants, and sunglasses. Then the location information of all cameras is input into the large language model, and the large language model can extract location semantic information from the location information of the camera that detects the target person, such as: the target person passes by a supermarket, Primary School A. The prompt information in the two dimensions of vision and location can describe the target person more comprehensively, provide more accurate information about the target person, and help train the detection model so that it can obtain more accurate target relocation capabilities without being easily disturbed by the environment and multiple target occlusions.

[0070] In one embodiment, target detection is performed on the target to be located in the first image sample set based on visual features to obtain a detection bounding box corresponding to the target to be located, including: bounding box prediction is performed on the target to be located based on visual features to obtain bounding box information of multiple predicted bounding boxes corresponding to the target to be located, and a detection bounding box is screened out from the multiple predicted bounding boxes.

[0071] In some of the embodiments, the movement trajectory of the target person can be determined from multiple camera points in combination with the time when the sample image is collected. Specifically, the semantic conversion of the position information of the camera corresponding to the image sample containing the detection frame in the first image sample set to obtain the position semantic information of the target to be located includes: obtaining the position information of the camera corresponding to the image sample containing the detection frame in the first image sample set; sorting the position information of the camera corresponding to the image sample containing the detection frame according to the time sequence in which the image sample containing the detection frame is collected by the camera; generating the movement trajectory of the target to be located based on the sorted camera position information, wherein the position semantic information includes the movement trajectory of the target to be located.

[0072] In this embodiment, the location semantic information includes the motion trajectory, so it can describe the order of the target person's movement at various points. For example, the location semantic information is: the target person first appears in a supermarket, passes by Primary School A, and last appears in a commercial district. The motion trajectory can provide a more comprehensive description of the target person, so the fine-tuning effect of the detection model will be better.

[0073] In other embodiments, the location semantic information includes motion trajectory and appearance time point. Specifically, the appearance time point of the target person can be determined in combination with the shooting timestamp of the image sample where the target person is located, and this can be used as part of the location semantic information to optimize the detection model. For example, the location semantic information is: the target person appeared in a supermarket at 16:32 on November 26, 2024, passed Primary School A at 16:38 on November 26, 2024, and last appeared in a business district at 16:57 on November 26, 2024. Combined with the motion trajectory after the timestamp, the location of the target person is described more comprehensively, so richer prompt features can be provided for the detection model.

[0074] Step 203: input the sample prompt information and the second image sample set into the pre-trained detection model, so that the detection model performs border detection on the second image sample set under the prompt of the sample prompt information to obtain a detection border.

[0075] In traditional target relocation technology, the target person may be located in the previous video, but missed in the next video, resulting in the problem of losing the target person. This step inputs the sample prompt information and the second image sample set into the detection model, which is equivalent to inputting the descriptive features of the target person in the previous video together with the next video into the model for identification, so that the model can identify the target person based on the guidance of the descriptive features, and mark the target person in the image sample in the form of a detection border. It can be understood that the detection border is the target person identified by the pre-trained detection model under the original model parameters based on the sample prompt information and the second image sample set. The detection model is further trained by minimizing the loss function, thereby updating the model parameters of the detection model so that the model parameters have better relocation capabilities. For details about the training process, please refer to the description of step S204.

[0076] Step S204, calculate the border loss, and update the weight parameters of the detection model according to the border loss to obtain the target detection model, the target detection model is used to perform border detection on the target to be located in the first video to be detected, obtain the target prediction border and target prompt information, and relocate the target to be located in the second video to be detected in combination with the target prompt information.

[0077] The border loss refers to the gap between the detected border and the annotated border. The loss function of the detection model is constructed based on the border loss. The detection model is further optimized (or fine-tuned) with the goal of minimizing the loss function to obtain the target detection model. Compared with the detection model, the target detection model learns the relationship between the sample prompt information and the second video sample set. Therefore, it has a higher accuracy in target detection in complex scenes, and is not easy to lose the target person, that is, it has a stronger target relocation capability.

[0078] In the embodiment described in the above steps S201 to S204, a plurality of sets of pre-annotated image sample sets collected by at least two cameras and the position information of each camera are first obtained, wherein the annotation includes annotating the position of the target to be located in the image sample set, and the image sample set includes a first image sample set and a second image sample set; then, sample prompt information of the target to be located is extracted from the first image sample set and the position information of the camera; then, the sample prompt information and the second image sample set are input into a pre-trained detection model, so that the detection model performs border detection on the second image sample set under the prompt of the sample prompt information to obtain a detection border; the border loss is calculated, and the weight parameters of the detection model are updated according to the border loss to obtain a target detection model, and the target detection model is used to perform border detection on the target to be located in the first video to be detected, obtain a target predicted border and target prompt information, and relocate the target to be located in the second video to be detected in combination with the target prompt information. The present application fine-tunes the detection model through sample prompt information, so that the obtained target detection model has the ability to better detect the target to be located in the second image sample set in combination with the sample prompt information. In this way, the model's ability to recognize targets in scenarios with environmental changes and multi-target occlusion is improved. In addition, the number of image samples required for fine-tuning is small and the number of training rounds is small, so the training cost is low.

[0079] The following continues to use case investigation as an example to illustrate the method provided in the embodiment of the present application. The method is applied to a server. Figure 4 A schematic diagram of a target relocation method provided by an embodiment of the present application is shown, Figure 4 As shown, in this method, video data is first collected, and the server extracts frames from the video data, a part of which is used as annotation data for training the model, and a part of which is used as test data for testing the model. The server deploys the target model in the GPU (Graphics Processing Unit), uses the annotation data and test data to train and optimize the target model so that it has the ability to relocate the target, and outputs the prediction result of the target position on the image. The target model first needs to be pre-trained with a large number of pictures to obtain feature representation capabilities, and is trained through target detection tasks to obtain the ability to mark and detect borders, and then a small amount of fine-tuning samples are generated through a prompt word generation module to generate prompt words for prompt word fine-tuning. The fine-tuned target model can be relocated to the target person in the video frames captured by other camera devices.

[0080] Figure 5 A schematic diagram of a fine-tuning process of a target model provided in an embodiment of the present application is shown, Figure 5As shown in the figure, the process of fine-tuning the target model through sample prompt information includes: inputting video frames and prompt words into the backbone network, the backbone network performs feature extraction, and the detection head is responsible for category prediction and detection regression to obtain categories and detection borders, calculate the loss, the loss includes: border loss and category loss, and update the weight parameters according to the border loss and category loss, so as to achieve fine-tuning of the target model.

[0081] After obtaining the target model, reasoning can be achieved through the target model. Figure 6 A schematic diagram of the reasoning process of a target model provided in an embodiment of the present application is shown, such as Figure 6 As shown, the video frame and the prompt word of the previous video are input into the backbone network, which extracts features and outputs the prediction results through the detection head.

[0082] Among them, the backbone network can use an image encoder, and the detection head part is composed of an MLP (multi-layer perceptron). The features output by the backbone network are sorted into multiple prediction bounding boxes after passing through the MLP, and then the prediction bounding box with the highest confidence is selected as the final output of the detection bounding box.

[0083] Figure 7 A structural block diagram of a target relocation device provided in an embodiment of the present application is shown. Figure 7 As shown, the device comprises:

[0084] A data acquisition module 71 is used to acquire a plurality of sets of image samples collected by at least two cameras and marked in advance, and the position information of each camera, wherein the marking includes marking the position of the target to be located in the image sample set, and the image sample set includes a first image sample set and a second image sample set;

[0085] An information extraction module 72, used to extract sample prompt information of the target to be located from the first image sample set and the position information of the camera;

[0086] The model training module 73 is used to input the sample prompt information and the second image sample set into the pre-trained detection model, so that the detection model performs border detection on the second image sample set under the prompt of the sample prompt information to obtain a detection border; calculates the border loss, and updates the weight parameters of the detection model according to the border loss to obtain a target detection model. The target detection model is used to perform border detection on the target to be located in the first video to be detected, obtain a target predicted border and target prompt information, and relocate the target to be located in the second video to be detected in combination with the target prompt information.

[0087] In some embodiments, the data acquisition module 71 is also used to obtain multiple video samples collected by at least two cameras; perform frame extraction on the multiple video samples to obtain an image sample set corresponding to the video samples collected by each camera; wherein the number of samples in the second image sample set is less than the number of samples in the first image sample set.

[0088] In some of the embodiments, the information extraction module 72 is also used to perform semantic extraction on the first image sample set and the position information of the camera, respectively, to obtain the visual semantic information and position semantic information of the target to be located; and use the visual semantic information and position semantic information as sample prompt information.

[0089] In some of the embodiments, the information extraction module 72 is also used to extract features of the target to be located in the first image sample set, obtain visual features corresponding to the target to be located, and perform semantic conversion on the visual features to obtain visual semantic information of the target to be located; perform target detection on the target to be located in the first image sample set based on the visual features to obtain a detection bounding box corresponding to the target to be located; perform semantic conversion on the position information of the camera corresponding to the image sample containing the detection bounding box in the first image sample set to obtain the position semantic information of the target to be located.

[0090] In some of the embodiments, the information extraction module 72 is also used to obtain the position information of the camera corresponding to the image samples containing the detection border in the first image sample set; sort the position information of the camera corresponding to the image samples containing the detection border according to the time sequence in which the image samples containing the detection border are collected by the camera; and generate the motion trajectory of the target to be located based on the sorted camera position information, wherein the position semantic information includes the motion trajectory of the target to be located.

[0091] In some of the embodiments, the model training module 73 is also used to predict the border of the target to be located based on the visual features, obtain the border information of multiple predicted borders corresponding to the target to be located, and filter out the detection border from the multiple predicted borders.

[0092] In some of the embodiments, the model training module 73 is further used to take the first image sample set as input, and take the annotated bounding box that annotates the target to be located in the first image sample as a label, to train the detection model with the goal of minimizing the loss between the detection bounding box output by the detection model and the annotated bounding box.

[0093] It should be noted that the above modules can be functional modules or program modules, and can be implemented by software or hardware. For modules implemented by hardware, the above modules can be located in the same processor; or the above modules can be located in different processors in any combination.

[0094] In this embodiment, a computer device is further provided, including a memory and a processor, wherein a computer program is stored in the memory, and the processor is configured to run the computer program to execute the steps in any one of the above method embodiments.

[0095] Optionally, the computer device may further include a transmission device and an input / output device, wherein the transmission device is connected to the processor, and the input / output device is connected to the processor.

[0096] Optionally, in this embodiment, the processor may be configured to execute the following steps S1 to S4 through a computer program.

[0097] S1. Obtain a plurality of sets of pre-annotated image sample sets collected by at least two cameras, and location information of each camera, wherein the annotation includes annotating the position of a target to be located in the image sample set, and the image sample set includes a first image sample set and a second image sample set.

[0098] S2. Extracting sample prompt information of the target to be located from the first image sample set and the position information of the camera.

[0099] S3. Input the sample prompt information and the second image sample set into a pre-trained detection model, so that the detection model performs border detection on the second image sample set under the prompt of the sample prompt information to obtain a detection border.

[0100] S4. Calculate the border loss, and update the weight parameters of the detection model according to the border loss to obtain a target detection model. The target detection model is used to perform border detection on the target to be located in the first video to be detected, obtain the target prediction border and target prompt information, and relocate the target to be located in the second video to be detected in combination with the target prompt information.

[0101] It should be noted that the specific examples in this embodiment can refer to the examples described in the above embodiments and optional implementation modes, and will not be repeated in this embodiment.

[0102] In addition, in combination with the method provided in the above embodiments, a storage medium may be provided in this embodiment to implement the method. The storage medium stores a computer program; when the computer program is executed by a processor, any one of the target relocation methods in the above embodiments is implemented.

[0103] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer executes each function or step executed by the processor in the above method embodiment.

[0104] It should be understood that the specific embodiments described herein are only used to explain the application, rather than to limit it. Based on the embodiments provided in this application, all other embodiments obtained by ordinary technicians in this field without creative work are within the protection scope of this application.

[0105] Obviously, the drawings are only some examples or embodiments of the present application. For ordinary technicians in the field, the present application can also be applied to other similar situations based on these drawings without creative work. In addition, it is understandable that although the work done in this development process may be complicated and lengthy, for ordinary technicians in the field, certain changes in design, manufacturing or production based on the technical content disclosed in this application are only conventional technical means and should not be regarded as insufficient content disclosed in this application.

[0106] The term "embodiment" in this application refers to a specific feature, structure or characteristic described in conjunction with the embodiment that can be included in at least one embodiment of the present application. The appearance of this phrase in various locations in the specification does not necessarily mean the same embodiment, nor does it mean that it is mutually exclusive with other embodiments and is independent or optional. It is clearly or implicitly understood by those of ordinary skill in the art that the embodiments described in this application can be combined with other embodiments without conflict.

[0107] The above-mentioned embodiments only express several implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of patent protection. It should be pointed out that, for a person of ordinary skill in the art, several variations and improvements can be made without departing from the concept of the present application, and these all belong to the scope of protection of the present application. Therefore, the scope of protection of the present application shall be subject to the attached claims.

Claims

1. A target relocation method, characterized in that: The method comprises: Acquire a plurality of sets of pre-annotated image sample sets collected by at least two cameras, and position information of each of the cameras, wherein the annotation includes annotating the position of the target to be located in the image sample set, and the image sample set includes a first image sample set and a second image sample set; Extracting sample prompt information of the target to be located from the first image sample set and the position information of the camera; Inputting the sample prompt information and the second image sample set into a pre-trained detection model, so that the detection model performs border detection on the second image sample set under the prompt of the sample prompt information to obtain a detection border; The border loss is calculated, and the weight parameters of the detection model are updated according to the border loss to obtain a target detection model. The target detection model is used to perform border detection on the target to be located in the first video to be detected, obtain a target predicted border and target prompt information, and relocate the target to be located in the second video to be detected in combination with the target prompt information.

2. The target relocation method according to claim 1, characterized in that: The extracting the sample prompt information of the target to be located from the first image sample set and the position information of the camera includes: Semantically extracting the first image sample set and the position information of the camera respectively, and correspondingly obtaining visual semantic information and position semantic information of the target to be located; The visual semantic information and the position semantic information are used as the sample prompt information.

3. The target relocation method according to claim 2, characterized in that: The semantic extraction is performed on the first image sample set and the position information of the camera respectively, and the visual semantic information and the position semantic information of the target to be located are obtained accordingly, including: Extracting features of the target to be located in the first image sample set to obtain visual features corresponding to the target to be located, and performing semantic conversion on the visual features to obtain visual semantic information of the target to be located; Based on the visual features, target detection is performed on the target to be located in the first image sample set to obtain a detection bounding box corresponding to the target to be located; and the position information of the camera corresponding to the image sample containing the detection bounding box in the first image sample set is semantically converted to obtain the position semantic information of the target to be located.

4. The target relocation method according to claim 3, characterized in that: The step of performing semantic conversion on the position information of the camera corresponding to the image sample containing the detection frame in the first image sample set to obtain the position semantic information of the target to be located includes: Obtaining position information of the camera corresponding to the image sample containing the detection border in the first image sample set; Sort the position information of the camera corresponding to the image samples including the detection frame according to the time sequence in which the image samples including the detection frame are collected by the camera; The motion trajectory of the target to be located is generated based on the sorted position information of the cameras, wherein the position semantic information includes the motion trajectory of the target to be located.

5. The target relocation method according to claim 3, characterized in that: The performing target detection on the target to be located in the first image sample set based on the visual feature to obtain a detection bounding box corresponding to the target to be located includes: Based on the visual features, a border prediction is performed on the target to be located, border information of multiple predicted borders corresponding to the target to be located is obtained, and a detection border is screened out from the multiple predicted borders.

6. The target relocation method according to any one of claims 1 to 5, characterized in that: The step of obtaining a plurality of sets of image samples collected by at least two cameras includes: Acquire a plurality of video samples collected by at least two cameras; Performing frame extraction processing on the multiple video samples to obtain an image sample set corresponding to the video samples collected by each camera; The number of samples in the second image sample set is smaller than the number of samples in the first image sample set.

7. The target relocation method according to claim 1, characterized in that: The pre-training includes: The first image sample set is taken as input, and the annotated bounding box of the target to be located in the first image sample is taken as a label. The detection model is trained with the goal of minimizing the loss between the detection bounding box output by the detection model and the annotated bounding box.

8. A target relocation device, characterized in that: include: A data acquisition module, used to acquire a plurality of sets of image samples collected by at least two cameras and labeled in advance, and the position information of each of the cameras, wherein the labeling includes labeling the position of the target to be located in the image sample set, and the image sample set includes a first image sample set and a second image sample set; An information extraction module, used to extract sample prompt information of the target to be located from the first image sample set and the position information of the camera; A model training module is used to input the sample prompt information and the second image sample set into a pre-trained detection model, so that the detection model performs border detection on the second image sample set under the prompt of the sample prompt information to obtain a detection border; calculates the border loss, and updates the weight parameters of the detection model according to the border loss to obtain a target detection model, wherein the target detection model is used to perform border detection on the target to be located in the first video to be detected, obtain a target predicted border and target prompt information, and relocate the target to be located in the second video to be detected in combination with the target prompt information.

9. A computer device comprising a memory and a processor, characterized in that: A computer program is stored in the memory, and the processor is configured to run the computer program to execute the target relocation method according to any one of claims 1 to 7.

10. A storage medium, characterized in that: The storage medium stores a computer program, wherein the computer program is configured to execute the target relocation method according to any one of claims 1 to 7 when running.

Images