Target perception method and device based on content generation

Abstract

The invention discloses a target perception method and device based on content generation, and the method comprises the steps: obtaining a to-be-detected image which is called as a first corresponding image, and randomly generating a random Gaussian noise image which has the same resolution with the first corresponding image, and is called as a second corresponding image; the target perception model is trained based on the self-supervised training set through the target perception model, so that the target perception model has potential knowledge, targets with different target categories in the training data set can be perceived, and the perception ability of a complex environment and the adaptive ability of cross-domain recognition are effectively improved; meanwhile, in the training process, the quantity of learnable parameters of the target perception model is small, original model parameters are kept unchanged, the feature extraction capacity of an original model is exerted, meanwhile, the capacity of a content generation task can be improved, it can be guaranteed that the content of a generated perception result is kept consistent with the content of an original image, and target perception of a complex environment is effectively improved.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of intelligent perception, and particularly relates to a target perception method and device based on content generation. BACKGROUND

[0002] Target perception refers to automatically analyzing video, image, voice and other data collected by sensors to extract useful information, such as recognizing people, vehicles, objects and other targets in video images, aiming to comprehensively and accurately analyze target and situation information in the environment to provide support for downstream tasks.

[0003] Existing target perception methods are mostly based on convolutional neural network methods, which are trained through pre-labeled images and target position information, so as to have certain target detection capability. These methods have obvious limitations in processing capability, model generalization capability and intelligent processing means. They often rely on data in specific fields, are difficult to adapt to different scenes, and cannot predict different categories of targets. SUMMARY

[0004] The purpose of the application is to solve the problems raised in the background art, and a target perception method and device based on content generation are proposed.

[0005] To achieve the above purpose, the technical solution adopted by the application is as follows: The target perception method based on content generation proposed by the application comprises: An image to be detected, referred to as a first corresponding image, is obtained, and a random Gaussian noise image with the same resolution as the first corresponding image is randomly generated and referred to as a second corresponding image; The first corresponding image, the second corresponding image and a preset image template are input into a trained target perception model and sampled for a preset number of times to obtain a target perception result image; The target perception model comprises an image feature extraction module, a text feature extraction module, a perception content generation module and an incremental feature extraction module. The second corresponding image and the image template are input into the image feature extraction module to obtain first image features, and the preset task instruction is input into the text feature extraction module to obtain first text features. The first image features and the first text features are fused to obtain first fusion features. The first fusion features and the first corresponding image are input into the perception content generation module and the incremental feature extraction module respectively, and the perception content generation module and the incremental feature extraction module are cross-fused to obtain the output of the target perception model. Subtract the second corresponding image from the output of the target perception model, and re-input the result of the subtraction into the perception content generation module, and re-input the first corresponding image into the incremental feature extraction module and perform forward inference as a sampling process. After repeating the process for a preset number of times, the output of the target perception model obtained by the forward inference before the last sampling is taken as the target perception result image.

[0006] Preferably, the second corresponding image and the preset image template pass through the image feature extraction module to obtain the first image feature, including: The image feature extraction module includes a first preprocessor, an image feature encoder, and a feature alignment module connected in sequence from the data input to the output direction; The second corresponding image and the preset image template pass through the first preprocessor to obtain the first input image and the second input image after performing the preset resolution scaling and normalization operation, and the first input image and the second input image are spliced to obtain a spliced image, wherein the preset image template is an image with all pixel values being zero. The spliced image passes through the image feature encoder and the feature alignment module in sequence to obtain the first image feature, wherein the image feature encoder is a DINO visual large model, and the feature alignment module is a fully connected neural network.

[0007] Preferably, the preset task instruction passes through the text feature extraction module to obtain the first text feature, including: The text feature extraction module includes a second preprocessor and a text feature encoder connected in sequence from the data input to the output direction; The preset task instruction is a text prompt task instruction, which passes through the second preprocessor for text enhancement and then passes through the text feature encoder to obtain the first text feature, wherein the text feature encoder is a BERT model.

[0008] Preferably, the perception content generation module includes M shallow network layers and N deep network layers connected in sequence from the data input to the output direction, each shallow network layer includes a self-attention mechanism layer and a first feedforward neural network connected in sequence from the data input to the output direction, each deep network layer includes a layer normalization layer, a multi-head self-attention mechanism layer, and a second feedforward neural network connected in sequence from the data input to the output direction, and the input of each deep network layer and the output of the second feedforward neural network are added as the output of the deep network layer; The incremental feature extraction module includes M low-rank feature extraction structures and N-1 consistency control structures connected in sequence from the data input to the output direction, and the M low-rank feature extraction structures correspond to the M shallow network layers one by one, and the N-1 consistency control structures correspond to the first N-1 deep network layers one by one. Each of the low-rank feature extraction structures includes two fully connected layers connected in sequence, and each of the consistency control structures includes a first convolutional layer, a feature layer, and a second convolutional layer connected in sequence from data input to output, and the structure of the feature layer is consistent with the structure of the deep network layer.

[0009] Preferably, in each corresponding shallow network layer and low-rank feature extraction structure, the output of the self-attention mechanism layer is used as the input of the low-rank feature extraction structure, and the output of the low-rank feature extraction structure is added to the output of the shallow network layer as the input of the next shallow network layer, and the output of the Mth low-rank feature extraction structure is added to the output of the Mth shallow network layer as the input of the first deep network layer. In the first N-2 corresponding deep network layers and consistency control structures, the output of the deep network layer is fused with the output of the consistency control structure to obtain a second fused feature. The second fused feature is used as the input of the next deep network layer, and the second fused feature is fused with the output of the consistency control structure to obtain a third fused feature. The third fused feature is used as the input of the next consistency control structure, and the first corresponding image is used as the input of the first consistency control structure. In the (N-1)th corresponding deep network layer and consistency control structure, the output of the (N-1)th consistency control structure is fused with the output of the (N-1)th deep network layer as the input of the Nth deep network layer, and the output of the Nth deep network layer is used as the output of the target perception model.

[0010] Preferably, the first fused feature is used as the input to the first shallow network layer before sampling; In each sampling process, the result of subtracting the output of the target perception model from the second corresponding image is used as the input of the first shallow network layer.

[0011] Preferably, during the training of the target perception model, an original dataset is collected, and the original dataset includes target images, continuous video data of target images, and image pairs of target images in different scenes; Data for a continuous preset time period is extracted from continuous video data to obtain an image frame sequence. Two frames are extracted from the image frame sequence to obtain a first image frame and a second image frame. The two images in the image pair are respectively used as the first image frame and the second image frame, and the second image frame is used as the first corresponding image. The target is extracted from the first image frame, and the target is matched in the second image frame using the image matching method to obtain the position of the target in the second image frame. An outer rectangle is generated in the second image frame according to the position. The second image frame with the outer rectangle is used as the second corresponding image, and the target image is used as the image template. The target image, the first corresponding image, and the second corresponding image are used as samples in the training dataset, and the target perception model is trained based on the samples in the training dataset. During a training process, the second corresponding image and the image template are used as inputs to the image feature extraction module, and the first corresponding image is used as inputs to the first consistency control structure. After forward inference, the output of the target perception model is obtained, and the output of the target perception model is used to calculate the loss. Continuing to train until the loss converges, a well-trained target perception model is obtained.

[0012] Preferably, in each of the deep network layers, the output of the layer normalization layer is scaled and translated, and the calculation formula for the scaling and translation operation is as follows: ; in, The output of the normalization layer is represented. Indicates the scaling factor. Indicates the translation coefficient. The first feature is obtained by scaling and translating the output of the representation layer normalization layer, and this first feature is used as the input of the multi-head self-attention mechanism layer. The output of the multi-head self-attention mechanism layer is scaled to obtain a second feature, which is then used as the input to the second feedforward neural network. The scaling operation performed on the output of the multi-head self-attention mechanism layer is consistent with the scaling operation performed on the output of the layer normalization layer.

[0013] Preferably, the fusion operation is a weighted summation.

[0014] A content-generated target perception device includes a processor and a memory storing a plurality of computer instructions, which, when executed by the processor, implement the steps of a content-generated target perception method.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This content-generated target perception method and device uses a target perception model and trains the target perception model on a self-supervised training set. This allows the target perception model to possess latent knowledge that enables it to perceive targets of different categories in the training dataset, effectively improving the perception capability in complex environments and the adaptability of cross-domain recognition. Meanwhile, during training, the target perception model can learn a small number of parameters, keeping the original model parameters unchanged. While leveraging the feature extraction capabilities of the original model, it can improve the ability to complete the content generation task, and ensure that the generated perception results are consistent with the original image, effectively improving target perception in complex environments. Attached Figure Description

[0016] Figure 1 This is a flowchart illustrating the content-based target perception method and apparatus of the present invention. Figure 2 This is a schematic diagram of the target perception model of the present invention; Figure 3 This is a schematic diagram of the structure of the content generation module and the incremental feature extraction module of the present invention; Figure 4 This is a schematic diagram of the deep network layer structure of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0018] In one embodiment, such as Figures 1-4 As shown, a content-generation-based target awareness method is provided, including: Step 1: Obtain the image to be detected, called the first corresponding image, and randomly generate a random Gaussian noise image with the same resolution as the first corresponding image, called the second corresponding image; Step 2: Input the first corresponding image, the second corresponding image, and the preset image template into the trained target perception model and perform a preset number of samplings to obtain the target perception result image; Step 2.1: The target perception model includes an image feature extraction module, a text feature extraction module, a perception content generation module, and an incremental feature extraction module; Step 2.2: After the second corresponding image and the image template are processed by the image feature extraction module, the first image feature is obtained, specifically: The image feature extraction module includes a first preprocessor, an image feature encoder, and a feature alignment module connected sequentially from data input to output. The second corresponding image and the preset image template are respectively processed by the first preprocessor. After the preset resolution scaling and normalization operations are performed (the resolution is scaled first, and then normalized, such as scaling the resolution to 336x336 pixels and normalizing the pixel values ​​to the range of [-1,1]), the first input image and the second input image are obtained (the first input image corresponds to the second corresponding image, and the second input image corresponds to the preset image template). The first input image and the second input image are then stitched together to obtain a stitched image (stitching is performed along the channel dimension). The preset image template is an image with all pixel values ​​being zero (wherein, if the preset image template is an image with all pixel values ​​being zero, it means that a perception result will be generated for the target of the corresponding category in the image to be detected according to the task instruction). The stitched image is then sequentially processed by an image feature encoder (which does not participate in parameter updates during training) and a feature alignment module (whose parameters can be learned and updated) to obtain the first image feature. The image feature encoder is a DINO visual large model, and the feature alignment module is a fully connected neural network.

[0019] Step 2.3: After the preset task instruction passes through the text feature extraction module, the first text feature is obtained, specifically: The text feature extraction module includes a second preprocessor and a text feature encoder connected sequentially from data input to output. The preset task instruction is a text prompt (it should be noted that the task instruction can be "Please detect people (or cars, or trees, etc.) in the image to be detected"). After the second preprocessor performs text enhancement (generating a more standardized text description), it is then passed through a text feature encoder to obtain the first text features. The text feature encoder is a BERT model (or other models, such as the clip text encoder). The second preprocessor is an existing pre-trained large model, such as the Tongyi Qianwen model.

[0020] Step 2.4: Perform a fusion operation (weighted summation, where the weights in the weighted summation are set according to actual needs, as below) on the first image features and the first text features to obtain the first fused features.

[0021] Step 2.5: The first fusion feature and the first corresponding image are used as inputs to the perceptual content generation module and the incremental feature extraction module, respectively. The perceptual content generation module and the incremental feature extraction module are cross-fused to obtain the output of the target perceptual model. Step 2.5.1: The perceptual content generation module includes M shallow network layers and N deep network layers connected sequentially from data input to output (i.e., the M shallow network layers are connected sequentially from data input to output, and the N deep network layers are connected sequentially from data input to output, with the output of the Mth shallow network layer serving as the input of the first deep network layer; the first shallow network layer in the data input-to-output direction is called the first shallow network layer, and so on, with the first deep network layer being called the first deep network layer, where M / (M+N) is not less than 70%). The network layers consist of a self-attention mechanism layer (i.e., the self-attention mechanism layer in a transformer neural network) and a first feedforward neural network (FFN), connected sequentially from data input to output. Each deep network layer consists of a layer normalization layer, a multi-head self-attention mechanism layer, and a second feedforward neural network, connected sequentially from data input to output. The input of each deep network layer is added to the output of the second feedforward neural network to obtain the output of the deep network layer. Furthermore, within each deep network layer, the output of the layer normalization layer undergoes scaling and translation operations, calculated using the following formulas: ; in, The output of the layer normalization layer is indicated (the specific operation of the layer normalization layer is: the layer normalization layer normalizes the input parameters: ,in The input parameters for the layer normalization layer, and These are the mean and variance of the input parameters, respectively. Indicates the scaling factor. Indicates the translation coefficient. The first feature is obtained by scaling and translating the output of the normalized layer (the parameters after the normalized layer are scaled and translated to increase the diversity of data distribution), and this first feature is used as the input of the multi-head self-attention mechanism layer; the output of the multi-head self-attention mechanism layer is scaled to obtain the second feature, and this second feature is used as the input of the second feedforward neural network, and the scaling operation performed on the output of the multi-head self-attention mechanism layer is consistent with the scaling operation performed on the output of the normalized layer.

[0022] Step 2.5.2: The incremental feature extraction module includes M low-rank feature extraction structures and N-1 consistency control structures connected sequentially from data input to output. The M low-rank feature extraction structures correspond one-to-one with the M shallow network layers, and the N-1 consistency control structures correspond one-to-one with the first N-1 deep network layers. Each low-rank feature extraction structure includes two fully connected layers connected in sequence, and each consistency control structure includes a first convolutional layer, a feature layer, and a second convolutional layer connected in sequence from data input to output, and the structure of the feature layer is consistent with the structure of the deep network layer.

[0023] Step 2.5.3: In each corresponding shallow network layer and low-rank feature extraction structure (i.e., the first shallow network layer and the first low-rank feature extraction structure, ..., the Mth shallow network layer and the Mth low-rank feature extraction structure), the output of the self-attention mechanism layer is used as the input of the low-rank feature extraction structure, and the output of the low-rank feature extraction structure is added to the output of the shallow network layer as the input of the next shallow network layer, and the output of the Mth low-rank feature extraction structure is added to the output of the Mth shallow network layer as the input of the first deep network layer, and the first fusion feature is used as the input of the first shallow network layer; Step 2.5.4: In the first N-2 corresponding deep network layers and consistency control structures (i.e., the first deep network layer and the first consistency control structure, ..., the N-2th deep network layer and the N-2th consistency control structure), the output of the deep network layer is fused with the output of the consistency control structure (weighted summation) to obtain the second fused feature. The second fused feature is used as the input of the next deep network layer, and the second fused feature is fused with the output of the consistency control structure (weighted summation) to obtain the third fused feature. The third fused feature is used as the input of the next consistency control structure, and the first corresponding image is used as the input of the first consistency control structure. Step 2.5.5: In the (N-1)th corresponding deep network layer and consistency control structure, the output of the (N-1)th consistency control structure and the output of the (N-1)th deep network layer are fused as the input of the Nth deep network layer, and the output of the Nth deep network layer is used as the output of the target perception model.

[0024] Step 2.6: Subtract the output of the target perception model from the second corresponding image, and use the result of the subtraction as the input to the perception content generation module (i.e., use the result of the subtraction as the input to the first shallow network layer), and use the first corresponding image as the input to the incremental feature extraction module, and perform forward inference as one sampling process. Repeat this process a preset number of times (in the next sampling process, subtract the output of the target perception model from the second corresponding image again, use the result of the subtraction as the input to the first shallow network layer, and use the first corresponding image as the input to the incremental feature extraction module, and perform forward inference, repeating this process multiple times, such as no less than 30 times, the specific number can be set according to actual needs). Use the output of the target perception model obtained from the last sampling forward inference as the target perception result image. The target perception result image is the target in the image to be detected with bounding rectangles added, indicating that the target in the image to be detected can be detected. The specific target in the image to be detected depends on what target is to be detected in the task instruction. For example, if the task instruction is to detect people in the image to be detected, then the target perception result image is all targets in the image to be detected with bounding rectangles added.

[0025] During the training of the target-aware model, a self-supervised training dataset is constructed, including: The raw dataset is collected, which includes target images, continuous video data of the target images, and image pairs of the target images in different scenes (an image pair is two images). The target image contains only the target itself and no other background. The continuous video data and image pairs of the target images contain the target itself and other background. It should be noted that the continuous video data can make full use of existing public datasets in the industry, such as video datasets for single target tracking, which can greatly reduce the workload of data collection. The target images and image pairs of the target in different scenes are mainly collected from data from major e-commerce websites, including close-up images of products and images of products in use in application scenarios, which makes it easy to obtain image pairs.

[0026] Image processing is performed on the original dataset to obtain the training dataset (i.e., the self-supervised training dataset), which includes: Extract data from continuous video data for a continuous preset time period (e.g., extract 2 seconds of video data, which contains multiple frames of images) to obtain an image frame sequence. Extract two frames from the image frame sequence (the two extracted frames are different) to obtain a first image frame and a second image frame. The two images in the image pair are respectively used as the first image frame and the second image frame, and the second image frame is used as the first corresponding image. The target is cropped from the first image frame. Image matching methods (extracting features of the cropped target and then matching them in the second image frame using the extracted features, such as SIFT or ORB-based image matching methods) are used to match the target in the second image frame to obtain the target's position in the second image frame. An outer bounding box (with a width of at least 5 pixels) is generated in the second image frame based on the position. The second image frame with the outer bounding box is used as the second corresponding image, and the target image is used as the image template. Different colors are used to represent the bounding boxes for different target types. The target image, the first corresponding image, and the second corresponding image are used as samples in the training dataset, and the target perception model is trained based on the samples in the training dataset (where the training dataset contains no less than 1.05 million samples). During a training process, the second corresponding image and the image template are used as inputs to the image feature extraction module, and the first corresponding image is used as input to the first consistency control structure. After forward inference, the output of the target perception model is obtained (the specific forward inference process is the same as steps 2.1-2.5). The output of the target perception model is used to calculate the loss (and update the parameters of the target perception model, where); Continuing to train until the loss converges, a well-trained target perception model is obtained.

[0027] It should be noted that during training, the parameters of the target perception model are first initialized. The parameters of the feature alignment module and the low-rank feature extraction structure are initialized using Gaussian random parameters. The parameters of the perceptual content generation module are loaded with the parameters of a pre-defined training model (e.g., a Stable Diffusion model). The parameters of the first and second convolutional layers are initialized to zero. The parameters of the feature layers are consistent with those of the deep network layers. Furthermore, during the training of the target perception model, the parameters of the perceptual content generation module, the text feature encoder, and the image feature encoder are not updated. During training, the text feature encoder and image feature encoder can be used in advance to perform batch feature extraction on the task instructions and stitched images after the second preprocessor, obtaining and storing the first text features and first image features. During training, the stored first text features and first image features are directly read as input, thereby accelerating the training process.

[0028] In another embodiment, based on a content-generated target perception method, a content-generated target perception device is extracted, including a processor and a memory storing a number of computer instructions, wherein the computer instructions are executed by the processor to implement the steps of the content-generated target perception method.

[0029] For specific limitations on a content-generated target perception device, please refer to the limitations on a content-generated target perception method mentioned above, which will not be repeated here.

[0030] This content-generated target perception method and device uses a target perception model and trains the target perception model on a self-supervised training set. This allows the target perception model to possess latent knowledge that enables it to perceive targets of different categories in the training dataset, effectively improving the perception capability in complex environments and the adaptability of cross-domain recognition. Meanwhile, during training, the target perception model can learn a small number of parameters, keeping the original model parameters unchanged (i.e., the parameters do not need to be updated). While leveraging the feature extraction capabilities of the original model, it can improve the ability to complete the content generation task, and ensure that the generated perception results are consistent with the original image, effectively improving target perception in complex environments.

[0031] It should be understood that, although Figure 1 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 1 At least some of the steps in the process may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be executed in turn or alternately with other steps or at least some of the sub-steps or stages of other steps.

[0032] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.

Claims

1. A content-generation-based target awareness method, characterized in that: The content-generated target awareness method includes: The image to be detected is obtained and referred to as the first corresponding image. A random Gaussian noise image with the same resolution as the first corresponding image is randomly generated and referred to as the second corresponding image. The first corresponding image, the second corresponding image, and the preset image template are all input into the trained target perception model and sampled a preset number of times to obtain the target perception result image; The target perception model includes an image feature extraction module, a text feature extraction module, a perception content generation module, and an incremental feature extraction module. After the second corresponding image and image template are processed by the image feature extraction module, the first image feature is obtained. After the preset task instruction is processed by the text feature extraction module, the first text feature is obtained. The first image feature and the first text feature are fused to obtain the first fused feature. The first fused feature and the first corresponding image are used as inputs to the perceptual content generation module and the incremental feature extraction module, respectively, and the perceptual content generation module and the incremental feature extraction module are cross-fused to obtain the output of the target perceptual model; The second corresponding image is subtracted from the output of the target perception model, and the result of the subtraction is used as the input of the perception content generation module. The first corresponding image is used as the input of the incremental feature extraction module, and forward inference is performed as one sampling process. This process is repeated a preset number of times, and the output of the target perception model obtained by the forward inference of the last sampling is used as the target perception result image.

2. The content-generation-based target awareness method as described in claim 1, characterized in that: After the second corresponding image and the preset image template are processed by the image feature extraction module, the first image features are obtained, including: The image feature extraction module includes a first preprocessor, an image feature encoder, and a feature alignment module connected sequentially from data input to output. The second corresponding image and the preset image template are respectively processed by the first preprocessor, and after the preset resolution scaling and normalization operations are performed, the first input image and the second input image are obtained. The first input image and the second input image are then stitched together to obtain a stitched image, wherein the preset image template is an image with all pixel values ​​being zero. The stitched image is then sequentially processed by an image feature encoder and a feature alignment module to obtain the first image feature, wherein the image feature encoder is a DINO visual large model and the feature alignment module is a fully connected neural network.

3. The target awareness method based on content generation as described in claim 1, characterized in that: After the preset task instruction is processed by the text feature extraction module, the first text feature is obtained, including: The text feature extraction module includes a second preprocessor and a text feature encoder connected sequentially from data input to output. The preset task instruction is a text prompt task instruction, which is enhanced by a second preprocessor and then passed through a text feature encoder to obtain the first text feature, wherein the text feature encoder is a BERT model.

4. The target awareness method based on content generation as described in claim 1, characterized in that: The perception content generation module includes M shallow network layers and N deep network layers connected sequentially from data input to output. Each shallow network layer includes a self-attention mechanism layer and a first feedforward neural network connected sequentially from data input to output. Each deep network layer includes a layer normalization layer, a multi-head self-attention mechanism layer and a second feedforward neural network connected sequentially from data input to output. The input of each deep network layer is added to the output of the second feedforward neural network to obtain the output of the deep network layer. The incremental feature extraction module includes M low-rank feature extraction structures and N-1 consistency control structures connected sequentially from data input to output. The M low-rank feature extraction structures correspond one-to-one with the M shallow network layers, and the N-1 consistency control structures correspond one-to-one with the first N-1 deep network layers. Each of the low-rank feature extraction structures includes two fully connected layers connected in sequence, and each of the consistency control structures includes a first convolutional layer, a feature layer, and a second convolutional layer connected in sequence from data input to output, and the structure of the feature layer is consistent with the structure of the deep network layer.

5. The content-generation-based target awareness method as described in claim 4, characterized in that: In each corresponding shallow network layer and low-rank feature extraction structure, the output of the self-attention mechanism layer is used as the input of the low-rank feature extraction structure, and the output of the low-rank feature extraction structure is added to the output of the shallow network layer as the input of the next shallow network layer. The output of the Mth low-rank feature extraction structure is added to the output of the Mth shallow network layer as the input of the first deep network layer. In the first N-2 corresponding deep network layers and consistency control structures, the output of the deep network layer is fused with the output of the consistency control structure to obtain a second fused feature. The second fused feature is used as the input of the next deep network layer, and the second fused feature is fused with the output of the consistency control structure to obtain a third fused feature. The third fused feature is used as the input of the next consistency control structure, and the first corresponding image is used as the input of the first consistency control structure. In the (N-1)th corresponding deep network layer and consistency control structure, the output of the (N-1)th consistency control structure is fused with the output of the (N-1)th deep network layer as the input of the Nth deep network layer, and the output of the Nth deep network layer is used as the output of the target perception model.

6. The content-generation-based target awareness method as described in claim 5, characterized in that: Before sampling, the first fused feature is used as the input to the first shallow network layer; In each sampling process, the result of subtracting the output of the target perception model from the second corresponding image is used as the input of the first shallow network layer.

7. The content-generation-based target awareness method as described in claim 6, characterized in that: During the training of the target perception model, a raw dataset is collected, which includes target images, continuous video data of target images, and image pairs of target images in different scenes. Data for a continuous preset time period is extracted from continuous video data to obtain an image frame sequence. Two frames are extracted from the image frame sequence to obtain a first image frame and a second image frame. The two images in the image pair are respectively used as the first image frame and the second image frame, and the second image frame is used as the first corresponding image. The target is extracted from the first image frame, and the target is matched in the second image frame using the image matching method to obtain the position of the target in the second image frame. An outer rectangle is generated in the second image frame according to the position. The second image frame with the outer rectangle is used as the second corresponding image, and the target image is used as the image template. The target image, the first corresponding image, and the second corresponding image are used as samples in the training dataset, and the target perception model is trained based on the samples in the training dataset. During a training process, the second corresponding image and the image template are used as inputs to the image feature extraction module, and the first corresponding image is used as inputs to the first consistency control structure. After forward inference, the output of the target perception model is obtained, and the output of the target perception model is used to calculate the loss. Continuing to train until the loss converges, a well-trained target perception model is obtained.

8. The target awareness method based on content generation as described in claim 4, characterized in that: In each of the deep network layers, the output of the normalized layer is scaled and translated, and the calculation formula for the scaling and translation operation is as follows: ； in, The output of the normalization layer is represented. Indicates the scaling factor. Indicates the translation coefficient. The first feature is obtained by scaling and translating the output of the representation layer normalization layer, and this first feature is used as the input of the multi-head self-attention mechanism layer. The output of the multi-head self-attention mechanism layer is scaled to obtain a second feature, which is then used as the input to the second feedforward neural network. The scaling operation performed on the output of the multi-head self-attention mechanism layer is consistent with the scaling operation performed on the output of the layer normalization layer.

9. The content-generation-based target awareness method as described in claim 5, characterized in that: The fusion operation is a weighted summation.

10. A content-generated target perception device, comprising a processor and a memory storing a plurality of computer instructions, characterized in that: When the computer instructions are executed by the processor, they implement the steps of the method according to any one of claims 1 to 9.

Images