Model training method, model training device and computer equipment

Abstract

The application relates to the technical field of computer vision, and provides a model training method, a model training device and computer equipment. A digital twin scene corresponding to a to-be-trained space is constructed through a small amount of real space images, simulation space images corresponding to the to-be-trained space and labeling information are generated in batches by using the digital twin scene for model training, dependence on large-scale real labeling data is significantly reduced, and data acquisition and manual labeling costs are greatly compressed; meanwhile, through a target-based mask mechanism, only the features of the target are allowed to participate in domain confrontation, domain difference of background features is avoided from "occupying" optimization resources of the domain confrontation, learning of a target detection model is ensured to be "domain-invariant features of the target itself", rather than "domain-invariant features of the background", and therefore, under the premise of low-cost training of the target detection model, the detection accuracy of the target detection model on small targets is improved.

Description

Technical Field

[0001] This invention relates to the field of computer vision technology, and in particular to a model training method, a model training device, and a computer equipment. Background Technology

[0002] Object detection, which aims to identify all targets of interest in an image and determine their location and category, is one of the core foundational tasks in computer vision. With the rapid development of the low-altitude economy, the need for real-time perception of aircraft in complex airspace environments is becoming increasingly urgent, giving rise to the task of object detection for aerial targets. For example, aircraft can use their onboard visual sensors to acquire airspace images and combine them with locally deployed or cloud-deployed object detection models to identify and locate potential threat targets such as birds, drones, and balloons in real time, thus providing crucial information for obstacle avoidance decisions. Therefore, how to train a low-cost object detection model suitable for small aerial targets has become a key bottleneck restricting the improvement of aircraft environmental perception capabilities. Summary of the Invention

[0003] This invention provides a model training method, a model training device, and a computer equipment, which can achieve low-cost training of a target detection model for small aerial targets.

[0004] Firstly, the object detection model includes a feature extraction network and a detection head network. The model training method provided by this invention includes: Acquire real airspace images of the airspace to be trained, and based on the real airspace images, reconstruct the 3D scene using a 3D reconstruction model to generate real 3D scene point clouds corresponding to the airspace to be trained. Import real 3D scene point clouds into rendering and annotation tools to generate digital twin scenes of the spatial domain to be trained; Based on the digital twin scenario, simulated spatial domain images and annotation information corresponding to the spatial domain to be trained are obtained through rendering and annotation tools; Construct a discriminator network; fix the weight parameters of the feature extraction network and the detection head network, extract the first feature maps of the real spatial domain image and the simulated spatial domain image through the feature extraction network, and update the weight parameters of the discriminator network with the optimization objective of maximizing the discriminator network's ability to distinguish between the two first feature maps; fix the weight parameters of the discriminator network, extract the second feature map of the simulated spatial domain image through the feature extraction network, and obtain a mask matrix with the same size as the second feature map based on the annotation information; perform a masking operation on the second feature map based on the mask matrix to obtain the masked second feature map; update the weight parameters of the feature extraction network and the detection head network with the joint optimization objective of minimizing the discriminator network's ability to distinguish the masked second feature map and the detection loss of the detection head network on the second feature map; alternately perform weight updates for the discriminator network and joint weight updates for the feature extraction network and the detection head network until a preset stopping condition is met.

[0005] Secondly, the target detection model includes a feature extraction network and a detection head network. The model training device provided by this invention includes: The 3D reconstruction module is used to acquire real spatial images of the spatial domain to be trained, and based on the real spatial images, to reconstruct the 3D scene through the 3D reconstruction model, generating a real 3D scene point cloud corresponding to the spatial domain to be trained. The scene generation module is used to import real 3D scene point clouds into rendering and annotation tools to generate digital twin scenes of the spatial domain to be trained. The sample generation module is used to obtain simulated spatial domain images and annotation information corresponding to the spatial domain to be trained based on the digital twin scene through rendering and annotation tools; The model training module is used to construct a discriminator network. With fixed weight parameters for the feature extraction network and the detection head network, the feature extraction network extracts the first feature maps of both the real and simulated spatial domain images. The discriminator network's weight parameters are updated with the optimization objective of maximizing its ability to distinguish between these first feature maps. With the discriminator network's weight parameters fixed, the feature extraction network extracts the second feature map of the simulated spatial domain image. Based on the annotation information, a mask matrix of the same size as the second feature map is obtained. The second feature map is then masked using the mask matrix to obtain the masked second feature map. The weight parameters of the feature extraction network and the detection head network are updated with the joint optimization objective of minimizing the discriminator network's ability to distinguish the masked second feature map and the detection loss of the detection head network. The weight updates for the discriminator network and the joint weight updates for the feature extraction network and the detection head network are performed alternately until a preset stopping condition is met.

[0006] Optionally, in one embodiment, the sample generation module is used to identify dynamic targets in the digital twin scene through rendering and annotation tools, and obtain the kinematic model corresponding to the dynamic targets; based on the kinematic model, the dynamic targets are driven to perform simulated motion in the digital twin scene through rendering and annotation tools; during the simulated motion, simulated spatial images with image content differences and their annotation information are collected through rendering and annotation tools.

[0007] Optionally, in one embodiment, the model training module is further configured to, for each target indicated by the annotation information, if the target area in the second feature map is less than or equal to an area threshold, expand the mask region corresponding to the target in the mask matrix according to the target area.

[0008] Optionally, in one embodiment, the model training module is used to expand the mask region corresponding to the target in the mask matrix according to the expansion pixel threshold if the integer part of the square root of the ratio of the area threshold to the target area is less than or equal to the expansion pixel threshold; otherwise, it expands the mask region corresponding to the target in the mask matrix according to the pixel value represented by the integer part.

[0009] Optionally, in one embodiment, the model training module is further configured to perform Gaussian softening on the boundary of the mask region of each target indicated by the annotation information in the mask matrix.

[0010] Optionally, in one embodiment, the feature extraction network includes a general feature extraction sub-network and a domain adaptation sub-network. The general feature extraction sub-network is used to extract low-level features shared by different spatial domains, and the domain adaptation sub-network is used to extract high-level features adapted to different spatial domains. The model training module is used to fix the weight parameters of the general feature extraction sub-network and update the weight parameters of the domain adaptation sub-network and the detection head network with the joint optimization objective of minimizing the discriminator network's ability to discriminate the masked second feature map and the detection head network's detection loss on the second feature map.

[0011] Optionally, in one embodiment, the model training module is used to construct a first discriminator network corresponding to the general feature extraction sub-network and a second discriminator network corresponding to the domain adaptation sub-network; the general feature extraction sub-network extracts the first low-level features of the real spatial image and the simulated spatial image respectively, and the domain adaptation sub-network extracts the first high-level features of the real spatial image and the simulated spatial image respectively; the weight parameters of the first discriminator network and the second discriminator network are updated with the joint optimization objective of maximizing the discriminative ability of the first discriminator network to the first low-level features of the two images and the discriminative ability of the second discriminator network to the first high-level features of the two images.

[0012] Optionally, in one embodiment, the sample generation module is further configured to determine the image sensor of the detection device to be deployed by the target detection model, and obtain the imaging characteristic parameters of the image sensor; based on the imaging characteristic parameters, the simulated spatial image is enhanced by imaging characteristic matching to obtain an enhanced simulated spatial image; The model training module is used to perform domain adversarial training on the target detection model based on real spatial images, enhanced simulated spatial images, and annotation information.

[0013] Thirdly, the computer device provided by the present invention includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program in the memory to implement the model training method provided by the present invention.

[0014] The model training scheme provided by this invention acquires a real airspace image of the airspace to be trained, and based on the real airspace image, reconstructs a 3D scene using a 3D reconstruction model to generate a real 3D scene point cloud corresponding to the airspace to be trained; imports the real 3D scene point cloud into a rendering and annotation tool to generate a digital twin scene of the airspace to be trained; based on the digital twin scene, obtains a simulated airspace image and annotation information corresponding to the airspace to be trained using a rendering and annotation tool; and performs domain adversarial training on the target detection model based on the real airspace image, the simulated airspace image, and the annotation information until a preset stopping condition is met. Therefore, only a small number of real spatial images are needed to construct a digital twin scene corresponding to the spatial domain to be trained. This digital twin scene can then be used to generate simulated spatial images and annotation information corresponding to the spatial domain to be trained in batches for model training. This significantly reduces the dependence on large-scale real annotation data, thereby greatly reducing the cost of data collection and manual annotation. At the same time, through a target-based masking mechanism, only the features of the target are allowed to participate in domain adversarial training, avoiding the domain differences of background features from "stealing" the optimization resources of domain adversarial training. This ensures that the target detection model learns "the domain-invariant features of the target itself" rather than "the domain-invariant features of the background". Thus, while achieving low-cost training of the target detection model, the detection accuracy of the target detection model for small targets is improved. Attached Figure Description

[0015] To more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings used in the description of the embodiments of the present invention will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0016] Figure 1 This is a flowchart illustrating the model training method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the target detection model in an embodiment of the present invention; Figure 3 This is a detailed schematic diagram of the feature extraction network in the object detection model; Figure 4 This is a schematic diagram of the structure of the model training device provided in an embodiment of the present invention; Figure 5 This is a schematic diagram of the structure of a computer device provided in an embodiment of the present invention. Detailed Implementation

[0017] To make the technical problems solved, the technical solutions, and the beneficial effects of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative of the invention and are not intended to limit the invention.

[0018] It should be understood that, when used in this specification and the appended claims, the term "comprising" indicates the presence of the described features, integrals, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integrals, steps, operations, elements, components and / or collections thereof.

[0019] It should also be understood that the term “and / or” as used in this specification and the appended claims refers to any combination of one or more of the associated listed items and all possible combinations, and includes such combinations.

[0020] As used in this specification and the appended claims, the term "if" may be interpreted, depending on the context, as "when," "once," "in response to determination," or "in response to detection." Similarly, the phrase "if determined" or "if [described condition or event] is detected" may be interpreted, depending on the context, as meaning "once determined," "in response to determination," "once [described condition or event] is detected," or "in response to detection of [described condition or event]."

[0021] Furthermore, in the description of this invention and the appended claims, the terms "first," "second," "third," etc., are used only for distinguishing descriptions and should not be construed as indicating or implying relative importance.

[0022] References to "one embodiment" or "some embodiments" as described in this specification mean that one or more embodiments of the invention include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," "in still other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized. The terms "comprising," "including," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0023] This invention provides a model training method, a model training device, a computer device, and a computer-readable storage medium. The model training method includes: acquiring a real spatial image of the spatial domain to be trained; reconstructing a 3D scene based on the real spatial image using a 3D reconstruction model to generate a real 3D scene point cloud corresponding to the spatial domain to be trained; importing the real 3D scene point cloud into a rendering and annotation tool to generate a digital twin scene of the spatial domain to be trained; acquiring a simulated spatial image and annotation information corresponding to the spatial domain to be trained using the rendering and annotation tool based on the digital twin scene; and performing domain adversarial training on the target detection model based on the real spatial image, the simulated spatial image, and the annotation information until a preset stopping condition is met. This model training scheme can achieve low-cost training of the target detection model while ensuring its detection accuracy.

[0024] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the embodiments described below are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0025] Please refer to Figure 1 , Figure 1 This is a flowchart illustrating a model training method provided in an embodiment of the present invention, as shown below. Figure 1 As shown, the training process for this model can be described as follows: In S110, a real spatial image of the spatial domain to be trained is acquired, and based on the real spatial image, a 3D scene is reconstructed using a 3D reconstruction model to generate a real 3D scene point cloud corresponding to the spatial domain to be trained.

[0026] The training airspace refers to the specific airspace range required for the target detection model to be actually deployed and to perform detection tasks. For example, for a drone performing an inspection task, the training airspace can be defined as the low-altitude three-dimensional geographic area covered by the inspection task performed by the drone. For instance, for a drone performing power inspection over the core business district of a city, the training airspace can be defined as a cylindrical airspace with a radius of 5 kilometers and an altitude of 0-300 meters, centered on the geographic center of the core business district. Or, for a drone performing fire prevention inspection in a mountainous area, the training airspace can be defined as an irregular three-dimensional airspace with an altitude of 500-2000 meters within the geographic coordinates of the mountainous area.

[0027] It should be noted that, in practice, the designation of the training airspace should take into account constraints such as mission type, aircraft performance, air traffic control restrictions and geographical characteristics to ensure that it is both feasible for actual deployment and sufficient to cover all key testing scenarios.

[0028] The following section uses a computer device as the execution subject to elaborate on the model training method provided by this invention. The physical form of this computer device is not specifically limited here; it can be a lightweight server deployed on an edge computing node, a high-performance computing cluster in a cloud data center, or an airborne computing unit embedded within the aircraft itself, etc.

[0029] In this embodiment of the invention, for a defined training airspace, firstly, multi-view, overlapping real airspace images within the training airspace are acquired. To ensure that the acquired real airspace images can cover typical elements within the training airspace, parameters such as viewpoint distribution, overlap, and number of images can be determined according to actual needs. For example, in a scenario of a core urban business district, 50 multi-view real airspace images with an overlap rate of 30% can be acquired, covering key building facades, power facilities, and traffic elements.

[0030] After acquiring the real spatial image of the training spatial domain, the computer device inputs it into the 3D reconstruction model to reconstruct the 3D scene, generating a high-fidelity real 3D scene point cloud of the training spatial domain. The model architecture and training strategy for the 3D reconstruction model are not limited here. For example, the VisualGeometry Grounded Transformer (VGGT) architecture can be used, which achieves millimeter-level reconstruction accuracy in end-to-end training by fusing geometric priors and semantic features from multi-view images.

[0031] In S120, real 3D scene point clouds are imported into rendering and annotation tools to generate digital twin scenes of the spatial domain to be trained.

[0032] Rendering and annotation tools refer to software tools that can support the import of 3D point cloud data and high-fidelity rendering, semantic segmentation and instance annotation capabilities. There is no limitation on which rendering and annotation tool to use. For example, Unity Perception Toolkit can be used as a rendering and annotation tool.

[0033] In this embodiment of the invention, after generating a real 3D scene point cloud of the airspace to be trained through a 3D reconstruction model, the computer device calls rendering and annotation tools to assign materials, simulate lighting, and model dynamic weather, thereby generating a digital twin scene that is physically consistent with the real airspace.

[0034] In S130, based on the digital twin scene, the simulated spatial domain image and annotation information corresponding to the spatial domain to be trained are obtained through rendering and annotation tools.

[0035] After generating the digital twin scene of the airspace to be trained, the computer equipment uses rendering and annotation tools to render the digital twin scene in multiple rounds according to different camera positions, viewpoints, lighting conditions and weather parameters, generating simulated airspace images covering different observation angles and environmental changes (i.e. different lighting and weather conditions); each round of rendering synchronously generates pixel-level semantic segmentation maps and instance masks.

[0036] For each generated simulated spatial image, the computer device associates its corresponding semantic segmentation map with an instance mask to generate corresponding annotation information. Specifically, the computer device can calculate the bounding box coordinates of each target based on the instance mask, and, combined with the semantic segmentation map, identify the semantic category to which each target belongs. The bounding box coordinates and semantic category then constitute two-dimensional annotation information indicating the location and category of the target.

[0037] In other embodiments, the computer device can also generate a depth map synchronously with the rendering and annotation tools in each round of rendering. Accordingly, when generating annotation information, in addition to the bounding box coordinates and semantic category, the depth information of each target can also be attached to construct a three-dimensional perception standard information containing spatial location, semantic category and depth dimension, thereby improving the generalization ability and robustness of the target detection model, especially significantly improving the recognition accuracy of low-altitude slow-moving small targets in complex airspace.

[0038] In step S140, a discriminator network is constructed. The weight parameters of the feature extraction network and the detection head network are fixed. The feature extraction network extracts the first feature maps of both the real and simulated spatial domain images. The weight parameters of the discriminator network are updated with the optimization objective of maximizing the discriminator network's ability to distinguish between these two first feature maps. The weight parameters of the discriminator network are fixed, and the feature extraction network extracts the second feature map of the simulated spatial domain image. Based on the annotation information, a mask matrix of the same size as the second feature map is obtained. The second feature map is masked according to the mask matrix to obtain the masked second feature map. The weight parameters of the feature extraction network and the detection head network are updated with the joint optimization objective of minimizing the discriminator network's ability to distinguish the masked second feature map and the detection loss of the detection head network on the second feature map. The weight updates of the discriminator network and the joint weight updates of the feature extraction network and the detection head network are performed alternately until a preset stopping condition is met.

[0039] In this embodiment, domain adversarial training is used to train the target detection model. Domain adversarial training, also known as domain adaptation training, is a core technology of transfer learning, used to eliminate feature distribution differences between different domains. The purpose of domain adversarial training is to map data from source and target domains with different distributions to a shared feature space, making the distribution of source and target domain features in that feature space as consistent as possible.

[0040] Accordingly, in this embodiment of the invention, the computer device can use simulated spatial images as source domain data and real spatial images as target domain data, and train the target detection model using domain adversarial training until a preset stopping condition is met, thereby enabling the target detection model to have strong generalization ability and high detection accuracy on real spatial images.

[0041] For example, a computer device acquires 50 real airspace images of the airspace to be trained, and uses these real airspace images to generate a digital twin scene corresponding to the airspace to be trained. Then, based on the digital twin scene, 2000 simulated airspace images and corresponding annotation information are generated. Subsequently, using these 2000 simulated airspace images as the source domain and the 50 real airspace images as the target domain, domain adversarial training is performed on the target detection model. This allows the target detection model to complete efficient training using 2000 simulated airspace images with only 50 real airspace images as support. This enables the target detection model to generalize to real airspace scenes, achieve high-precision target detection of real airspace images, and accurately identify key targets such as birds and low-altitude obstacles.

[0042] For details, please refer to Figure 2In this embodiment of the invention, the object detection model is composed of a feature extraction network and a detection head network working together. The feature extraction network is responsible for mapping the input image into a feature map, and the detection head network performs bounding box regression and category classification based on the feature map. The specific structures of the feature extraction network and the detection head network are not limited here. For example, the feature extraction network can adopt the ResNet architecture, and the detection head network can adopt the Faster R-CNN architecture.

[0043] Furthermore, to enable domain adversarial training of the target detection model, the computer device also constructs a discriminator network. This discriminator takes the feature map output by the feature extraction network as input, determines whether it originates from a real spatial domain image or a simulated spatial domain image, and outputs the discrimination result and its confidence level. Similarly, the specific structure of the discriminator network is not limited in the embodiments of the present invention. For example, the discriminator network can adopt a multi-layer fully connected network structure.

[0044] After the discriminator network is constructed, the computer device alternately performs weight updates for the discriminator network and joint weight updates for the feature extraction network and the detection head network until the preset stopping condition is met.

[0045] In the discriminator network weight update, the computer device fixes the weight parameters of the feature extraction network and the detection head network. The real spatial image and the simulated spatial image are respectively processed by the feature extraction network to extract features to obtain the first feature map, which is then input into the discriminator network for domain discrimination. With the optimization goal of maximizing the discriminator network's ability to distinguish the source of the first feature map of the two, the weight parameters of the discriminator network are updated through backpropagation, so that the discriminator network can distinguish whether the first feature map comes from the real spatial image or the simulated spatial image.

[0046] For the joint weight update of the feature extraction network and the detection head network, to avoid the domain differences of background features masking the domain adaptation effect of the target features, this embodiment of the invention only performs masking focusing on the feature regions of the target, so that the domain adaptation process acts more accurately on the target semantic region, thereby focusing on the "domain adaptation of the target itself". Specifically, the computer device first determines the spatial position of the target's feature regions on the second feature map based on the semantic category in the annotation information and the spatial correspondence between the original simulated spatial domain image and the second feature map, and then generates a mask matrix that only covers the target's feature regions. The values ​​of the mask matrix are set to 1 in the target's feature regions and 0 in the background regions.

[0047] Subsequently, the computer device performs a masking operation on the feature map based on the acquired mask matrix. This involves multiplying the second feature map element-wise with the mask matrix, thereby setting the background feature regions in the second feature map to zero and retaining only the target feature regions. In practice, this element-wise multiplication of the mask matrix and the second feature map can be achieved through a broadcast mechanism. This involves expanding the mask matrix along the channel dimension to match the number of channels in the second feature map, and then broadcasting and copying it along the spatial dimension to achieve element-wise multiplication.

[0048] Finally, the computer device fixes the weight parameters of the discriminator network, with the joint optimization objective of minimizing the discriminator network's ability to distinguish the masked second feature map and the detection loss of the detection head network on the original second feature map. The weight parameters of the feature extraction network and the detection head network are updated through backpropagation, making the feature maps extracted by the feature extraction network more discriminatively invariant across domains. This reduces the distribution difference between the source domain (simulation) and the target domain (real), while simultaneously improving the detection head network's generalization ability in real-world scenarios. The total loss can be expressed as... ,in To detect loss, it is used to measure the accuracy of bounding box regression and category classification; The domain adversarial loss is used to measure how easily the masked second feature map confuses the discriminator network. , This is a balancing coefficient used to balance the optimization weights of the detection task and the domain adversarial task, and can be adjusted according to actual needs.

[0049] It is understandable that small targets have a very low target proportion, and the target pixel proportion in the masked second feature map is usually less than 0.5%. In order to avoid the gradient of the target being diluted by a large number of zero-value background pixels, the embodiments of this application further perform foreground normalization processing on the domain adversarial loss: For the masked second feature map, each target pixel is traversed, and the domain prediction result (belonging to the simulated domain or the real domain) is output by the discriminator network. The binary cross-entropy is calculated with the domain label (1 when the masked second feature map comes from the real spatial domain image and 0 when it comes from the simulated spatial domain image). Then, the sum of the loss values ​​of all target pixels is divided by the total number of target pixels to obtain the final value of the domain adversarial loss, thereby eliminating the gradient dilution effect caused by the small target proportion and ensuring that the domain adversarial loss has a sufficient and balanced gradient backpropagation strength for the target semantic region.

[0050] It should be noted that the embodiments of this application do not restrict the order of weight updates for the discriminator network and the joint weight updates for the feature extraction network and the detection head network. The discriminator network weight update can be performed first, followed by the joint weight update for the feature extraction network and the detection head network, or vice versa. The core principle is to use a minimization-maximization game mechanism to make the feature map output by the feature extraction network gradually indistinguishable between the source domain (simulation) and the target domain (real), while simultaneously ensuring that the detection head network maintains high-precision detection performance in the domain-invariant feature space. This adversarial training mechanism effectively alleviates the distribution shift between the real and simulated spatial domains, thereby significantly improving the generalization ability of the target detection model on real spatial images.

[0051] Furthermore, the embodiments of the present invention do not limit the configuration of the preset stopping conditions. For example, training round number threshold, loss function convergence threshold, or detection accuracy threshold can be used as the judgment criteria. For instance, the preset stopping conditions can be configured as the number of training rounds reaching the training round number threshold, or the detection accuracy reaching the detection accuracy threshold, or the change of the loss function within a certain number of consecutive rounds being lower than the convergence threshold, and so on.

[0052] Optionally, in one embodiment, before performing a masking operation on the second feature map according to the mask matrix to obtain the masked second feature map, the method further includes: For each target indicated by the annotation information, if the target area in the second feature map is less than or equal to the area threshold, then the mask region corresponding to the target in the mask matrix is ​​expanded according to the target area.

[0053] It is understandable that small targets such as drones and birds may only occupy 1-4 pixels in the feature map. During the downsampling process, the target features will spread to the surrounding space, causing the mask matrix to fail to cover the complete feature response area, thereby weakening the effectiveness of the masking operation. Therefore, this application further introduces an adaptive expansion mechanism for the mask matrix.

[0054] In this embodiment, an area threshold is configured, and the unit of the area threshold is the square of the feature map pixels. The specific value of the area threshold is not limited here, and it may be determined according to the actual detection needs. For example, in this embodiment, the area threshold is configured as 16, which corresponds to a 4×4 pixel area on the feature map.

[0055] In this embodiment of the application, after the computer device obtains the mask matrix, it does not directly use the mask matrix to perform masking operations on the second feature map. Instead, it first determines whether the target area of ​​each target in the second feature map is too small based on the area threshold. If so, it adaptively expands the corresponding mask area according to the target area of ​​each target to ensure that the masking operation can completely cover the target feature response range.

[0056] Optionally, in one embodiment, the mask region corresponding to the target in the target area expansion mask matrix includes: If the integer part of the square root of the ratio of the area threshold to the target area is less than or equal to the expansion pixel threshold, then the mask region in the mask matrix corresponding to the target is expanded according to the expansion pixel threshold; otherwise, the mask region in the mask matrix corresponding to the target is expanded according to the pixel value represented by the integer part.

[0057] This application further provides an optional mask expansion strategy, which introduces an expansion pixel threshold to prevent insufficient mask coverage due to the small target area, while suppressing background interference caused by blind expansion.

[0058] In this process, for each target indicated by the annotation information, the computer device first calculates the ratio of the area threshold to the target area, then calculates the square root of the ratio, and takes the integer part as the initial expansion coefficient. Subsequently, this coefficient is compared with a preset expansion pixel threshold, and the larger one is selected as the expansion coefficient to expand the mask region corresponding to the target in the mask matrix. It should be noted that the specific value of the expansion pixel threshold is not limited in this embodiment.

[0059] For example, assuming the area threshold is 16 and the expansion pixel threshold is 2, if the target area in the second feature map is 4, then the ratio of the area threshold to the target area is 8, the square root of which is approximately 2.83, and the integer part is 2. Since this integer part is equal to the expansion pixel threshold (2), 2 is finally used as the expansion coefficient to expand the mask area corresponding to the target in the mask matrix by 2 pixels. For example, if the area of ​​a target in the second feature map is 1, then the ratio of the area threshold to the target area is 16, the square root of which is 4, and the integer part is 4. Since the integer part is greater than the expansion pixel threshold (2), 4 is finally used as the expansion coefficient to expand the mask area corresponding to the target in the mask matrix by 4 pixels.

[0060] Optionally, in one embodiment, before performing a masking operation on the second feature map according to the mask matrix to obtain the masked second feature map, the method further includes: For each target indicated by the annotation information, the boundary of the mask region of that target in the mask matrix is ​​subjected to Gaussian softening.

[0061] This application further introduces Gaussian softening to alleviate the feature truncation effect caused by hard boundaries and avoid gradient discontinuity caused by hard truncation boundaries.

[0062] For each target indicated by the annotation information, the computer device performs Gaussian softening on the boundary of the corresponding mask region in the mask matrix to make the boundary transition more natural. Specifically, this Gaussian softening can be implemented by convolving the mask region boundary with a Gaussian filter operator of standard deviation σ, causing the mask value to smoothly decay from 1 inside the mask region to 0 outside. The value of σ is not limited here; for example, in this embodiment, σ can be 1.5.

[0063] In other embodiments, the mask matrix can be expanded first, followed by Gaussian softening. In this case, for a mask region, σ can be dynamically determined based on the expansion coefficient of the mask region; that is, the larger the expansion coefficient, the larger the value of σ, in order to match the boundary smoothing requirements of targets at different scales.

[0064] Optionally, in one embodiment, the feature extraction network includes a general feature extraction sub-network and a domain adaptation sub-network. The general feature extraction sub-network is used to extract low-level features shared across different spatial domains, and the domain adaptation sub-network is used to extract high-level features adapted to different spatial domains. The joint optimization objective is to minimize the discriminant network's ability to discriminate the masked second feature map and the detection head network's detection loss on the second feature map. The weight parameters of the feature extraction network and the detection head network are updated, including: The weight parameters of the general feature extraction sub-network are fixed, and the joint optimization objective is to minimize the discriminant network's ability to discriminate the masked second feature map and the detection head network's detection loss on the second feature map. The weight parameters of the domain adaptation sub-network and the detection head network are then updated.

[0065] It is understandable that during the actual flight of an aircraft, there are a large number of different airspace environments. If a dedicated model is trained separately for each airspace environment, the deployment cost and maintenance complexity will be significantly increased. Therefore, this invention proposes a cross-domain adaptive mechanism.

[0066] Please refer to Figure 3 In this embodiment of the invention, the feature extraction network is divided into two parts: a general feature extraction subnetwork and a domain adaptation subnetwork. The general feature extraction subnetwork is responsible for extracting low-level features shared across spatial domains, while the domain adaptation subnetwork focuses on extracting high-level features adapted to different spatial domains.

[0067] Therefore, after the object detection model has been trained in at least one spatial domain, when transferring the model to the new training spatial domain, only the parameters of the domain adaptation sub-network and the detection head network need to be fine-tuned to quickly adapt to the new spatial domain, without retraining the entire network. Correspondingly, when the computer device jointly updates the feature extraction network and the detection head network, in addition to fixing the weight parameters of the discriminator network, it also fixes the weight parameters of the general feature extraction sub-network. The joint optimization objective is to minimize the discriminative ability of the discriminator network on the masked second feature map and the detection loss of the detection head network on the second feature map, updating the weight parameters of the domain adaptation sub-network and the detection head network. The total loss can be expressed as... , The newly added domain-invariant class-prototype contrastive loss is used to constrain the degree of forgetting of learned spatial domain knowledge and the domain adaptability of the object detection model, ensuring that the object detection model can robustly maintain its detection performance in historical spatial domains when adapting to new spatial domains. The newly added balance coefficient, and , We optimize the weights of detection loss, domain adversarial loss, and domain invariant class prototype contrast loss by balancing them.

[0068] Among them, the domain-invariant class prototype contrastive loss is based on class prototype construction. The domain-invariant class prototype is the "cross-domain shared center vector" of each target category in the feature space. It also integrates simulation and real data features from multiple spatial domains, which will be explained in detail below.

[0069] First, let's introduce the basic definitions involved in the domain-invariant class prototype contrastive loss: Historical airspace set already adapted: {D1, D2, ..., D...} K (A total of K airspaces) Target category set: C = {c1, c2, ..., c m (e.g., drones, birds, balloons, etc.) Domain-invariant class prototype: Each class c corresponds to a d-dimensional vector π c ∈R d (d is the feature dimension output by the feature extraction network), storing the common feature representation of this category across all spatial domains.

[0070] After completing the training of the first spatial domain D1, the feature extraction network extracts all feature maps of category c from all simulated spatial domain images and all real spatial domain images corresponding to this spatial domain. The harmonic mean is taken as the initial domain-invariant class prototype of category c. ; This represents the set of feature maps of category c in the simulated spatial domain image of the first spatial domain. This represents the set of feature maps of category c in the true spatial image of the first spatial domain.

[0071] After training on the (k+1)th new airspace, the prototype is updated using an exponential moving average method, incorporating the feature information of the new airspace: ; in, This represents the momentum coefficient, with a value range of [0.1, 0.3]. This represents the set of all feature maps of category c in the simulated spatial domain image of the (k+1)th spatial domain. Let represent the set of all feature maps of category c in the real spatial image of the (k+1)th spatial domain.

[0072] As shown above, by continuously updating the domain-invariant class prototypes of each category and constraining cross-domain consistency, the domain-invariant class prototypes of each category can integrate the category features of each category in the historical and new spatial domains, and represent the "semantic anchors" shared by each category across domains in the feature space.

[0073] Based on the above domain-invariant class prototypes, the domain-invariant class prototype contrastive loss can be expressed as: ; in, This represents the second feature map after masking the i-th simulated spatial domain image of the training spatial domain in the current training iteration. This indicates the real category to which the i-th simulated spatial image belongs. This represents the domain-invariant class prototype corresponding to the real category to which the i-th simulated spatial image belongs. Therefore, the numerator part encourages... and Alignment, suppressed by the denominator part Move closer to prototypes of other categories.

[0074] In practical implementation, a prototype memory can be used to manage the domain-invariant class prototypes of historical spatial domains. Specifically, the prototype memory stores data in key-value pairs, with each record containing: a category identifier c, a spatial domain identifier d, and the current domain-invariant category prototype. Cross-domain alignment contribution. When the prototype memory storage capacity reaches its limit, the record with the lowest cross-domain alignment contribution is discarded. Specifically, when training in a new spatial domain is completed, the current domain-invariant class prototype can be calculated. The mean cosine similarity with all historical spatial domain invariant class prototypes is used as the cross-domain alignment contribution.

[0075] Optionally, in one embodiment, constructing a discriminator network includes: Construct a first discriminator network corresponding to the general feature extraction subnetwork and a second discriminator network corresponding to the domain adaptation subnetwork; The first feature maps of the real and simulated spatial domain images are extracted using a feature extraction network. The discriminator network's weight parameters are then updated with the optimization objective of maximizing its ability to distinguish between these two first feature maps. These parameters include: The first low-level features of the real spatial domain image and the simulated spatial domain image are extracted by the general feature extraction subnetwork, and the first high-level features of the real spatial domain image and the simulated spatial domain image are extracted by the domain adaptation subnetwork. The weight parameters of the first discriminator network and the second discriminator network are updated with the joint optimization objective of maximizing the discriminant ability of the first discriminator network to distinguish the first low-level features of the two and the discriminator network to distinguish the first high-level features of the two.

[0076] In this embodiment of the invention, when constructing the discriminator network, the computer device can construct dual-path discriminators corresponding to the general feature extraction sub-network and the domain adaptation sub-network, respectively. These are a first discriminator network focusing on low-level feature discrimination and a second discriminator network focusing on high-level feature discrimination, thereby coordinating and constraining domain differences at multiple abstract levels of feature extraction.

[0077] Correspondingly, when updating the weight parameters of the discriminator network, the computer device extracts the first low-level features of the real spatial image and the simulated spatial image respectively through the general feature extraction sub-network, and extracts the first high-level features of the real spatial image and the simulated spatial image respectively through the domain adaptation sub-network. Then, with the joint optimization objective of maximizing the discrimination ability of the first discriminator network on the first low-level features of the two and the discrimination ability of the second discriminator network on the first high-level features of the two, the weight parameters of the first discriminator network and the second discriminator network are updated synchronously through back-phase propagation, so that the differences between low-level features and high-level features are fully decoupled and synergistically suppressed in adversarial training, thereby enabling the domain adaptation process to converge synchronously on low-level features and high-level features.

[0078] The layered domain adversarial mechanism described above can alleviate the domain difference modeling bias caused by the limited receptive field of a single discriminator network, and improve the robustness of the target detection model to complex spatial domains.

[0079] Optionally, in one embodiment, based on the digital twin scene, a simulated spatial domain image and annotation information corresponding to the spatial domain to be trained are obtained through rendering and annotation tools, including: Identify dynamic targets in digital twin scenes using rendering and annotation tools, and obtain kinematic models corresponding to the dynamic targets; Based on kinematic models, dynamic targets are driven to simulate motion in digital twin scenes through rendering and annotation tools; During the simulated motion process, simulated spatial images with different image content and their annotation information are collected through rendering and annotation tools.

[0080] To enhance the ability of target detection models to detect dynamic targets in the air, this embodiment of the invention further proposes a dynamic target enhancement strategy driven by a kinematic model.

[0081] Dynamic targets refer to physical targets in digital twin scenarios that have modelable motion trajectories, such as drones, birds, kites, and balloons. Their kinematic models are mathematical expressions based on the principles of Newtonian mechanics and aerodynamics, covering time-varying state quantities such as position, velocity, acceleration, attitude angle, and angular velocity, which are used to accurately describe the motion evolution process of dynamic targets in the three-dimensional spatial domain.

[0082] In this embodiment of the invention, the computer device first performs semantic parsing on the digital twin scene using rendering and annotation tools, identifies the semantic categories of all targets present, and matches the semantic categories with a predefined dynamic target category library, thereby accurately selecting the dynamic targets that need to drive the simulation. For example, a dynamic target category library containing typical low-altitude dynamic targets such as drones, birds, kites, and balloons can be predefined. When the semantic segmentation identifies the "bird" target in the digital twin scene, the computer device identifies it as the dynamic target that needs to drive the simulation.

[0083] After identifying dynamic targets in the digital twin scene, the computer device further acquires a kinematic model corresponding to each dynamic target. This kinematic model can be pre-built by the computer device, retrieved from a professional kinematic database, or generated in real time through an external interface connected to a third-party high-precision motion simulation system, and so on.

[0084] After acquiring the kinematic model, the computer device embeds it into rendering and annotation tools to drive the dynamic target to simulate motion within the digital twin scene, thereby evolving its three-dimensional motion trajectory over time. During this process, the computer device simultaneously performs multiple rounds of rendering through rendering and annotation tools, generating simulated spatial images that cover different observation angles, environmental changes (i.e., different lighting and weather conditions), and dynamic target attitude changes. Each round of rendering simultaneously generates pixel-level semantic segmentation maps, instance masks, and motion parameters of the dynamic target in the corresponding simulated spatial image (such as motion-related parameters like velocity, acceleration, and attitude angles).

[0085] For each generated simulated spatial image, the computer device associates it with its corresponding semantic segmentation map, instance mask, and motion parameters to generate corresponding annotation information. Specifically, the computer device can calculate the bounding box coordinates of each target based on the instance mask, and identify the semantic category of each target by combining the semantic segmentation map. The bounding box coordinates, semantic category, and motion parameters together constitute three-dimensional annotation information indicating the target's position, category, and motion state.

[0086] The above dynamic target enhancement strategy based on kinematic model can make the generated simulated airspace images more closely match the physical motion laws of dynamic targets in the real airspace, providing high-fidelity and strong generalization training samples for subsequent training processes, and significantly improving the detection accuracy and robustness of the target detection model for dynamic targets.

[0087] Optionally, in one embodiment, before performing domain adversarial training on the target detection model based on the real spatial domain image, the simulated spatial domain image, and the annotation information, the method further includes: Identify the image sensor of the detection device to be deployed for the target detection model, and obtain the imaging characteristic parameters of the image sensor; Based on the imaging characteristic parameters, the simulated spatial image is enhanced by imaging characteristic matching to obtain an enhanced simulated spatial image; Based on real spatial domain images, simulated spatial domain images, and annotation information, the target detection model undergoes domain adversarial training, including: Based on real airspace images, enhanced simulated airspace images, and annotation information, the target detection model is trained for domain adversarial training.

[0088] To further enhance the generalization ability of the target detection model in real deployment scenarios, this embodiment of the invention proposes an optional imaging feature matching mechanism.

[0089] In this approach, the computer equipment does not directly use the originally generated simulated spatial domain images for domain adversarial training. Instead, it first determines the image sensor used by the detection device to which the target detection model will be deployed, and obtains the imaging characteristic parameters of that image sensor, including but not limited to spectral response curves, dynamic range, noise model, pixel size, and lens distortion coefficients. For example, the computer equipment can construct a query request based on the sensor signal of the determined image sensor and initiate a parameter retrieval from databases such as sensor manufacturer databases or standardized parameter libraries to obtain the imaging characteristic parameters of that image sensor.

[0090] Subsequently, the computer equipment performs physically interpretable imaging characteristic matching enhancement on the simulated spatial image based on the acquired imaging characteristic parameters to obtain an enhanced simulated spatial image. For example, taking the acquired imaging characteristic parameters including the spectral response curve, noise model, and lens distortion coefficient as an example, when the computer equipment performs imaging characteristic matching enhancement on the simulated spatial image, it first performs channel reweighting correction on the simulated spatial image based on the spectral response curve, then superimposes Poisson-Gaussian mixed noise that conforms to the noise model, and finally applies the lens distortion coefficient for geometric distortion compensation to obtain an enhanced simulated spatial image. This ensures that the enhanced simulated spatial image is highly consistent with the real spatial image actually acquired by the detection equipment through the image sensor in terms of spectral distribution, noise texture, and geometric deformation, effectively bridging the physical gap between simulation and real imaging.

[0091] Accordingly, after obtaining the enhanced simulated airspace image, the computer device performs domain adversarial training on the target detection model based on the real airspace image, the enhanced simulated airspace image, and the annotation information, until a preset stopping condition is met. For details on how to perform domain adversarial training on the target detection model based on the real airspace image, the enhanced simulated airspace image, and the annotation information, please refer to the relevant descriptions in the above embodiments regarding domain adversarial training of the target detection model based on the real airspace image, the simulated airspace image, and the annotation information; these details will not be repeated here.

[0092] The above imaging matching mechanism can bridge the physical gap between simulation and real imaging, and significantly improve the robustness and generalization ability of the target detection model in real deployment scenarios.

[0093] As can be seen from the above, the model training scheme provided by the present invention obtains a real airspace image of the airspace to be trained, and based on the real airspace image, performs 3D scene reconstruction through a 3D reconstruction model to generate a real 3D scene point cloud corresponding to the airspace to be trained; imports the real 3D scene point cloud into a rendering and annotation tool to generate a digital twin scene of the airspace to be trained; based on the digital twin scene, obtains a simulated airspace image and annotation information corresponding to the airspace to be trained through a rendering and annotation tool; and performs domain adversarial training on the target detection model according to the real airspace image, the simulated airspace image, and the annotation information until a preset stopping condition is met. Therefore, only a small number of real spatial images are needed to construct a digital twin scene corresponding to the spatial domain to be trained. This digital twin scene can then be used to generate simulated spatial images and annotation information corresponding to the spatial domain to be trained in batches for model training. This significantly reduces the dependence on large-scale real annotation data, thereby greatly reducing the cost of data collection and manual annotation. At the same time, through a target-based masking mechanism, only the features of the target are allowed to participate in domain adversarial training, avoiding the domain differences of background features from "stealing" the optimization resources of domain adversarial training. This ensures that the target detection model learns "the domain-invariant features of the target itself" rather than "the domain-invariant features of the background". Thus, while achieving low-cost training of the target detection model, the detection accuracy of the target detection model for small targets is improved.

[0094] To facilitate better implementation of the above model training methods, this application also provides a corresponding model training apparatus. The meanings of the terms used are the same as in the above model training methods; for specific implementation details, please refer to the descriptions in the above method embodiments.

[0095] Please refer to Figure 4 , Figure 4 This is a schematic diagram of the model training device provided in an embodiment of the present invention, as shown below. Figure 4 As shown, the model training device may include a 3D reconstruction module 210, a scene generation module 220, a sample generation module 230, and a model training module 240, wherein, The 3D reconstruction module 210 is used to acquire the real spatial image of the spatial domain to be trained, and based on the real spatial image, to reconstruct the 3D scene through the 3D reconstruction model to generate a real 3D scene point cloud corresponding to the spatial domain to be trained. The scene generation module 220 is used to import real 3D scene point clouds into rendering and annotation tools to generate a digital twin scene of the spatial domain to be trained. The sample generation module 230 is used to obtain simulated spatial domain images and annotation information corresponding to the spatial domain to be trained based on the digital twin scene through rendering and annotation tools; The model training module 240 is used to construct a discriminator network; fix the weight parameters of the feature extraction network and the detection head network, extract the first feature maps of the real spatial image and the simulated spatial image through the feature extraction network, and update the weight parameters of the discriminator network with the optimization objective of maximizing the discriminator network's ability to distinguish between the two first feature maps; fix the weight parameters of the discriminator network, extract the second feature map of the simulated spatial image through the feature extraction network, and obtain a mask matrix with the same size as the second feature map according to the annotation information; perform a masking operation on the second feature map according to the mask matrix to obtain the masked second feature map; update the weight parameters of the feature extraction network and the detection head network with the joint optimization objective of minimizing the discriminator network's ability to distinguish the masked second feature map and the detection loss of the detection head network on the second feature map; alternately perform weight updates for the discriminator network and joint weight updates for the feature extraction network and the detection head network until a preset stopping condition is met.

[0096] Optionally, in one embodiment, the sample generation module 230 is used to identify dynamic targets in the digital twin scene through rendering and annotation tools, and obtain the kinematic model corresponding to the dynamic targets; based on the kinematic model, the dynamic targets are driven to perform simulated motion in the digital twin scene through rendering and annotation tools; during the simulated motion, simulated spatial images with image content differences and their annotation information are collected through rendering and annotation tools.

[0097] Optionally, in one embodiment, the model training module 240 is further configured to, for each target indicated by the annotation information, if the target area in the second feature map is less than or equal to an area threshold, expand the mask region corresponding to the target in the mask matrix according to the target area.

[0098] Optionally, in one embodiment, the model training module 240 is configured to, if the integer part of the square root of the ratio of the area threshold to the target area is less than or equal to the expansion pixel threshold, expand the mask region in the mask matrix corresponding to the target according to the expansion pixel threshold; otherwise, expand the mask region in the mask matrix corresponding to the target according to the pixel value represented by the integer part.

[0099] Optionally, in one embodiment, the model training module 240 is further configured to perform Gaussian softening on the boundary of the mask region of each target indicated by the annotation information in the mask matrix.

[0100] Optionally, in one embodiment, the feature extraction network includes a general feature extraction sub-network and a domain adaptation sub-network. The general feature extraction sub-network is used to extract low-level features shared by different spatial domains, and the domain adaptation sub-network is used to extract high-level features adapted to different spatial domains. The model training module 240 is used to fix the weight parameters of the general feature extraction sub-network and update the weight parameters of the domain adaptation sub-network and the detection head network with the joint optimization objective of minimizing the discriminator network's ability to discriminate the masked second feature map and the detection loss of the detection head network on the second feature map.

[0101] Optionally, in one embodiment, the model training module 240 is used to construct a first discriminator network corresponding to the general feature extraction sub-network and a second discriminator network corresponding to the domain adaptation sub-network; the general feature extraction sub-network extracts the first low-level features of the real spatial image and the simulated spatial image respectively, and the domain adaptation sub-network extracts the first high-level features of the real spatial image and the simulated spatial image respectively; the weight parameters of the first discriminator network and the second discriminator network are updated with the joint optimization objective of maximizing the discriminative ability of the first discriminator network to the first low-level features of the two and the discriminative ability of the second discriminator network to the first high-level features of the two.

[0102] Optionally, in one embodiment, the sample generation module 230 is further configured to determine the image sensor of the detection device to be deployed by the target detection model, and obtain the imaging characteristic parameters of the image sensor; based on the imaging characteristic parameters, the simulated spatial image is enhanced by imaging characteristic matching to obtain an enhanced simulated spatial image; The model training module 240 is used to perform domain adversarial training on the target detection model based on real spatial images, enhanced simulated spatial images, and annotation information.

[0103] It should be noted that the information interaction and execution process between the above modules are based on the same concept as the method embodiments of this application. For details on their specific functions and technical effects, please refer to the method embodiments section, which will not be repeated here.

[0104] In one embodiment, a computer device is provided, the internal structure of which can be shown in the figure below. Figure 5 As shown.

[0105] The computer device may include components such as a processor 101 with one or more processing cores, a memory 102 with one or more computer-readable storage media, a power supply 103, and an input unit 104. Those skilled in the art will understand that... Figure 5 The computer device structure shown does not constitute a limitation on the computer device and may include more or fewer components than shown, or combine certain components, or have different component arrangements. Wherein: The processor 101 is the control center of the computer device, connecting various parts of the computer device through various interfaces and lines. It performs various functions and processes data by running or executing software programs and / or modules stored in the memory 102, and by calling data stored in the memory 102. Optionally, the processor 101 may include one or more processing cores; optionally, the processor 101 may integrate an application processor and a modem processor, wherein the application processor mainly handles the operating system, user interface, and applications, and the modem processor mainly handles wireless communication. It is understood that the modem processor may also not be integrated into the processor 101.

[0106] The memory 102 can be used to store software programs and modules. The processor 101 executes various functional applications and data processing by running the software programs and modules stored in the memory 102. The memory 102 may mainly include a program storage area and a data storage area. The program storage area may store the operating system and application programs required for at least one function (such as sound playback function, image playback function, etc.). The data storage area may store data created according to the use of the computer device. In addition, the memory 102 may include high-speed random access memory and may also include non-volatile memory, such as at least one disk storage device, flash memory device, or other non-volatile solid-state storage device. Accordingly, the memory 102 may also include a memory controller to provide the processor 101 with access to the memory 102.

[0107] Specifically, in this embodiment, the processor 101 loads the executable code corresponding to one or more computer programs into the memory 102, and the processor 101 executes them. Figure 5 The corresponding implementation examples provide model training methods, such as: Acquire real airspace images of the airspace to be trained, and based on the real airspace images, reconstruct the 3D scene using a 3D reconstruction model to generate real 3D scene point clouds corresponding to the airspace to be trained. Import real 3D scene point clouds into rendering and annotation tools to generate digital twin scenes of the spatial domain to be trained; Based on the digital twin scenario, simulated spatial domain images and annotation information corresponding to the spatial domain to be trained are obtained through rendering and annotation tools; Construct a discriminator network; fix the weight parameters of the feature extraction network and the detection head network, extract the first feature maps of the real spatial domain image and the simulated spatial domain image through the feature extraction network, and update the weight parameters of the discriminator network with the optimization objective of maximizing the discriminator network's ability to distinguish between the two first feature maps; fix the weight parameters of the discriminator network, extract the second feature map of the simulated spatial domain image through the feature extraction network, and obtain a mask matrix with the same size as the second feature map based on the annotation information; perform a masking operation on the second feature map based on the mask matrix to obtain the masked second feature map; update the weight parameters of the feature extraction network and the detection head network with the joint optimization objective of minimizing the discriminator network's ability to distinguish the masked second feature map and the detection loss of the detection head network on the second feature map; alternately perform weight updates for the discriminator network and joint weight updates for the feature extraction network and the detection head network until a preset stopping condition is met.

[0108] It should be noted that the computer device provided in the embodiments of the present invention and the model training method provided in the present invention belong to the same concept, and their specific implementation process can be found in the above related embodiments, which will not be repeated here.

[0109] In one embodiment, a computer-readable storage medium is provided, on which a computer program is stored, which, when executed by a processor, implements the model training method described in the above embodiment.

[0110] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. This computer program can be stored in a non-volatile computer-readable storage medium. When executed, the computer program can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided by this invention can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), Rambus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM), etc.

[0111] The present invention also provides a computer program product comprising a computer program that, when executed on a processor, causes the processor to implement the steps in the model training method provided by the present invention.

[0112] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is used as an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above.

[0113] The above-described embodiments are only used to illustrate the technical solutions of the present invention, and are not intended to limit it. Although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present invention, and should all be included within the protection scope of the present invention.

[0114] It should be noted that if any software tools or components not belonging to our company appear in the embodiments of this application, they are merely examples and do not represent actual use. Furthermore, when the above embodiments of this invention are applied to specific products or technologies, and user-related data is involved, user permission or consent must be obtained, and the collection, use, and processing of related data must comply with the relevant laws, regulations, and standards of the relevant countries and regions.

Claims

1. A model training method, characterized in that, The object detection model includes a feature extraction network and a detection head network, and the model training method includes: Acquire a real airspace image of the airspace to be trained, and based on the real airspace image, reconstruct a 3D scene using a 3D reconstruction model to generate a real 3D scene point cloud corresponding to the airspace to be trained. The real 3D scene point cloud is imported into the rendering and annotation tool to generate a digital twin scene of the spatial domain to be trained; Based on the digital twin scenario, the simulation spatial domain image and annotation information corresponding to the spatial domain to be trained are obtained through the rendering and annotation tools; A discriminator network is constructed. The weight parameters of the feature extraction network and the detection head network are fixed. The feature extraction network extracts first feature maps of both the real and simulated spatial domain images. The weight parameters of the discriminator network are updated with the optimization objective of maximizing the discriminator network's ability to distinguish between these two first feature maps. The weight parameters of the discriminator network are fixed, and the feature extraction network extracts a second feature map of the simulated spatial domain image. A mask matrix with the same size as the second feature map is obtained based on the annotation information. The second feature map is masked using the mask matrix to obtain a masked second feature map. The weight parameters of the feature extraction network and the detection head network are updated with the joint optimization objective of minimizing the discriminator network's ability to distinguish the masked second feature map and the detection loss of the detection head network on the second feature map. The weight updates of the discriminator network and the joint weight updates of the feature extraction network and the detection head network are performed alternately until a preset stopping condition is met.

2. The model training method according to claim 1, characterized in that, The step of obtaining a simulated spatial domain image and annotation information corresponding to the spatial domain to be trained based on the digital twin scene through the rendering and annotation tools includes: The rendering and annotation tools are used to identify dynamic targets in the digital twin scene and obtain kinematic models corresponding to the dynamic targets. Based on the kinematic model, the dynamic target is driven to perform simulated motion in the digital twin scene using the rendering and annotation tools; During the simulation motion, the rendering and annotation tools are used to collect simulated spatial images with differences in image content and their annotation information.

3. The model training method according to claim 1, characterized in that, Before performing a masking operation on the second feature map according to the mask matrix to obtain the masked second feature map, the method further includes: For each target indicated by the annotation information, if the target area in the second feature map is less than or equal to the area threshold, then the mask region in the mask matrix corresponding to the target is expanded according to the target area.

4. The model training method according to claim 3, characterized in that, Expanding the mask region in the mask matrix corresponding to the target area based on the target area includes: If the integer part of the square root of the ratio of the area threshold to the target area is less than or equal to the expansion pixel threshold, then the mask region in the mask matrix corresponding to the target is expanded according to the expansion pixel threshold; otherwise, the mask region in the mask matrix corresponding to the target is expanded according to the pixel value represented by the integer part.

5. The model training method according to claim 1, characterized in that, Before performing a masking operation on the second feature map according to the mask matrix to obtain the masked second feature map, the method further includes: For each target indicated by the annotation information, the boundary of the mask region of the target in the mask matrix is ​​subjected to Gaussian softening.

6. The model training method according to claim 1, characterized in that, The feature extraction network includes a general feature extraction subnetwork and a domain adaptation subnetwork. The general feature extraction subnetwork is used to extract low-level features shared across different spatial domains, and the domain adaptation subnetwork is used to extract high-level features adapted to different spatial domains. The joint optimization objective is to minimize the discriminant network's ability to distinguish the masked second feature map and the detection head network's detection loss on the second feature map. The weight parameters of the feature extraction network and the detection head network are updated, including: The weight parameters of the general feature extraction sub-network are fixed, and the joint optimization objective is to minimize the discriminator network's ability to discriminate the masked second feature map and the detection head network's detection loss on the second feature map. The weight parameters of the domain adaptation sub-network and the detection head network are then updated.

7. The model training method according to claim 6, characterized in that, The construction of the discriminator network includes: Construct a first discriminator network corresponding to the general feature extraction subnetwork and a second discriminator network corresponding to the domain adaptation subnetwork; The step involves extracting first feature maps from both the real spatial image and the simulated spatial image using the feature extraction network, and updating the weight parameters of the discriminator network with the optimization objective of maximizing the discriminator network's ability to distinguish between the two first feature maps. This includes: The first low-level features of the real spatial image and the simulated spatial image are extracted by the general feature extraction subnetwork, and the first high-level features of the real spatial image and the simulated spatial image are extracted by the domain adaptation subnetwork. The weight parameters of the first discriminator network and the second discriminator network are updated with the joint optimization objective of maximizing the discriminant ability of the first discriminator network to distinguish the first low-level features of the two and the discriminator network to distinguish the first high-level features of the two.

8. The model training method according to any one of claims 1-7, characterized in that, Before performing domain adversarial training on the target detection model based on the real spatial image, the simulated spatial image, and the annotation information, the method further includes: The image sensor of the detection device to be deployed with the target detection model is determined, and the imaging characteristic parameters of the image sensor are obtained; Based on the imaging characteristic parameters, the simulated spatial image is enhanced by imaging characteristic matching to obtain an enhanced simulated spatial image. The step of performing domain adversarial training on the target detection model based on the real spatial domain image, the simulated spatial domain image, and annotation information includes: Based on the real spatial image, the enhanced simulated spatial image, and the annotation information, the target detection model is trained for domain adversarial training.

9. A model training device, characterized in that, The object detection model includes a feature extraction network and a detection head network, and the model training device includes: The 3D reconstruction module is used to acquire a real spatial image of the spatial domain to be trained, and based on the real spatial image, to reconstruct a 3D scene using a 3D reconstruction model to generate a real 3D scene point cloud corresponding to the spatial domain to be trained. The scene generation module is used to import the real 3D scene point cloud into the rendering and annotation tool to generate a digital twin scene of the spatial domain to be trained. The sample generation module is used to obtain, based on the digital twin scene, a simulated spatial domain image and annotation information corresponding to the spatial domain to be trained using the rendering and annotation tools; The model training module is used to construct a discriminator network. It fixes the weight parameters of the feature extraction network and the detection head network, extracts first feature maps of the real spatial image and the simulated spatial image through the feature extraction network, and updates the weight parameters of the discriminator network with the optimization objective of maximizing the discriminator network's ability to distinguish between the two first feature maps. It then fixes the weight parameters of the discriminator network, extracts a second feature map of the simulated spatial image through the feature extraction network, and obtains a mask matrix of the same size as the second feature map based on the annotation information. It performs a masking operation on the second feature map based on the mask matrix to obtain a masked second feature map. It updates the weight parameters of the feature extraction network and the detection head network with the joint optimization objective of minimizing the discriminator network's ability to distinguish the masked second feature map and the detection loss of the detection head network on the second feature map. It alternately performs weight updates for the discriminator network and joint weight updates for the feature extraction network and the detection head network until a preset stopping condition is met.

10. A computer device, characterized in that, The system includes a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program in the memory to implement the model training method according to any one of claims 1 to 8.

Images