Three-dimensional aircraft component positioning method and device based on semantic alignment model, equipment and medium

Abstract

The application discloses a three-dimensional aircraft component positioning method and device based on a semantic alignment model, equipment and a medium, relates to the technical field of semantic alignment, and comprises the following steps: acquiring multi-view image data corresponding to each semantic component of an original three-dimensional aircraft model by using a preset virtual camera to perform multi-view image acquisition on the semantic components, and extracting geometric information corresponding to the semantic components; constructing a multi-modal data set corresponding to the original three-dimensional aircraft model based on the multi-view image data, the geometric information, and text description information corresponding to each semantic component, training an initial semantic alignment model by using the multi-modal data set, and obtaining a target positioning model; inputting natural language description text of a target three-dimensional component sent by a user end and component information of each candidate three-dimensional component into the target positioning model to obtain component semantic matching results, and completing positioning of the target three-dimensional component based on the component semantic matching results. In this way, the target component can be quickly positioned in a complex three-dimensional scene.

Description

Technical Field

[0001] This invention relates to the field of semantic alignment technology, and in particular to a method, apparatus, equipment and medium for 3D aircraft component localization based on a semantic alignment model. Background Technology

[0002] In the field of scientific computing visualization, especially in the post-processing of Computational Fluid Dynamics (CFD), taking aircraft models as an example, these models include components with clear engineering semantics, such as the nose, fuselage, main wing, horizontal tail, and vertical tail. During flow field analysis, researchers need to perform local analysis operations such as streamline drawing and isosurface extraction for different components. How to quickly locate target components in complex 3D scenes becomes a key issue. Traditional methods rely on manual operations or predefined identifiers, which are cumbersome and require a high level of user experience. In recent years, natural language-driven mechanisms have been introduced to improve interaction efficiency, but existing methods are mostly based on predefined names or rule mappings, making it difficult to accurately associate descriptive semantics such as "nose" and "main lifting component" with 3D components. Furthermore, while vision-language pre-trained models can achieve semantic alignment between images and text, they are mainly trained on natural images and have significant limitations in industrial 3D model scenarios: models often lack texture information, topologically similar components such as the main wing and horizontal tail are easily confused in 2D views, and existing methods rely heavily on 2D image features, lacking effective utilization of 3D geometric information. In addition, there is a lack of fine-grained multimodal datasets for scientific visualization, and mainstream datasets only provide category-level annotations, which further limits the accuracy of semantic alignment. This paper addresses the aforementioned problems of inaccurate semantic mapping, easy confusion of similar parts, and insufficient utilization of geometric information.

[0003] Therefore, it is necessary to solve the problem of how to generate a 3D component localization method that can improve the recognition accuracy and interactive effect of complex 3D components. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for 3D aircraft component localization based on a semantic alignment model, capable of rapidly locating target components in complex 3D scenes. The specific solution is as follows: In a first aspect, this application discloses a method for locating three-dimensional aircraft components based on a semantic alignment model, including: A preset virtual camera is used to acquire multi-view images of each semantic component corresponding to the original 3D aircraft model to obtain multi-view image data corresponding to each semantic component, and to extract the geometric information corresponding to the semantic component. The process of acquiring multi-view images of each semantic component corresponding to the original 3D aircraft model using a preset virtual camera includes: performing component-level segmentation on the original 3D aircraft model to obtain each semantic component corresponding to the original 3D aircraft model; generating several view sampling points in the spherical space corresponding to each semantic component, and setting a preset virtual camera at each view sampling point; and acquiring multi-view images of each semantic component using the preset virtual camera to obtain the corresponding multi-view image data. The step of acquiring multi-view images of each semantic component using the preset virtual camera to obtain corresponding multi-view image data includes: acquiring a first view image and a second view image of each semantic component using the preset virtual camera; the first view image is an independent view image containing only the semantic component; the second view image is an enhanced view image containing the semantic component and its contextual information; and determining multi-view image data of each semantic component based on the first view image and the second view image. Based on the multi-view image data, the geometric information, and the text description information corresponding to each semantic component, a multimodal dataset corresponding to the original 3D aircraft model is constructed, and the initial semantic alignment model is trained using the multimodal dataset to obtain the target localization model. The natural language description text of the target 3D component sent by the user and the component information of each candidate 3D component are input into the target localization model to obtain the component semantic matching result, and the localization of the target 3D component is completed based on the component semantic matching result.

[0005] Optionally, constructing the multimodal dataset corresponding to the original 3D aircraft model based on the multi-view image data, the geometric information, and the text description information corresponding to each semantic component includes: The multi-view image data is subjected to image processing to obtain processed image data; the image processing includes image occlusion removal processing and image sharpening processing. Based on preset structured text prompts, text description information corresponding to the processed image data is generated, and a mapping relationship between the text description information and the multi-view image data is constructed. Based on the multi-view image data, the geometric information, the text description information, and the mapping relationship, a multimodal dataset corresponding to the original 3D aircraft model is constructed.

[0006] Optionally, training the initial semantic alignment model using the multimodal dataset to obtain the target localization model includes: Using a preset symmetric loss function, and based on the mapping relationship between the text description information and the multi-view image data, a mapping relationship between image features and text features in the initial semantic alignment model is established to obtain the trained semantic alignment model; The trained semantic alignment model is trained using the multi-view image data and the geometric information to obtain a target localization model.

[0007] Optionally, training the trained semantic alignment model using the multi-view image data and the geometric information to obtain the target localization model includes: The multi-view image data is input into a preset visual encoder to obtain the corresponding visual feature vector, and the geometric information is input into a preset geometric encoder to obtain a preset geometric feature vector; Using a preset geometric feature mapping method, the visual feature vector and the preset geometric feature vector are mapped to the target feature space to obtain initial visual features, and a preset attention mechanism is used to calculate the weights of the initial visual features to obtain the corresponding weight coefficients. The initial visual features are weighted and fused based on the weight coefficients to obtain the target visual features; The target visual features are used to train the trained semantic alignment model to obtain the target localization model.

[0008] Optionally, the step of inputting the natural language description text of the target 3D part sent by the user terminal and the part information of each candidate 3D part into the target localization model to obtain the part semantic matching result includes: The system acquires natural language description text for the target 3D component sent by the user terminal, and uses a preset text encoder to extract features from the natural language description text to obtain a text feature vector. Feature extraction is performed on the component information of each candidate 3D component to obtain component feature vectors. The text feature vectors and component feature vectors are then input into the target localization model to obtain component semantic matching results.

[0009] Secondly, this application discloses a three-dimensional aircraft component positioning device based on a semantic alignment model, comprising: The data extraction module is used to acquire multi-view images of each semantic component corresponding to the original 3D aircraft model using a preset virtual camera, so as to obtain multi-view image data corresponding to each semantic component, and extract the geometric information corresponding to the semantic component; wherein, the step of acquiring multi-view images of each semantic component corresponding to the original 3D aircraft model using a preset virtual camera to obtain multi-view image data corresponding to each semantic component includes: performing component-level segmentation processing on the original 3D aircraft model to obtain each semantic component corresponding to the original 3D aircraft model; generating a number of view sampling points in the spherical space corresponding to each semantic component, and setting a preset virtual camera at each view sampling point; and acquiring multi-view images of each semantic component using the preset virtual camera to obtain the corresponding multi-view image data. The step of acquiring multi-view images of each semantic component using the preset virtual camera to obtain corresponding multi-view image data includes: acquiring a first view image and a second view image of each semantic component using the preset virtual camera; the first view image is an independent view image containing only the semantic component; the second view image is an enhanced view image containing the semantic component and its contextual information; and determining multi-view image data of each semantic component based on the first view image and the second view image. The model training module is used to construct a multimodal dataset corresponding to the original 3D aircraft model based on the multi-view image data, the geometric information, and the text description information corresponding to each semantic component, and to train the initial semantic alignment model using the multimodal dataset to obtain the target localization model. The component localization module is used to input the natural language description text of the target 3D component sent by the user terminal and the component information of each candidate 3D component into the target localization model to obtain the component semantic matching result, and to complete the localization of the target 3D component based on the component semantic matching result.

[0010] Thirdly, this application discloses an electronic device, including: Memory, used to store computer programs; A processor is used to execute the computer program to implement the aforementioned three-dimensional aircraft component localization method based on a semantic alignment model.

[0011] Fourthly, this application discloses a computer-readable storage medium for storing a computer program, which, when executed by a processor, implements the aforementioned three-dimensional aircraft component localization method based on a semantic alignment model.

[0012] As can be seen, in this invention, a preset virtual camera is used to acquire multi-view images of each semantic component corresponding to the original 3D aircraft model to obtain multi-view image data corresponding to each semantic component, and geometric information corresponding to the semantic component is extracted. Based on the multi-view image data, the geometric information, and the text description information corresponding to each semantic component, a multimodal dataset corresponding to the original 3D aircraft model is constructed, and the initial semantic alignment model is trained using the multimodal dataset to obtain a target localization model. The natural language description text for the target 3D component sent by the user terminal and the component information of each candidate 3D component are input into the target localization model to obtain the component semantic matching result, and the localization of the target 3D component is completed based on the component semantic matching result. That is, by introducing the geometric information of the semantic component to train the semantic alignment model, the distinguishability between low-texture and topologically similar 3D components is improved, thereby achieving accurate semantic alignment between natural language and the target component, and obtaining a more accurate 3D component localization model. In this way, during use, by improving the accuracy of semantic alignment, the recognition accuracy of 3D components can be improved. Attached Figure Description

[0013] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0014] Figure 1 This is a flowchart of a three-dimensional aircraft component localization method based on a semantic alignment model disclosed in this application; Figure 2 This is a schematic diagram of a semantic segmentation label for an aircraft component disclosed in this application; Figure 3 This is a schematic diagram of a specific context-enhanced view sampling disclosed in this application; Figure 4 This application discloses a specific semantic alignment result and a visualization diagram. Figure 5 This is a schematic diagram of specific multimodal data of an aircraft disclosed in this application; Figure 6 This is a schematic diagram of a three-dimensional aircraft component positioning device based on a semantic alignment model disclosed in this application; Figure 7 This is a structural diagram of an electronic device disclosed in this application. Detailed Implementation

[0015] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0016] In computational fluid dynamics post-processing, the identification and interaction of aircraft components (such as the nose and wings) are crucial. Existing methods rely on manual operation or predefined rules, resulting in inaccurate natural language semantic alignment; visual language models lack 3D geometric information, easily confusing similar components and significantly impacting the localization of 3D components. Therefore, this application will specifically introduce a 3D aircraft component localization method based on a semantic alignment model, which can solve the above problems.

[0017] See Figure 1 As shown in the figure, this application discloses a method for locating three-dimensional aircraft components based on a semantic alignment model, including: Step S11: Use a preset virtual camera to acquire multi-view images of each semantic component corresponding to the original 3D aircraft model, so as to obtain multi-view image data corresponding to each semantic component, and extract the geometric information corresponding to the semantic component.

[0018] In this embodiment, the step of using a preset virtual camera to acquire multi-view images of each semantic component corresponding to the original 3D aircraft model to obtain multi-view image data corresponding to each semantic component includes: performing component-level segmentation processing on the original 3D aircraft model to obtain each semantic component corresponding to the original 3D aircraft model; generating a number of view sampling points in the spherical space corresponding to each semantic component, and setting a preset virtual camera at each view sampling point; and using the preset virtual camera to acquire multi-view images of each semantic component to obtain corresponding multi-view image data.

[0019] That is, such as Figure 2 As shown, the original 3D aircraft model undergoes component-level segmentation, dividing the overall model into multiple semantic components, each corresponding to an independent data block. Then, based on the aforementioned semantic components, multiple view sampling points are generated in spherical space, and virtual cameras are set up at each sampling point to acquire multi-view images of the components, while simultaneously extracting the geometric information of the corresponding components.

[0020] The step of acquiring multi-view images of each semantic component using the preset virtual camera to obtain corresponding multi-view image data includes: acquiring a first view image and a second view image of each semantic component using the preset virtual camera; the first view image is an independent view image containing only the semantic component; the second view image is an enhanced view image containing the semantic component and its contextual information; and determining the multi-view image data of each semantic component based on the first view image and the second view image. That is, as... Figure 3 As shown, an independent view image containing only the target component, i.e., the first view image, is obtained, eliminating background interference. The first view image can guide the model to learn the pure geometric topological features of the component, avoiding interference from the color or shape of non-target components in the feature representation of the embedded space. An enhanced view image containing the target component and its contextual information, i.e., the second view image, is obtained, where the target component is highlighted. The second view image is used to provide relative spatial position information between the target component and other components, which can resolve visual symmetry ambiguities of topologically similar components. The geometric information of the component is extracted. In order to capture the spatial distribution features of the component in the 3D scene, the pipeline script of the visualization software automatically traverses each component and extracts the original bounding box coordinates of each component in real time.

[0021] Step S12: Based on the multi-view image data, the geometric information, and the text description information corresponding to each semantic component, construct a multimodal dataset corresponding to the original 3D aircraft model, and use the multimodal dataset to train the initial semantic alignment model to obtain the target localization model.

[0022] In this embodiment, constructing the multimodal dataset corresponding to the original 3D aircraft model based on the multi-view image data, the geometric information, and the text description information corresponding to each semantic component includes: performing image processing on the multi-view image data to obtain processed image data; the image processing includes image occlusion removal processing and image sharpening processing; generating text description information corresponding to the processed image data based on preset structured text prompts, and constructing a mapping relationship between the text description information and the multi-view image data; constructing the multimodal dataset corresponding to the original 3D aircraft model based on the multi-view image data, the geometric information, the text description information, and the mapping relationship. That is, generating corresponding text description information for the multi-view images, wherein the text description is generated using a preset structured description template to form image-text pairs of data with one-to-one correspondence between the image and the text. Finally, storing the image-text pairs of data and the corresponding geometric information to construct a multimodal dataset containing multi-view images, text descriptions, and geometric information. It should be noted that during the processing, when processing the multi-view images, occluded or unclear images are removed, and only clear and complete images are retained to ensure the generation of high-quality text descriptions. Two types of captured view images (independent view and context-enhanced view) are used as input, and text descriptions are generated for the images using pre-defined structured prompts. Images are then paired with their corresponding text descriptions to construct an image-text pair dataset.

[0023] In this embodiment, training the initial semantic alignment model using the multimodal dataset to obtain the target localization model includes: using a preset symmetric loss function and establishing a mapping relationship between image features and text features in the initial semantic alignment model based on the mapping relationship between the text description information and the multi-view image data to obtain a trained semantic alignment model; and training the trained semantic alignment model using the multi-view image data and the geometric information to obtain the target localization model. That is, the semantic alignment model is trained based on the image-text pair dataset to establish a semantic mapping relationship between image features and text features. Then, a geometric feature enhancement and multi-view feature fusion mechanism is introduced into the semantic alignment model to construct a unified semantic representation that integrates visual features, text features, and geometric features. The input image data is processed by a visual encoder to extract features, resulting in an image feature vector; the input text data is processed by a text encoder to extract features, resulting in a text feature vector; and the image feature vector and text feature vector are jointly trained based on a symmetric contrastive loss function to reduce the feature distance between matching samples and increase the feature distance between non-matching samples.

[0024] In this embodiment, training the trained semantic alignment model using the multi-view image data and the geometric information to obtain a target localization model includes: inputting the multi-view image data into a preset visual encoder to obtain corresponding visual feature vectors, and inputting the geometric information into a preset geometric encoder to obtain preset geometric feature vectors; mapping the visual feature vectors and the preset geometric feature vectors to a target feature space using a preset geometric feature mapping method to obtain initial visual features, and calculating the weights of the initial visual features using a preset attention mechanism to obtain corresponding weight coefficients; performing weighted fusion of each initial visual feature based on the weight coefficients to obtain target visual features; and training the trained semantic alignment model using the target visual features to obtain a target localization model. Specifically, multi-view images are input into the visual encoder to obtain multiple sets of corresponding visual feature vectors; then, the geometric information of the component is input into the geometric encoder to obtain geometric feature vectors. The geometric feature vectors are mapped to the same feature space as the visual feature vectors and used to spatially constrain the fusion process of visual features from each viewpoint. Weights are calculated for each viewpoint visual feature based on a preset attention mechanism to obtain weight coefficients corresponding to each viewpoint. The visual features from each perspective are weighted and fused according to the weight coefficients to obtain the final global visual semantic feature vector. This vector is then used to train the trained semantic alignment model to obtain the target localization model. In this way, by jointly optimizing the global visual semantic feature vector and the text feature vector, visual features, text features, and geometric features satisfy semantic consistency constraints in a unified feature space.

[0025] Step S13: Input the natural language description text of the target 3D component sent by the user terminal and the component information of each candidate 3D component into the target localization model to obtain the component semantic matching result, and complete the localization of the target 3D component based on the component semantic matching result.

[0026] In this embodiment, the step of inputting the natural language description text of the target 3D component sent by the user terminal and the component information of each candidate 3D component into the target localization model to obtain the component semantic matching result includes: acquiring the natural language description text of the target 3D component sent by the user terminal, and extracting features from the natural language description text using a preset text encoder to obtain a text feature vector; extracting features from the component information of each candidate 3D component to obtain a component feature vector, and inputting the text feature vector and the component feature vector into the target localization model to obtain the component semantic matching result. That is, based on the semantic alignment model, semantic matching is performed between the input natural language and the component to be matched to obtain a matching result. Finally, based on the matching result, a visualization operation is performed on the target component. Specifically, the input natural language is extracted using a text encoder to obtain a text feature vector. The multi-view images and geometric information of each candidate component are extracted using a visual encoder and a geometric encoder, respectively, to obtain the corresponding component feature vectors. Based on the semantic alignment model, the text feature vector and the feature vectors of each candidate component are mapped to a unified feature space. Calculate the similarity between the text feature vector and the feature vectors of each candidate component, and determine the target component that best matches the natural language based on the similarity.

[0027] As can be seen, in this embodiment, a preset virtual camera is used to acquire multi-view images of each semantic component corresponding to the original 3D aircraft model to obtain multi-view image data corresponding to each semantic component, and the geometric information corresponding to the semantic component is extracted. Based on the multi-view image data, the geometric information, and the text description information corresponding to each semantic component, a multimodal dataset corresponding to the original 3D aircraft model is constructed, and the initial semantic alignment model is trained using the multimodal dataset to obtain a target localization model. The natural language description text for the target 3D component sent by the user terminal and the component information of each candidate 3D component are input into the target localization model to obtain the component semantic matching result, and the localization of the target 3D component is completed based on the component semantic matching result. That is, by introducing the geometric information of the semantic component to train the semantic alignment model, the distinguishability between low-texture and topologically similar 3D components is improved, thereby achieving accurate semantic alignment between natural language and the target component, and obtaining a more accurate 3D component localization model. In this way, during use, by improving the accuracy of semantic alignment, the recognition accuracy of 3D components can be improved.

[0028] To further verify the effectiveness of the method proposed in this application, based on the method in the above embodiments, typical components in the 3D model of an aircraft were selected as test objects to conduct semantic alignment experiments. Comparison methods included existing visual-language alignment models (CLIP model, Contrastive Language-Image Pre-training), alignment models trained solely on image and text data, and the method proposed in this application. The test results are shown in Table 1 below.

[0029] Table 1. Results of Semantic Alignment Experiment

[0030] like Figure 4 As shown, the upper and lower groups represent the semantic alignment process of components under different natural language input conditions. In each group, from left to right: loading 3D data using the input natural language command; identifying the target component based on the semantic alignment model; and the result image after performing visualization operations on the target component. In the upper example, the input natural language is "extract the corresponding component of the aircraft's main lift surface." The model performs semantic matching on the aircraft's main wing and displays the target component in red in the final result. In the lower example, the input natural language is "extract the aircraft's vertical tail fin." The model accurately locates the corresponding component and marks it in blue in the final result.

[0031] In practical operation, taking DLR-F12 aircraft data as an example, the operation process of this solution is as follows: First, the original overall aircraft model data is divided into preset areas, such as... Figure 2 Using ParaView software, the aircraft was manually divided into five component regions by selecting and extracting cells. Data collection then commenced. First, the bounding box coordinates of each component and the overall aircraft were collected. Then, based on Fibonacci spherical sampling, the geometric center of the bounding box of each component was used as the center of the sphere, with a radius 2-3 times the radius of the circumscribed sphere of that component's bounding box. Sixty-four camera positions were set on this sphere to collect images with and without surrounding components. Figure 3 The target component, along with surrounding parts, is highlighted in red during rendering. Finally, the collected images are processed using a multimodal large model to remove severely occluded or blurry images. The processed images are then input into the multimodal large model for further processing. The large model is required to output a text description of the corresponding image according to a preset output template, such as: [3D ParaView rendering of a gray fuselage with a red main wing viewed from the front.] Figure 5 This yields multimodal data containing image-text pairs and bounding box information.

[0032] In the model training process, the first stage uses only collected image-text pairs to train the semantic alignment model. A symmetric loss function based on contrastive learning is employed for training. By maximizing the similarity between image and text embeddings and minimizing their distance, the semantic consistency of image-text pairing is promoted, allowing the model to master the representation of images in this domain. In the second stage, the model obtained in the first stage is trained. First, the collected geometric information vectors are encoded and adaptively fused (trainable) into the image encoding vector. Then, based on an attention mechanism, different weights (trainable) are assigned to images from different viewpoints for multi-view image feature fusion. Finally, the resulting fused vector is trained with contrastive loss against the text vector, and the geometric vector is also trained with contrastive loss against the text vector. The final loss function is obtained by weighting both. By jointly optimizing the fused representation and the pure geometric representation, the model is forced to establish a decoupled physical space mapping in the latent space. This ensures that when visual features fail due to occlusion or extreme viewpoints, this independent geometric discriminative power can serve as a "logical foundation" to maintain the accuracy of semantic alignment.

[0033] Finally, in the semantic alignment stage, the previously pre-segmented DLR-F12 aircraft data is read first. A preset function automatically collects multi-view images and bounding box information of each component of the aircraft, and stores the bounding box information and images of each component and the whole aircraft in a folder named after the component ID, just like the data collected in the multimodal data collection stage. Then, the user inputs specific natural language descriptions such as "main wing" and "horizontal tail". The model encodes the input text and performs vector comparison matching with the previously collected multimodal data (images + bounding box information) in the folder. The image and bounding box information of the component with the highest matching score are matched, and the corresponding component ID is output for subsequent operations to complete the semantic alignment.

[0034] refer to Figure 6 The present application also discloses a three-dimensional aircraft component positioning device based on a semantic alignment model, comprising: The data extraction module 11 is used to acquire multi-view images of each semantic component corresponding to the original 3D aircraft model using a preset virtual camera, so as to obtain multi-view image data corresponding to each semantic component, and extract the geometric information corresponding to the semantic component; wherein, the step of acquiring multi-view images of each semantic component corresponding to the original 3D aircraft model using a preset virtual camera to obtain multi-view image data corresponding to each semantic component includes: performing component-level segmentation processing on the original 3D aircraft model to obtain each semantic component corresponding to the original 3D aircraft model; generating a number of view sampling points in the spherical space corresponding to each semantic component, and setting a preset virtual camera at each view sampling point; and acquiring multi-view images of each semantic component using the preset virtual camera to obtain the corresponding multi-view image data. The step of acquiring multi-view images of each semantic component using the preset virtual camera to obtain corresponding multi-view image data includes: acquiring a first view image and a second view image of each semantic component using the preset virtual camera; the first view image is an independent view image containing only the semantic component; the second view image is an enhanced view image containing the semantic component and its contextual information; and determining multi-view image data of each semantic component based on the first view image and the second view image. The model training module 12 is used to construct a multimodal dataset corresponding to the original 3D aircraft model based on the multi-view image data, the geometric information, and the text description information corresponding to each semantic component, and to train the initial semantic alignment model using the multimodal dataset to obtain the target localization model. The component localization module 13 is used to input the natural language description text of the target three-dimensional component sent by the user terminal and the component information of each candidate three-dimensional component into the target localization model to obtain the component semantic matching result, and to complete the localization of the target three-dimensional component based on the component semantic matching result.

[0035] In this embodiment, the semantic alignment model is trained by incorporating geometric information of semantic components, improving the distinguishability between low-texture and topologically similar 3D components. This achieves accurate semantic alignment between natural language and target components, resulting in a more precise localization model for 3D components. Consequently, by improving the accuracy of semantic alignment during use, the recognition accuracy of 3D components can be enhanced.

[0036] In some specific embodiments, the model training module 12 may specifically include: An image processing unit is used to perform image processing on the multi-view image data to obtain processed image data; the image processing includes image occlusion removal processing and image sharpening processing; The mapping relationship construction unit is used to generate text description information corresponding to the processed image data based on the preset structured text prompts, and to construct the mapping relationship between the text description information and the multi-view image data; The dataset construction unit is used to construct a multimodal dataset corresponding to the original 3D aircraft model based on the multi-view image data, the geometric information, the text description information, and the mapping relationship.

[0037] In some specific embodiments, the model training module 12 may specifically include: The semantic alignment model training unit is used to establish the mapping relationship between image features and text features in the initial semantic alignment model based on the mapping relationship between the text description information and the multi-view image data using a preset symmetric loss function, so as to obtain the trained semantic alignment model. The target localization model determination submodule is used to train the trained semantic alignment model using the multi-view image data and the geometric information to obtain the target localization model.

[0038] In some specific embodiments, the target localization model determination submodule may specifically include: The vector editing unit is used to input the multi-view image data into a preset visual encoder to obtain the corresponding visual feature vector, and to input the geometric information into a preset geometric encoder to obtain a preset geometric feature vector; The weight coefficient determination unit is used to map the visual feature vector and the preset geometric feature vector to the target feature space using a preset geometric feature mapping method to obtain initial visual features, and to calculate the weights of the initial visual features using a preset attention mechanism to obtain the corresponding weight coefficients. The feature fusion unit is used to perform weighted fusion of each of the initial visual features based on the weight coefficients to obtain the target visual features; The model determination unit is used to train the trained semantic alignment model using the target visual features to obtain the target localization model.

[0039] In some specific embodiments, the component positioning module 13 may specifically include: The text acquisition unit is used to acquire the natural language description text of the target three-dimensional component sent by the user terminal, and to extract features from the natural language description text using a preset text encoder to obtain a text feature vector. The feature matching unit is used to extract features from the component information of each candidate 3D component to obtain component feature vectors, and input the text feature vectors and component feature vectors into the target localization model to obtain component semantic matching results.

[0040] Furthermore, embodiments of this application also disclose an electronic device, Figure 7 This is a structural diagram of an electronic device 20 according to an exemplary embodiment. The content of the diagram should not be construed as limiting the scope of this application.

[0041] Figure 7 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. Specifically, the electronic device 20 may include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the semantic alignment model-based 3D aircraft component localization method disclosed in any of the foregoing embodiments. Alternatively, the electronic device 20 in this embodiment may specifically be an electronic computer.

[0042] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0043] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon can include operating system 221, computer program 222, etc., and the storage method can be temporary storage or permanent storage.

[0044] The operating system 221 is used to manage and control the various hardware devices on the electronic device 20 and the computer program 222, which may be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the semantic alignment model-based three-dimensional aircraft component localization method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0045] Furthermore, this application also discloses a computer-readable storage medium for storing a computer program; wherein, when the computer program is executed by a processor, it implements the aforementioned disclosed method for 3D aircraft component localization based on a semantic alignment model. Specific steps of this method can be found in the corresponding content disclosed in the foregoing embodiments, and will not be repeated here.

[0046] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0047] Those skilled in the art will further recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of both. To clearly illustrate the interchangeability of hardware and software, the components and steps of the various examples have been generally described in terms of functionality in the foregoing description. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0048] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0049] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0050] The technical solutions provided in this application have been described in detail above. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for locating three-dimensional aircraft components based on a semantic alignment model, characterized in that, include: A preset virtual camera is used to acquire multi-view images of each semantic component corresponding to the original 3D aircraft model to obtain multi-view image data corresponding to each semantic component, and to extract the geometric information corresponding to the semantic component. The process of acquiring multi-view images of each semantic component corresponding to the original 3D aircraft model using a preset virtual camera includes: performing component-level segmentation on the original 3D aircraft model to obtain each semantic component corresponding to the original 3D aircraft model; generating several view sampling points in the spherical space corresponding to each semantic component, and setting a preset virtual camera at each view sampling point; and acquiring multi-view images of each semantic component using the preset virtual camera to obtain the corresponding multi-view image data. The step of acquiring multi-view images of each semantic component using the preset virtual camera to obtain corresponding multi-view image data includes: acquiring a first view image and a second view image of each semantic component using the preset virtual camera; the first view image is an independent view image containing only the semantic component; the second view image is an enhanced view image containing the semantic component and its contextual information; and determining multi-view image data of each semantic component based on the first view image and the second view image. Based on the multi-view image data, the geometric information, and the text description information corresponding to each semantic component, a multimodal dataset corresponding to the original 3D aircraft model is constructed, and the initial semantic alignment model is trained using the multimodal dataset to obtain the target localization model. The step of training the initial semantic alignment model using the multimodal dataset to obtain the target localization model includes: using a preset symmetric loss function and establishing a mapping relationship between image features and text features in the initial semantic alignment model based on the mapping relationship between the text description information and the multi-view image data to obtain the trained semantic alignment model; and training the trained semantic alignment model using the multi-view image data and the geometric information to obtain the target localization model. The step of training the post-trained semantic alignment model using the multi-view image data and the geometric information to obtain a target localization model includes: inputting the multi-view image data into a preset visual encoder to obtain corresponding visual feature vectors, and inputting the geometric information into a preset geometric encoder to obtain preset geometric feature vectors; mapping the visual feature vectors and the preset geometric feature vectors to a target feature space using a preset geometric feature mapping method to obtain initial visual features, and calculating the weights of the initial visual features using a preset attention mechanism to obtain corresponding weight coefficients; performing weighted fusion of each initial visual feature based on the weight coefficients to obtain target visual features; and training the post-trained semantic alignment model using the target visual features to obtain a target localization model. The natural language description text of the target 3D component sent by the user and the component information of each candidate 3D component are input into the target localization model to obtain the component semantic matching result, and the localization of the target 3D component is completed based on the component semantic matching result.

2. The three-dimensional aircraft component localization method based on a semantic alignment model according to claim 1, characterized in that, The construction of the multimodal dataset corresponding to the original 3D aircraft model based on the multi-view image data, the geometric information, and the text description information corresponding to each semantic component includes: The multi-view image data is subjected to image processing to obtain processed image data; the image processing includes image occlusion removal processing and image sharpening processing. Based on preset structured text prompts, text description information corresponding to the processed image data is generated, and a mapping relationship between the text description information and the multi-view image data is constructed. Based on the multi-view image data, the geometric information, the text description information, and the mapping relationship, a multimodal dataset corresponding to the original 3D aircraft model is constructed.

3. The three-dimensional aircraft component localization method based on a semantic alignment model according to claim 1, characterized in that, The step of inputting the natural language description text of the target 3D component sent by the user terminal and the component information of each candidate 3D component into the target localization model to obtain the component semantic matching result includes: The system acquires natural language description text for the target 3D component sent by the user terminal, and uses a preset text encoder to extract features from the natural language description text to obtain a text feature vector. Feature extraction is performed on the component information of each candidate 3D component to obtain component feature vectors. The text feature vectors and component feature vectors are then input into the target localization model to obtain component semantic matching results.

4. A three-dimensional aircraft component positioning device based on a semantic alignment model, characterized in that, include: The data extraction module is used to acquire multi-view images of each semantic component corresponding to the original 3D aircraft model using a preset virtual camera, so as to obtain multi-view image data corresponding to each semantic component, and extract the geometric information corresponding to the semantic component; wherein, the step of acquiring multi-view images of each semantic component corresponding to the original 3D aircraft model using a preset virtual camera to obtain multi-view image data corresponding to each semantic component includes: performing component-level segmentation processing on the original 3D aircraft model to obtain each semantic component corresponding to the original 3D aircraft model; generating a number of view sampling points in the spherical space corresponding to each semantic component, and setting a preset virtual camera at each view sampling point; and acquiring multi-view images of each semantic component using the preset virtual camera to obtain the corresponding multi-view image data. The step of acquiring multi-view images of each semantic component using the preset virtual camera to obtain corresponding multi-view image data includes: acquiring a first view image and a second view image of each semantic component using the preset virtual camera; the first view image is an independent view image containing only the semantic component; the second view image is an enhanced view image containing the semantic component and its contextual information; and determining multi-view image data of each semantic component based on the first view image and the second view image. The model training module is used to construct a multimodal dataset corresponding to the original 3D aircraft model based on the multi-view image data, the geometric information, and the text description information corresponding to each semantic component, and to train the initial semantic alignment model using the multimodal dataset to obtain the target localization model. The step of training the initial semantic alignment model using the multimodal dataset to obtain the target localization model includes: using a preset symmetric loss function and establishing a mapping relationship between image features and text features in the initial semantic alignment model based on the mapping relationship between the text description information and the multi-view image data to obtain the trained semantic alignment model; and training the trained semantic alignment model using the multi-view image data and the geometric information to obtain the target localization model. The step of training the post-trained semantic alignment model using the multi-view image data and the geometric information to obtain a target localization model includes: inputting the multi-view image data into a preset visual encoder to obtain corresponding visual feature vectors, and inputting the geometric information into a preset geometric encoder to obtain preset geometric feature vectors; mapping the visual feature vectors and the preset geometric feature vectors to a target feature space using a preset geometric feature mapping method to obtain initial visual features, and calculating the weights of the initial visual features using a preset attention mechanism to obtain corresponding weight coefficients; performing weighted fusion of each initial visual feature based on the weight coefficients to obtain target visual features; and training the post-trained semantic alignment model using the target visual features to obtain a target localization model. The component localization module is used to input the natural language description text of the target 3D component sent by the user terminal and the component information of each candidate 3D component into the target localization model to obtain the component semantic matching result, and to complete the localization of the target 3D component based on the component semantic matching result.

5. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the three-dimensional aircraft component localization method based on a semantic alignment model as described in any one of claims 1 to 3.

6. A computer-readable storage medium, characterized in that, Used to store a computer program, which, when executed by a processor, implements the three-dimensional aircraft component localization method based on a semantic alignment model as described in any one of claims 1 to 3.

Images