Mechanical picture data set generation method based on fusion attention and dynamic convolution

Abstract

The invention discloses a mechanical picture data set generation method based on fusion attention and dynamic convolution, and relates to the technical field of large models, and the method comprises the steps: obtaining at least one to-be-recognized mechanical image, inputting all to-be-recognized mechanical images into a large model processing module, and carrying out the recognition of at least one to-be-recognized mechanical image; generating a mechanical type and annotation information corresponding to each to-be-identified mechanical image; associating the mechanical type, the annotation information and the corresponding to-be-identified mechanical image to generate a group of data information, and integrating all the groups of data information to generate a mechanical image data set; the large model processing module comprises a multi-modal fusion layer, a dynamic attention mechanism layer and a space-time enhancement network module fusing attention and dynamic convolution. According to the invention, the mechanical picture data set can be automatically extracted, sorted and marked from the original mechanical image through the large model processing module, and the efficiency and accuracy of data set generation are improved.

Description

Background Art

[0002] The generation of existing machine image datasets mostly relies on manual sorting and annotation, which is not only inefficient but also prone to errors. In addition, machine image data are often heterogeneous, massive, and subjective, making the generation of datasets more complicated. Summary of the invention

[0003] The technical problem to be solved by the present invention is to address the deficiencies of the prior art, and specifically provide a method for generating a mechanical image dataset based on fusion attention and dynamic convolution, as follows:

[0004] 1) In the first aspect, the present invention provides a method for generating a mechanical image dataset based on fusion attention and dynamic convolution. The specific technical solution is as follows:

[0005] Acquire at least one machine image to be identified, input all the machine images to be identified into a large model processing module, and generate a machine type and annotation information corresponding to each machine image to be identified;

[0006] Associating the machine type, the annotation information, and the corresponding machine image to be identified to generate a set of data information, and integrating all the sets of data information to generate a machine image data set;

[0007] The large model processing module includes a multimodal fusion layer, a dynamic attention mechanism layer, and a spatiotemporal enhancement network module that integrates attention and dynamic convolution.

[0008] The beneficial effects of the method for generating a mechanical image dataset based on fusion attention and dynamic convolution provided by the present invention are as follows:

[0009] The large model processing module can automatically extract, organize and annotate mechanical image datasets from original mechanical images, thereby improving the efficiency and accuracy of dataset generation.

[0010] Based on the above solution, the present invention can also be improved as follows.

[0011] Furthermore, the multimodal fusion layer is used to fuse text data and image data in any machine image to be identified to obtain a fused first feature.

[0012] Furthermore, the dynamic attention mechanism layer is used to output a maximum attention score matrix, and the maximum attention score matrix is: an attention score matrix with the largest value among the attention score matrices corresponding to all first features.

[0013] Furthermore, the spatiotemporal enhancement network module that integrates attention and dynamic convolution is used to fuse the output results of the spatiotemporal attention mechanism and the output results of the dynamic convolution to obtain the machine type and annotation information.

[0014] 2) In the second aspect, the present invention also provides a mechanical image dataset generation system based on fusion attention and dynamic convolution, and the specific technical solution is as follows:

[0015] The acquisition unit is used to: acquire at least one machine image to be identified, input all the machine images to be identified into the large model processing module, and generate the machine type and annotation information corresponding to each machine image to be identified;

[0016] The generating unit is used to: associate the machine type, the annotation information and the corresponding machine image to be identified to generate a set of data information, and integrate all the sets of data information to generate a machine image data set;

[0017] The large model processing module includes a multimodal fusion layer, a dynamic attention mechanism layer, and a spatiotemporal enhancement network module that integrates attention and dynamic convolution.

[0018] Based on the above solution, the present invention can also be improved as follows.

[0019] Furthermore, the multimodal fusion layer is used to fuse text data and image data in any machine image to be identified to obtain a fused first feature.

[0020] Furthermore, the dynamic attention mechanism layer is used to output a maximum attention score matrix, and the maximum attention score matrix is: an attention score matrix with the largest value among the attention score matrices corresponding to all first features.

[0021] Furthermore, the spatiotemporal enhancement network module that integrates attention and dynamic convolution is used to fuse the output results of the spatiotemporal attention mechanism and the output results of the dynamic convolution to obtain the machine type and annotation information.

[0022] 3) In a third aspect, the present invention further provides an electronic device, comprising a processor, wherein the processor is coupled to a memory, wherein at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements any of the above methods.

[0023] 4) In a fourth aspect, the present invention further provides a computer-readable storage medium, wherein at least one computer program is stored in the computer-readable storage medium, and the at least one computer program is loaded and executed by a processor so that a computer implements any of the above methods.

[0024] It should be noted that the beneficial effects achieved by the technical solutions of the second to fourth aspects of the present invention and the corresponding possible implementation methods can be found in the above-mentioned technical effects of the first aspect and its corresponding possible implementation methods, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings:

[0026] Figure 1 A schematic diagram of a flow chart of a method for generating a mechanical image dataset based on fusion of attention and dynamic convolution according to an embodiment of the present invention;

[0027] Figure 2 The present invention is a structural framework diagram of an electronic device. DETAILED DESCRIPTION

[0028] In order to make the objectives, technical solutions and advantages of the present invention more clear, the embodiments of the present invention will be further described in detail below with reference to the accompanying drawings.

[0029] like Figure 1 As shown, the method for generating a mechanical image dataset based on fusion attention and dynamic convolution in an embodiment of the present invention includes the following steps:

[0030] S1, obtaining at least one image of a machine to be identified, inputting all the images of the machine to be identified into a large model processing module, and generating a machine type and annotation information corresponding to each image of the machine to be identified;

[0031] S2, associating the machine type, the annotation information and the corresponding machine image to be identified to generate a set of data information, and integrating all the sets of data information to generate a machine image data set;

[0032] The large model processing module includes a multimodal fusion layer, a dynamic attention mechanism layer, and a spatiotemporal enhancement network module that integrates attention and dynamic convolution.

[0033] The beneficial effects of the method for generating a mechanical image dataset based on fusion attention and dynamic convolution provided by the present invention are as follows:

[0034] The large model processing module can automatically extract, organize and annotate mechanical image datasets from original mechanical images, thereby improving the efficiency and accuracy of dataset generation.

[0035] The machine image to be identified is an image taken from any angle of any machine equipment. After the image is input into the large model processing module, the specific name of the machine equipment in the image (machine type) and the corresponding annotation information for the image (such as popular science information about the machine equipment or whether the machine has defects or damage, etc.) can be output. The above content is to provide an effective data set for the subsequent semantic training of other large models outside this solution, ensuring the accuracy of other large model training.

[0036] The acquisition of the machine image to be identified is usually carried out through a camera. Before the acquisition of the machine image to be identified, the acquisition device needs to be debugged and configured, that is, before obtaining the machine image to be identified, the acquisition device is auxiliary debugged according to the test device. The acquisition device first acquires the image of the test device, and adjusts the parameters of the acquisition device according to the acquired image, that is, the configuration process of the sensitivity is as follows:

[0037] For the experimental area, collect multiple image data under the current configuration, calculate the clarity of each image data, determine the proportion of all clarity values ​​that exceed the threshold clarity, and when the proportion value is higher than the preset proportion value, output the sensitivity configuration result in the current configuration as the final result.

[0038] If the ratio is lower than the preset ratio, the contour of the target object in each image data is extracted, where the target object can be the contour of a mechanical device or a part. The extracted contour is compared with the manually marked contour to determine the target similarity between the two, determine the target difference between the target similarity and the preset similarity, and determine the target fine-tuning amount corresponding to the target difference in the corresponding relationship between the difference and the fine-tuning amount, and fine-tune the sensitivity under the current configuration based on the target fine-tuning amount. The target difference includes both positive and negative numbers.

[0039] Furthermore, the multimodal fusion layer is used to fuse text data and image data in any machine image to be identified to obtain a fused first feature.

[0040] Furthermore, the dynamic attention mechanism layer is used to output a maximum attention score matrix, and the maximum attention score matrix is: an attention score matrix with the largest value among the attention score matrices corresponding to all first features.

[0041] Furthermore, the spatiotemporal enhancement network module that integrates attention and dynamic convolution is used to fuse the output results of the spatiotemporal attention mechanism and the output results of the dynamic convolution to obtain the machine type and annotation information.

[0042] Embodiment 1, data preprocessing module: This module is responsible for receiving the original mechanical pictures and performing preprocessing operations such as data cleaning and format conversion to ensure the accuracy and consistency of the data.

[0043] The cleaning process can be:

[0044] Input the original machine image into the preset coarse-grained binary classifier, filter out the first image that meets the requirements, and then input the first image into the preset fine-grained binary classifier to determine the confidence of the category recognition result (if the image is too unclear to be recognized as a machine image, it will cause serious deviations in the construction of subsequent data sets). Determine in real time whether the first image needs to be manually cleaned, and if the judgment result is yes, perform manual cleaning in a timely manner.

[0045] The specific process of determining whether manual cleaning is required is as follows:

[0046] The contour in the first image is extracted, and the extraction result is fitted to obtain the line version fitting contour. The line version fitting contour is identified, and the similarity between the keyword corresponding to the identification result and the original category of the image is determined, that is, the keyword and the text of the original category are vectorized, and the cosine similarity between the two vectors is calculated. When the similarity exceeds the preset similarity, it is determined that manual cleaning is not required. If the similarity is lower than the preset similarity, it is determined that manual cleaning is required.

[0047] Big model processing module: This module uses a pre-trained big model to process the pre-processed data. The big model can automatically understand the semantics of medical data and extract and annotate data according to the characteristics of the specialty field. The specialty field refers to the characteristics of mechanical images, which contain the relative position information of many mechanical parts. Data extraction refers to the big model extracting question-answer knowledge pairs from the text based on the input image and text data, and generating question-answer knowledge pairs for the content in the image.

[0048] The model structure of the large model processing module is as follows:

[0049] 1. Multimodal fusion layer

[0050] The calculation formula is:

[0051] F=Concat(Ftext,Fimage)*Wfuse+bfuse

[0052] Where: Ftext is the feature representation of text data. Fimage is the feature representation of image data. Concat is a concatenation operation that concatenates feature representations of different modalities. Wfuse is the weight matrix of the fusion layer. bfuse is the bias term of the fusion layer. F is the fused feature representation.

[0053] 2. Dynamic Attention Mechanism Layer

[0054] The calculation formula is:

[0055]

[0056] A=softmax(E)

[0057] Where: Q is the query matrix, K is the key matrix, b is the bias matrix, W Q , W K are the linear transformation weights of the query and key matrices respectively. k is the dimension of the key, which is manually set to 64, E is the attention score matrix, and A is the maximum value in the attention score matrix, that is, the maximum attention score matrix.

[0058] 3. Spatiotemporal Enhanced Network Module Combining Attention and Dynamic Convolution

[0059] 3.1. Spatial-temporal attention mechanism

[0060] The spatiotemporal attention mechanism is used to calculate the importance weight of each spatiotemporal position, thereby enhancing the representation of key features and suppressing irrelevant information. The calculation formula is:

[0061] Con=Concat(A,AvgPool3D(A),MaxPool3D(A))

[0062] Conv_σ=σ(Conv3D(Con))

[0063] Att=Conv_σ*A

[0064] Where A represents the dynamic attention mechanism layer. AvgPool3D and MaxPool3D represent the three-dimensional average pooling and three-dimensional maximum pooling operations, respectively, which are used to extract global context information. Concat represents the concatenation operation, which concatenates the input data, the average pooling result, and the maximum pooling result. Conv3D represents the three-dimensional convolution operation, which is used to fuse the concatenated features and calculate the attention weight. σ represents the Sigmoid activation function. * represents the element-by-element multiplication operation, which multiplies the attention weight with the input data to achieve feature enhancement.

[0065] 3.2 Dynamic Convolution

[0066] Dynamic convolution dynamically adjusts the parameters of the convolution kernel according to the characteristics of the input data. The calculation formula is:

[0067] ML=MLP(Att)

[0068] Y = DConv(Att,ML)

[0069] Among them, Att represents the output of the spatiotemporal attention mechanism, MLP represents the multi-layer perceptron model, DConv represents the dynamic convolution operation, the convolution kernel parameters calculated by MLP perform convolution operation on the input data Att, and Y represents the output result of dynamic convolution.

[0070] 4. Integration strategy

[0071] The output of the spatiotemporal attention layer is fused with the output of the dynamic convolution layer. The calculation formula is:

[0072] Z=Fusion(Att,Y)=αAtt+(1-α)Y

[0073] Where A represents the output of the spatiotemporal attention layer, and Y represents the output of the dynamic convolution layer. α is a hyperparameter used to balance the output of the spatiotemporal attention layer and the dynamic convolution layer. Z represents the fused representation output.

[0074] Specialty Dataset Generation Module: This module generates a mechanical image dataset in a specific specialty field based on the output results of the large model processing module. The dataset includes multiple data types such as numerical, text, and images, and is accompanied by detailed annotation information.

[0075] In the above embodiments, although the steps are numbered S1, S2, etc., these are only specific embodiments given by the present invention. Those skilled in the art may adjust the execution order of S1, S2, etc. according to actual conditions, which is also within the protection scope of the present invention. It can be understood that in some embodiments, some or all of the above embodiments may be included.

[0076] This solution also provides a mechanical image dataset generation system based on fusion attention and dynamic convolution. The specific technical solution is as follows:

[0077] The acquisition unit is used to: acquire at least one machine image to be identified, input all the machine images to be identified into the large model processing module, and generate the machine type and annotation information corresponding to each machine image to be identified;

[0078] The generating unit is used to: associate the machine type, the annotation information and the corresponding machine image to be identified to generate a set of data information, and integrate all the sets of data information to generate a machine image data set;

[0079] The large model processing module includes a multimodal fusion layer, a dynamic attention mechanism layer, and a spatiotemporal enhancement network module that integrates attention and dynamic convolution.

[0080] Based on the above solution, the present invention can also be improved as follows.

[0081] Furthermore, the multimodal fusion layer is used to fuse text data and image data in any machine image to be identified to obtain a fused first feature.

[0082] Furthermore, the dynamic attention mechanism layer is used to output a maximum attention score matrix, and the maximum attention score matrix is: an attention score matrix with the largest value among the attention score matrices corresponding to all first features.

[0083] Furthermore, the spatiotemporal enhancement network module that integrates attention and dynamic convolution is used to fuse the output results of the spatiotemporal attention mechanism and the output results of the dynamic convolution to obtain the machine type and annotation information.

[0084] It should be noted that the beneficial effects of the mechanical picture dataset generation system based on fusion of attention and dynamic convolution provided by the above embodiment are the same as the beneficial effects of the mechanical picture dataset generation method based on fusion of attention and dynamic convolution, which will not be repeated here. In addition, when the system provided by the above embodiment realizes its functions, it only takes the division of the above functional modules as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the system is divided into different functional modules according to actual conditions to complete all or part of the functions described above. In addition, the system and method embodiments provided by the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be repeated here.

[0085] like Figure 2 As shown, an electronic device 300 according to an embodiment of the present invention includes a processor 320, the processor 320 is coupled to a memory 310, and the memory 310 stores at least one computer program 330, and the at least one computer program 330 is loaded and executed by the processor 320, so that the electronic device 300 implements any of the above methods, specifically:

[0086] The electronic device 300 may have relatively large differences due to different configurations or performances, and may include one or more processors 320 (Central Processing Units, CPU) and one or more memories 310, wherein the one or more memories 310 store at least one computer program 330, and the at least one computer program 330 is loaded and executed by the one or more processors 320, so that the electronic device 300 implements a method for generating a mechanical image data set based on fusion attention and dynamic convolution provided in the above embodiment. Of course, the electronic device 300 may also have components such as a wired or wireless network interface, a keyboard, and an input and output interface for input and output. The electronic device 300 may also include other components for realizing device functions, which will not be repeated here.

[0087] A computer-readable storage medium according to an embodiment of the present invention stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements any one of the above methods.

[0088] Optionally, the computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a compact disc (CD-ROM), a magnetic tape, a floppy disk, an optical data storage device, and the like.

[0089] In an exemplary embodiment, a computer program product or a computer program is also provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of an electronic device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the electronic device performs any of the above methods.

[0090] It should be noted that the terms "first", "second", etc. in the specification and claims of the present application are used to distinguish similar objects and represent the definition of a specific order or sequence. The order of use of similar objects can be interchanged where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than the order shown or described.

[0091] Those skilled in the art know that the present invention can be implemented as a system, method or computer program product. Therefore, the present disclosure can be specifically implemented in the following forms, namely: it can be complete hardware, it can be complete software (including firmware, resident software, microcode, etc.), or it can be a combination of hardware and software, which is generally referred to as "circuit", "module" or "system" herein. In addition, in some embodiments, the present invention can also be implemented in the form of a computer program product in one or more computer-readable media, and the computer-readable medium contains computer-readable program code.

[0092] Any combination of one or more computer-readable media may be used. A computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any combination thereof. More specific examples (a non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In this document, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in conjunction with an instruction execution system, device, or device.

[0093] Although the embodiments of the present invention have been shown and described above, it is to be understood that the above embodiments are exemplary and are not to be construed as limitations of the present invention. A person skilled in the art may change, modify, replace and vary the above embodiments within the scope of the present invention.

Claims

1. A method for generating a mechanical image dataset based on fusion attention and dynamic convolution, characterized in that: include: Acquire at least one machine image to be identified, input all the machine images to be identified into a large model processing module, and generate a machine type and annotation information corresponding to each machine image to be identified; Associating the machine type, the annotation information, and the corresponding machine image to be identified to generate a set of data information, and integrating all the sets of data information to generate a machine image data set; The large model processing module includes a multimodal fusion layer, a dynamic attention mechanism layer, and a spatiotemporal enhancement network module that integrates attention and dynamic convolution.

2. According to claim 1, a method for generating a mechanical image dataset based on fusion attention and dynamic convolution is characterized in that: The multimodal fusion layer is used to fuse text data and image data in any machine image to be identified to obtain a fused first feature.

3. According to claim 1, a method for generating a mechanical image dataset based on fusion attention and dynamic convolution is characterized in that: The dynamic attention mechanism layer is used to output a maximum attention score matrix, and the maximum attention score matrix is: an attention score matrix with the largest value among the attention score matrices corresponding to all first features.

4. According to claim 1, a method for generating a mechanical image dataset based on fusion attention and dynamic convolution is characterized in that: The spatiotemporal enhancement network module that integrates attention and dynamic convolution is used to fuse the output results of the spatiotemporal attention mechanism and the output results of the dynamic convolution to obtain the machine type and annotation information.

5. A mechanical image dataset generation system based on fusion attention and dynamic convolution, characterized in that: include: The acquisition unit is used to: acquire at least one machine image to be identified, input all the machine images to be identified into the large model processing module, and generate the machine type and annotation information corresponding to each machine image to be identified; The generating unit is used to: associate the machine type, the annotation information and the corresponding machine image to be identified to generate a set of data information, and integrate all the sets of data information to generate a machine image data set; The large model processing module includes a multimodal fusion layer, a dynamic attention mechanism layer, and a spatiotemporal enhancement network module that integrates attention and dynamic convolution.

6. The system for generating mechanical image dataset based on fusion of attention and dynamic convolution according to claim 5, characterized in that: The multimodal fusion layer is used to fuse text data and image data in any machine image to be identified to obtain a fused first feature.

7. The system for generating mechanical image dataset based on fusion attention and dynamic convolution according to claim 5, characterized in that: The dynamic attention mechanism layer is used to output a maximum attention score matrix, and the maximum attention score matrix is: an attention score matrix with the largest value among the attention score matrices corresponding to all first features.

8. The system for generating mechanical image dataset based on fusion attention and dynamic convolution according to claim 5, characterized in that: The spatiotemporal enhancement network module that integrates attention and dynamic convolution is used to fuse the output results of the spatiotemporal attention mechanism and the output results of the dynamic convolution to obtain the machine type and annotation information.

9. An electronic device, characterized in that: The electronic device comprises a processor, the processor is coupled to a memory, at least one computer program is stored in the memory, and the at least one computer program is loaded and executed by the processor so that the electronic device implements the method according to any one of claims 1 to 4.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores at least one computer program, and the at least one computer program is loaded and executed by a processor so that a computer implements the method according to any one of claims 1 to 4.

Images