Road disease detection method and device based on multi-modal large model

Abstract

The invention provides a road disease detection method and device based on a multi-modal large model. The method comprises the following steps: acquiring image information of a road and text information about disease types in the image information; outputting the text information to a semantic recognition model to determine semantic information; and inputting the image information and the semantic information into a multi-modal large model, and determining the disease information of the road, so that the road disease detection can be realized based on the multi-modal information, and the detection precision can be improved.

Description

Technical Field

[0001] The present application belongs to the technical field of road damage detection, and in particular, relates to a road damage detection method and device based on a multi-modal large model. Background Art

[0002] In urban planning and infrastructure construction, roads are an indispensable key component, and their condition has a decisive impact on the efficiency of urban operations and the quality of life of residents. However, affected by multiple factors such as the passage of time, environmental factors, traffic load, and material aging, the road structure will gradually show cracks, potholes, settlement and other defects. These defects not only threaten driving safety and reduce driving comfort, but also shorten the service life of the road and increase maintenance costs. Therefore, accurate detection of road defects becomes crucial. In related technologies, methods based on target detection and image segmentation are provided for road defect detection, but they are usually too dependent on predefined categories, which greatly restricts their application capabilities in open vocabulary scenarios. For example, if a model is trained specifically for identifying road defects such as "potholes", "cracks" and "net cracks", once a new defect category needs to be introduced, the entire model often has to be retrained, which obviously brings a heavy burden and great challenges for large-scale deployment, and the methods provided in related technologies still have the problem of low detection accuracy. Summary of the invention

[0003] In response to the above-mentioned problems, the embodiments of the present application provide a road damage detection method and device based on a multimodal large model, which can improve the detection accuracy of road damage.

[0004] In a first aspect, an embodiment of the present application provides a road damage detection method based on a multimodal large model, the method comprising: Image information of the road and text information about the type of damage in the image information are obtained.

[0005] Outputting the text information to a semantic recognition model to determine semantic information; The image information and the semantic information are input into a multimodal large model to determine the road damage information.

[0006] In some embodiments, the multimodal large model includes: a backbone network, a neck network and a detection head, the input of the backbone network is the image information, the backbone network is used to extract multi-dimensional feature information based on the image information, and output the multi-dimensional feature information to the neck network, the input of the neck network is the multi-dimensional feature information and the semantic information, the neck network is used to determine fused feature information based on the multi-dimensional feature information and the semantic information, and output the fused feature information to the detection head, the detection head is used to detect and output the road disease information based on the fused feature information.

[0007] In some embodiments, the backbone network includes: a plurality of deformable adaptive lightweight channel segmentation and rearrangement ALCSR modules, each ALCSR module is used to perform adaptive channel segmentation on the input feature information, obtain the feature information of each channel, and merge the feature information of each channel for output, each channel exchanges information between channels through a channel rearrangement mechanism, each channel is provided with a deformable convolution module, the deformable convolution module is used to output the feature information in the corresponding channel after changing its shape, the plurality of ALCSR modules include: a first ALCSR module, a second ALCSR module, a third ALCSR module, a fourth ALCSR module and a fifth ALCSR module, the backbone network also includes: a first CBS module, a second CBS module Block, the first CBS module and the fourth CBS module, the first CBS module, the second CBS module, the first CBS module and the fourth CBS module are connected in sequence, the input of the first CBS module is the image information, the output of the fourth CBS module is the input of the first ALCSR module, the input of the second ALCSR module is the output of the first ALCSR module, the output of the second ALCSR module is the input of the third ALCSR module and the neck network, the output of the third ALCSR module is the input of the fourth ALCSR module, the output of the fourth ALCSR module is the input of the fifth ALCSR and the neck network, and the output of the fifth ALCSR is the input of the neck network.

[0008] In some embodiments, the neck network includes: multiple image-text feature alignment TITFA modules based on attention mechanism, each TITFA module is used to fuse the input semantic information and feature information, the multiple TITFA modules include: a first TITFA module, a second TITFA module, and a third TITFA module, the neck network also includes: a semantic information interaction module based on attention mechanism, a first upsampling module, a first feature splicing module, a sixth ALCSR module, a second upsampling module, a second feature splicing module, a seventh ALCSR module, a fifth CBS module, a third feature splicing module, an eighth ALCSR module, a sixth CBS module, a fourth feature splicing module, and a ninth ALCSR module, the input of the semantic information interaction module is the output of the fifth ALCSR module, the output of the semantic information interaction module is the input of the first upsampling module, the input of the first feature splicing module is the output of the first upsampling module and the fourth ALCSR module, the output of the first feature splicing module is the input of the sixth ALCSR module, and the output of the sixth ALCSR module is the output of the second upsampling module and the first The input of the first TITFA module is the input of the seventh ALCSR module, the output of the seventh ALCSR module is the input of the fifth CBS module and the detection head, the output of the fifth CBS module is the input of the third feature splicing module, the input of the second TITFA module is the output of the second ALCSR module and the second upsampling module, the input of the first TITFA module is the input of the seventh ALCSR module, the output of the seventh ALCSR module is the input of the fifth CBS module and the detection head, the output of the fifth CBS module is the input of the third feature splicing module, the input of the second TITFA module is the semantic information and the output of the third feature splicing module, the output of the second TITFA module is the input of the eighth ALCSR module, the output of the eighth ALCSR module is the input of the sixth CBS module and the detection head, the input of the fourth feature splicing module is the output of the fifth ALCSR module and the sixth CBS module, the input of the third TITFA module is the semantic information and the output of the fourth feature splicing module, the output of the third TITFA module is the input of the ninth ALCSR module, and the output of the ninth ALCSR module is the input of the detection head.

[0009] In some embodiments, the semantic information interaction module includes: multiple shape change units and multiple residual connection units, the shape change unit is used to change the shape of the feature map, the residual connection unit is used to retain the original information, the multiple shape change units include: a first shape change unit and a second shape change unit, the multiple residual connection units include: a first residual connection unit and a second residual connection unit, the semantic information interaction module also includes: a first multi-head attention unit, a first multi-layer perceptron unit, the input of the first shape change unit is the output of the fifth ALCSR module, the output of the first shape change unit is the input of the first multi-head attention unit, the first multi-head attention unit is used to process the input information based on the multi-head attention mechanism, and output feature information to the first residual connection unit, the first residual connection unit is used to residually connect the information input into the semantic information interaction module with the features output by the first multi-head attention unit, the output of the first residual connection unit is the input of the first multi-layer perceptron unit, the first multi-layer perceptron unit performs nonlinear mapping and outputs feature information to the second residual connection unit, the output of the second residual connection unit is the input of the second shape change unit, and the output of the second shape change unit is the input of the first upsampling module.

[0010] In some embodiments, the TITFA module includes: a convolution unit, a third shape change unit, a second multi-head attention unit, a third residual connection unit, a second multi-layer perceptron unit, a fourth residual connection unit and a fourth shape change unit, wherein the convolution unit is used to convolve the semantic information and output the convolved semantic information to the second multi-head attention unit, the third shape change unit is used to change the shape of the feature information of the image, and output the changed feature information to the second multi-head attention unit, the second multi-head attention unit performs feature fusion based on the multi-head attention mechanism, and outputs the fused feature information to the third residual connection unit, the output of the third residual connection unit is the second multi-layer perceptron unit, the output of the second multi-layer perceptron unit is the fourth residual connection unit, the output of the fourth residual connection unit is the input of the fourth shape change unit, and the fourth shape change unit is used to output the fused feature information.

[0011] In some embodiments, the detection head includes: a lightweight image-text feature fusion LITFF module, the LITFF module includes: multiple convolution units and a normalization unit, the convolution unit is used to perform dimensionality reduction and scale optimization on the image features of the fused features, the normalization unit is used to normalize the features in the numerical space, the multiple convolution units include: a first convolution unit and a second convolution unit, the multiple normalization units include: a first normalization unit and a second normalization unit, the LITFF module also includes: an Einstein summation convention unit, a parameter adjustment unit and a splicing unit, the input of the first convolution unit and the second convolution unit is the fused image information, the output of the first convolution unit is the input of the first normalization unit, the input of the second normalization unit is the semantic information, the input of the Einstein summation convention unit is the output of the first normalization unit and the second normalization unit, the output of the Einstein summation convention unit is the parameter adjustment unit, the input of the splicing unit is the output of the second convolution unit and the parameter adjustment unit, the splicing unit is used to output joint feature information, and perform detection based on the joint feature information.

[0012] In a second aspect, an embodiment of the present application provides a road disease detection device based on a multimodal large model, comprising: The acquisition module is used to acquire the image information of the road and the text information about the disease type in the image information.

[0013] A determination module, used for outputting the text information to a semantic recognition model to determine semantic information; The detection module is used to input the image information and the semantic information into a multimodal large model to determine the disease information of the road.

[0014] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method provided in the first aspect when executing the computer program.

[0015] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the method provided in the first aspect is implemented.

[0016] In a fifth aspect, an embodiment of the present application provides a computer program product, which includes a computer program. When the computer program is executed by a processor, it is at least used to implement a method as described in any one of the first aspect or the third aspect.

[0017] Compared with the prior art, the embodiments of the present invention have the following beneficial effects: The road damage detection method based on a multimodal large model provided in the embodiment of the present application obtains image information of the road and text information about the damage type in the image information. The text information is output to a semantic recognition model to determine the semantic information; the image information and the semantic information are input to a multimodal large model to determine the damage information of the road, so that the road damage detection can be realized based on the multimodal information, and the detection accuracy can be improved.

[0018] It can be understood that the beneficial effects of the second to fifth aspects mentioned above can be found in the relevant description of the first aspect mentioned above, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0020] Figure 1 A schematic diagram of a road damage detection method based on a multimodal large model provided by the present application; Figure 2 A schematic diagram of the overall structure of a multimodal model provided in an embodiment of the present application; Figure 3 A schematic diagram of the structure of an ALCSR module provided in an embodiment of the present application; Figure 4 A schematic diagram of the structure of an ALCSR module provided in an embodiment of the present application; Figure 5 A schematic diagram of the structure of a semantic information interaction module provided in an embodiment of the present application; Figure 6 A schematic diagram of the structure of a TITFA module provided in an embodiment of the present application; Figure 7 A schematic diagram of the structure of a LITFF module provided in an embodiment of the present application; Figure 8 A schematic diagram of the structure of a road disease detection device based on a multimodal large model provided in an embodiment of the present application; Fig. 9 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In the following description, specific details such as specific system structures, technologies, etc. are provided for the purpose of illustration rather than limitation, so as to provide a thorough understanding of the embodiments of the present application. However, it should be clear to those skilled in the art that the present application may also be implemented in other embodiments without these specific details. In other cases, detailed descriptions of well-known systems, devices, circuits, and methods are omitted to prevent unnecessary details from obstructing the description of the present application.

[0022] It should be understood that when used in the present specification and the appended claims, the term "comprising" indicates the presence of described features, integers, steps, operations, elements and / or components, but does not exclude the presence or addition of one or more other features, integers, steps, operations, elements, components and / or combinations thereof.

[0023] It should also be understood that the term “and / or” used in the specification and appended claims refers to any and all possible combinations of one or more of the associated listed items, and includes these combinations.

[0024] As used in the specification of this application and the appended claims, the term "if" can be interpreted as "when" or "uponce" or "in response to determining" or "in response to detecting" depending on the context. Similarly, the phrases "if it is determined" or "if it is detected" can be interpreted as meaning "uponce it is determined" or "in response to determining" or "uponce detected" or "in response to detecting" depending on the context.

[0025] In addition, in the description of the present application specification and the appended claims, the terms "first", "second", "third", etc. are only used to distinguish the descriptions and cannot be understood as indicating or implying relative importance.

[0026] References to "one embodiment" or "some embodiments" etc. described in the specification of this application mean that one or more embodiments of the present application include specific features, structures or characteristics described in conjunction with the embodiment. Therefore, the phrases "in one embodiment", "in some embodiments", "in some other embodiments", "in some other embodiments", etc. appearing in different places in this specification do not necessarily refer to the same embodiment, but mean "one or more but not all embodiments", unless otherwise specifically emphasized in other ways.

[0027] Based on the technical problems of the related art, the embodiment of the present application provides a road disease detection method based on a multimodal large model that can be applied to electronic devices. The electronic devices may include: mobile phones, tablet computers, wearable devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPC), netbooks, personal digital assistants (PDA), and the embodiment of the present application does not impose any restrictions on the specific types of electronic devices.

[0028] The present application embodiment provides a road disease detection method based on a multimodal large model. Figure 1 The present application provides a flow chart of a road disease detection method based on a multimodal large model, as shown in FIG. Figure 1 As shown, the method includes: Step S101, obtaining image information of the road and text information about the disease type in the image information.

[0029] In the embodiment of the present application, the image information of the road is a photo or video frame of the road surface acquired by a camera, drone, satellite or other image acquisition device. These images contain the physical characteristics of the road and a visual representation of potential defects. The text information describes the types of road defects that may exist in the image, such as cracks, potholes, ruts, waterlogging, etc.

[0030] In the embodiment of the present application, the text information may be manually input information.

[0031] Step S102: output the text information to a semantic recognition model to determine semantic information.

[0032] In an embodiment of the present application, the semantic recognition model may be CLIP (Contrastive Language-Image Pre-training), which is a multimodal pre-training model proposed by OpenAI.

[0033] In the embodiment of the present application, the CLIP model includes a text encoder for processing text information and extracting its semantic features. The text encoder converts the input text information (such as descriptions of disease types such as "cracks" and "potholes") into an embedding vector, which is aligned with the image representation in the semantic space.

[0034] Step S103: input the image information and semantic information into the multimodal large model to determine the road damage information.

[0035] In the embodiment of the present application, the multimodal large model is a deep learning model that can simultaneously process data from different modalities (such as images and texts). It improves the accuracy and efficiency of disease identification by fusing image information and semantic information and utilizing the complementarity of the two.

[0036] In the embodiment of the present application, image information and semantic information are passed as input to the multimodal large model to extract visual features in the image and semantic features in the text. The model generates a comprehensive disease recognition result by fusing these features. This may include accurate classification of disease types, determination of disease locations, etc. The multimodal large model can output an image with the recognition results of the disease annotated in the image.

[0037] The road disease detection method based on a multimodal large model provided in the embodiment of the present application obtains image information of the road and text information about the disease type in the image information. The text information is output to a semantic recognition model to determine the semantic information; the image information and the semantic information are input to a multimodal large model to determine the road disease information, and the road disease detection can be realized based on the multimodal information, which can improve the detection accuracy.

[0038] In some embodiments, the multimodal large model includes: a backbone network, a neck network and a detection head. The input of the backbone network is image information. The backbone network is used to extract multi-dimensional feature information based on the image information, and output the multi-dimensional feature information to the neck network. The input of the neck network is multi-dimensional feature information and semantic information. The neck network is used to determine fused feature information based on the multi-dimensional feature information and semantic information, and output the fused feature information to the detection head. The detection head is used to detect and output road disease information based on the fused feature information.

[0039] In the embodiment of the present application, the backbone network is the basic part of the multimodal large model, which is responsible for processing the input image information. It is usually a deep convolutional neural network (CNN) that can extract multi-level and multi-dimensional feature information from the image. These feature information include local details (such as edges, textures) and global structures (such as shape, layout) of the image. The neck network is located between the backbone network and the detection head, responsible for further processing and fusing information from different sources. The input of the neck network includes multi-dimensional feature information extracted by the backbone network and semantic information from the semantic recognition module. The task of the neck network is to effectively fuse this information to generate more expressive fused feature information. The detection head is the final output part of the multimodal large model, responsible for disease detection based on fused feature information. It usually contains a series of fully connected layers or convolutional layers for classifying, regressing or other forms of processing the fused feature information to output road disease information. This information may include the type, location, size, etc. of the disease. Fused feature information refers to the feature representation obtained by fusing multi-dimensional feature information and semantic information by the neck network. This fusion is usually achieved through feature concatenation, feature addition, attention mechanism, etc., aiming to effectively combine information from different sources and improve the detection performance of the model.

[0040] In the embodiment of the present application, the neck network may be a network improved based on the basic structure of the feature pyramid network (FPN). The detection head may include: a plurality of classifiers and regressors for classifying and regressing the fused feature information.

[0041] In some embodiments, Figure 2 A schematic diagram of the overall structure of a multimodal model provided in an embodiment of the present application is shown in FIG. Figure 2As shown, the backbone network includes: a plurality of deformable adaptive lightweight channel segmentation and rearrangement (ALCSR, Adaptive Lightweight Channel Splitting and Reassembly) modules, each ALCSR module is used to perform adaptive channel segmentation on the input feature information, obtain the feature information of each channel, and merge the feature information of each channel for output, each channel exchanges information between channels through a channel rearrangement mechanism, each channel is provided with a deformable convolution module, and the deformable convolution module is used to output the feature information in the corresponding channel after changing the shape, the plurality of ALCSR modules include: a first ALCSR module, a second ALCSR module, a third ALCSR module, a fourth ALCSR module and a fifth ALCSR module, and the backbone network also includes: a first CBS module, a second CBS module, a third The CBS module and the fourth CBS module, the first CBS module, the second CBS module, the third CBS module, and the fourth CBS module are connected in sequence, the input of the first CBS module is image information, the output of the fourth CBS module is the input of the first ALCSR module, the input of the second ALCSR module is the output of the first ALCSR module, the output of the second ALCSR module is the input of the third ALCSR module and the neck network, the output of the third ALCSR module is the input of the fourth ALCSR module, the output of the fourth ALCSR module is the input of the fifth ALCSR and the neck network, and the output of the fifth ALCSR is the input of the neck network.

[0042] In the embodiment of the present application, the backbone network adopts an adaptive lightweight channel segmentation and rearrangement (ALCSR) module, and combines it with deformable convolution to achieve efficient feature extraction and information interaction. The ALCSR module optimizes feature extraction through adaptive channel segmentation, and strengthens information exchange between channels through the channel rearrangement mechanism. At the same time, the introduction of deformable convolution, combined with learnable offsets, improves the model's perception of object shape, size changes and deformed targets. In order to further enhance the global modeling capabilities of the model, an efficient hybrid encoder architecture is designed. This architecture combines the advantages of convolutional neural networks (CNNs) in local feature extraction and the powerful capabilities of Transformers in global context modeling, thereby achieving remarkable results in the comprehensive modeling of local details and global semantics, and greatly enhancing the generalization ability of the model.

[0043] In the embodiment of the present application, Figure 3 A structural diagram of a deformable ALCSR module provided in an embodiment of the present application is shown in FIG. Figure 4 A structural diagram of a deformable ALCSR module provided in an embodiment of the present application is shown in FIG. Figure 3 and Figure 4 As shown, the deformable ALCSR module can be divided into two structures. If the size of the feature information needs to be changed, it can be used Figure 4 If the structure shown does not need to change the size of the feature, you can use Figure 3 The structure shown.

[0044] In the embodiment of the present application, the first ALCSR module, the second ALCSR module, the third ALCSR module, the fourth ALCSR module and the fifth ALCSR module are connected in a specific order to achieve gradual extraction and transformation of feature information.

[0045] In the embodiment of the present application, the ALCSR module includes: a segmentation module, a basic network, a multi-layer network, a splicing unit and a channel mixing / channel rearrangement unit. The segmentation module is used to segment the features and input them into the basic network and the multi-layer network respectively. The outputs of the basic network and the multi-layer network are the inputs of the splicing unit. The output of the splicing unit is the input of the channel mixing / channel rearrangement unit, and finally the processed features are output. In the basic network, it includes: the convolution operation can also include a pooling operation. In the multi-layer network, it includes: DW Bottleneck, that is, Depthwise Separable Convolution Bottleneck.

[0046] In the embodiment of the present application, the ALCSR module can also be called a Deformable-ALCSR module. The ALCSR module adopts an adaptive feature processing strategy according to different feature levels, and realizes efficient feature extraction by adjusting the channel segmentation ratio and operation mechanism. At a lower feature level, the module focuses on the extraction of fine-grained information, adopts a smaller channel segmentation ratio, and allows most channels to be processed by a multi-layer network to enhance the feature capture capability. At a higher feature level, the module focuses on the extraction of abstract semantic features, uses a larger channel segmentation ratio and introduces jump connections to retain high-level semantic information and optimize computational efficiency. In addition, the module combines bottleneck structures and deep convolutions to achieve lightweight design. After introducing deformable convolutions into the ALCSR module, the module can learn spatial offsets, enabling it to better perceive the shape, scale changes and complex deformations of objects. This further enhances the module's perception of irregular shapes, different scales and complex deformation disease areas, and provides efficient and reliable feature extraction support for open vocabulary instance segmentation tasks.

[0047] In the embodiment of the present application, the CBS module is usually a basic component of a convolutional neural network (CNN) in the field of deep learning, especially in the target detection algorithm. It includes the following key parts: Conv layer (convolution layer): responsible for performing convolution operations and extracting local features of input data. BN layer (Batch Normalization layer): batch normalization of the output of the convolution layer helps to accelerate the training process and improve the stability of the model. Activation function: usually located after the BN layer, used to introduce nonlinear factors and enhance the expressiveness of the model. In different CBS module implementations, the activation function may be different, such as Sigmoid, ReLU, etc. The CBS module plays an important role in the target detection algorithm. By stacking multiple CBS modules, a convolutional neural network with powerful feature extraction capabilities can be constructed. These features are crucial for subsequent target detection, classification and other tasks. These modules are connected in sequence to form a feature extraction pipeline. Among them, the input of the first CBS module is image information, and the output of the fourth CBS module is used as the input of the first ALCSR module.

[0048] In an embodiment of the present application, the convolution kernel K, step size S, padding P and grouping G of the CBS module can be configured. For example, K can be configured to 3, S can be 2, P can be 1, and G can be 1.

[0049] In the embodiment of the present application, after being processed by a series of ALCSR modules and CBS modules, the backbone network outputs the transformed feature information to the neck network for further processing.

[0050] In some embodiments, see Figure 2, the neck network includes: multiple image-text feature alignment TITFA (Transformer-based image-text feature alignment) modules based on attention mechanism, each TITFA module is used to fuse the input semantic information and feature information, and the multiple TITFA modules include: a first TITFA module, a second TITFA module, and a third TITFA module. The neck network also includes: a semantic information interaction module based on attention mechanism, a first upsampling module, a first feature splicing module, a sixth ALCSR module, a second upsampling module, a second feature splicing module, a seventh ALCSR module, a fifth CBS module, a third feature splicing module, an eighth ALCSR module, a sixth CBS module, a fourth feature splicing module, and a ninth ALCSR module. The input of the semantic information interaction module is the output of the fifth ALCSR module, the output of the semantic information interaction module is the input of the first upsampling module, the input of the first feature splicing module is the output of the first upsampling module and the fourth ALCSR module, the output of the first feature splicing module is the input of the sixth ALCSR module, and the output of the sixth ALCSR module is the output of the second upsampling module. The input of the first TITFA module is the input of the seventh ALCSR module, the output of the seventh ALCSR module is the input of the fifth CBS module and the detection head, the output of the fifth CBS module is the input of the third feature splicing module, the input of the second TITFA module is the semantic information and the output of the third feature splicing module, the output of the second TITFA module is the input of the eighth ALCSR module, the output of the eighth ALCSR module is the input of the sixth CBS module and the detection head, the input of the fourth splicing module is the output of the fifth ALCSR module and the sixth CBS module, the input of the third TITFA module is the semantic information and the output of the fourth splicing module, the output of the third TITFA module is the input of the ninth ALCSR module, and the output of the ninth ALCSR module is the input of the detection head.

[0051] In the embodiment of the present application, each feature splicing module in the figure is represented by Concat.

[0052] In the embodiment of the present application, the first TITFA module, the second TITFA module, and the third TITFA module all receive inputs from different sources and output processed feature information. The upsampling module upsamples the feature map to increase its resolution. The feature splicing module splices the feature maps from different modules to fuse different feature information. The ALCSR module can function as a downsampling module.

[0053] In the embodiment of the present application, the semantic information interaction module based on the attention mechanism can be called a TSII (Transformer-based semantic information interaction) module.

[0054] In the embodiment of the present application, the sixth ALCSR module, the seventh ALCSR module, and the eighth ALCSR module in the neck network are used as downsampling modules, supporting a variety of downsampling mechanisms, such as convolution with a step size of 2, pooling, and their combination, to flexibly adapt to the needs of different resolutions. Through the channel rearrangement mechanism, the information interaction between channels is enhanced, thereby improving the diversity and expression ability of feature representation.

[0055] In some embodiments, Figure 5 A schematic diagram of the structure of a semantic information interaction module provided in an embodiment of the present application is shown in FIG. Figure 5 As shown, the semantic information interaction module includes: multiple shape change units and multiple residual connection units, the shape change unit (reshape) is used to change the shape of the feature map, the residual connection unit (Add&Norm) is used to retain the original information, the multiple shape change units include: a first shape change unit and a second shape change unit, the multiple residual connection units include: a first residual connection unit and a second residual connection unit, the semantic information interaction module also includes: a first multi-head attention unit, a first multi-layer perceptron unit, the input of the first shape change unit is the output of the fifth ALCSR module, the output of the first shape change unit is the input of the first multi-head attention unit, the first multi-head attention unit is used to process the input information based on the multi-head attention mechanism, and output feature information to the first residual connection unit, the first residual connection unit is used to residually connect the information input to the semantic information interaction module with the features output by the first multi-head attention unit, the output of the first residual connection unit is the input of the first multi-layer perceptron unit, the first multi-layer perceptron unit performs nonlinear mapping and outputs feature information to the second residual connection unit, the output of the second residual connection unit is the input of the second shape change unit, and the output of the second shape change unit is the input of the first upsampling module.

[0056] In the embodiment of the present application, each shape changing unit is used to change the shape of the feature map to adapt to the needs of the subsequent processing module, and each residual connection unit is used to retain the original information to avoid losing important features in the deep layer of the network. The first multi-head attention unit processes the input information based on the multi-head attention mechanism to capture the correlation between features.

[0057] In an embodiment of the present application, the output of the fifth ALCSR module is first input into the first shape change unit to change the shape of the feature map. The output of the first shape change unit is input into the first multi-head attention unit, and the features are processed based on the multi-head attention mechanism. The output of the first multi-head attention unit is input into the first residual connection unit, and a residual connection is performed with the information input into the semantic information interaction module. The output of the first residual connection unit is input into the first multi-layer perceptron unit for nonlinear mapping. The output of the first multi-layer perceptron unit is input into the second shape change unit to change the shape of the feature map again. The output of the second shape change unit is ultimately used as the input of the first upsampling module. The first multi-head attention unit includes multiple shape change units, multiple Einstein summation convention units, a regularization unit, and the like.

[0058] In some embodiments, Figure 6 A schematic diagram of the structure of a TITFA module provided in an embodiment of the present application is shown in FIG. Figure 6 As shown, the TITFA module includes: a convolution unit, a third shape change unit, a second multi-head attention unit, a third residual connection unit, a second multi-layer perceptron unit, a fourth residual connection unit and a fourth shape change unit, the convolution unit is used to convolve semantic information and output the convolved semantic information to the second multi-head attention unit, the third shape change unit is used to change the shape of the feature information of the image, and output the changed feature information to the second multi-head attention unit, the second multi-head attention unit performs feature fusion based on the multi-head attention mechanism, and outputs the fused feature information to the third residual connection unit, the output of the third residual connection unit is the second multi-layer perceptron unit, the output of the second multi-layer perceptron unit is the fourth residual connection unit, the output of the fourth residual connection unit is the input of the fourth shape change unit, and the fourth shape change unit is used to output the fused feature information.

[0059] In an embodiment of the present application, the TITFA module receives semantic information and feature information of the image as input, the semantic information is processed by the convolution unit and then output to the second multi-head attention unit, and the feature information of the image is changed in shape by the third shape change unit and then output to the second multi-head attention unit. The second multi-head attention unit fuses the input semantic information and image feature information based on the multi-head attention mechanism and outputs it to the third residual connection unit. The third residual connection unit performs residual connection and passes the output to the second multi-layer perceptron unit. The second multi-layer perceptron unit performs nonlinear mapping on the features and outputs them to the fourth residual connection unit. The fourth shape change unit performs the final shape change or processing on the features and outputs the final fused feature information of the TITFA module.

[0060] In the embodiment of the present application, in the feature fusion stage, the semantic information of the image and text in the TITFA module generates query (Q), key (K) and value (V) matrices respectively, which are used for the self-attention mechanism to model the correlation within each modality. The self-attention mechanism of text features helps to grasp the contextual information of the input semantics. In the cross-attention mechanism, the query vector of the image feature interacts with the key and value vectors of the text feature, and the effective fusion of cross-modal semantic information is achieved through weighted operations. This design allows text semantic information to guide the optimization of image features at the spatial and semantic levels, especially in open vocabulary scenarios, significantly enhancing the model's ability to understand diverse target instances. Next, the fused features are nonlinearly mapped through a multi-layer perceptron (MLP), and residual connections (Add&Norm) are used to retain the original information, while further strengthening the semantic expression after interaction. Finally, these optimized features are mapped back to the image space.

[0061] The method provided in the embodiment of the present application fully utilizes the global modeling ability and attention mechanism of Transformer, and successfully integrates the multimodal knowledge of the Chinese CLIP model into the instance segmentation task. This not only improves the model's ability to handle diverse instance descriptions, but also enhances its robustness and generalization capabilities in open vocabulary scenarios. Through the deep fusion of images and text, the model demonstrates excellent adaptability to complex instance segmentation tasks, providing an efficient solution for visual tasks in the open vocabulary field.

[0062] The neck network provided in the embodiment of the present application uses an efficient hybrid encoder architecture to cleverly combine the advantages of convolutional neural network CNN and Transformer. CNN is good at capturing local features and spatial relationships, while Transformer has excellent global context modeling capabilities and can effectively handle long-distance dependencies and semantic associations. By integrating these two technologies, the model not only retains the ability to extract accurate local features, but also improves the understanding of global information, significantly enhancing the generalization performance and application scope.

[0063] In the trunk network, we optimized the overall architecture based on the Feature Pyramid Network (FPN). For feature maps of smaller sizes, the global context modeling advantage of Transformer is introduced, especially the self-attention operation is applied on high-level feature maps rich in semantic concepts. This design effectively captures the long-distance dependencies between different conceptual entities, which helps the subsequent modules to locate and identify targets more accurately. At the same time, Transformer can aggregate global information at the high-level feature level to prevent target ambiguity and misjudgment caused by the loss of local features. However, in feature maps of larger sizes, due to the sparse semantic information and the redundancy and confusion that may occur when interacting with high-level features, we intentionally avoid excessive interactions within low-level features in the design. This strategy not only reduces the computational cost, but also prevents meaningless feature accumulation, thereby improving the efficiency and performance of the model. At the same time, the multi-scale feature aggregation capability of FPN is fully utilized in the architecture. By integrating feature maps of different scales layer by layer, the consistency of the model at different spatial resolutions is ensured, providing stronger support and higher robustness for instance segmentation tasks. This hybrid encoder architecture not only overcomes the shortcomings of traditional networks in global information modeling, but also avoids the waste of resources in the global modeling process.

[0064] In some embodiments, the detection head includes: a lightweight image text feature fusion LITFF module, Figure 7 A schematic diagram of the structure of a LITFF module provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, the LITFF module includes: multiple convolution units and normalization units, the convolution unit is used to perform dimensionality reduction and scale optimization on the image features of the fused features, the normalization unit is used to normalize the features in the numerical space, the multiple convolution units include: a first convolution unit, a second convolution unit, the multiple normalization units include: a first normalization unit and a second normalization unit, the LITFF module also includes: an Einstein sum convention unit, a parameter adjustment unit and a splicing unit, the input of the first convolution unit and the second convolution unit is the fused image feature, the output of the first convolution unit is the input of the first normalization unit, the input of the second normalization unit is semantic information, the input of the Einstein sum convention unit is the output of the first normalization unit and the second normalization unit, the output of the Einstein sum convention unit is the parameter adjustment unit, the input of the splicing unit is the output of the second convolution unit and the parameter adjustment unit, the splicing unit is used to output joint feature information, and detect based on the joint feature information.

[0065] In the embodiment of the present application, the convolution unit is used to perform dimensionality reduction and scale optimization on the image features of the fusion features. The convolution operation can extract local features in the image, and realize the transformation and dimensionality reduction of the features through different convolution kernels. The normalization unit is used to normalize the features in the numerical space to accelerate the training process and improve the generalization ability of the model. The normalization operation can make the eigenvalues ​​distributed in a reasonable range, thereby avoiding problems such as gradient disappearance or explosion. The Einstein summation convention is a concise way to represent element-level operations between multidimensional arrays (such as tensors). In the LITFF module, the Einstein summation convention unit may be used to perform some form of element-level operation, such as summation, weighted summation, etc., on the output of the first normalization unit and the second normalization unit to achieve feature fusion. The parameter adjustment unit further adjusts the parameters of the output of the Einstein summation convention unit, such as scaling, translation, etc., to meet the needs of subsequent processing. The splicing unit is used to splice the output of the second convolution unit and the output of the parameter adjustment unit to form joint feature information. The concatenation operation can combine features from different sources or different processing stages to enrich the feature representation.

[0066] In an embodiment of the present application, the LITFF module receives information after fusion of image information and semantic information as input, and the fused image information is subjected to dimensionality reduction processing and scale optimization through the first convolution unit, and then output to the first normalization unit for normalization. The semantic information is normalized through the second normalization unit, and the output of the first normalization unit and the output of the second normalization unit are fused through the Einstein sum convention unit. The output of the Einstein sum convention unit is further parameter adjusted through the parameter adjustment unit. On the other hand, the fused image information is also processed by another path through the second convolution unit and output to the splicing unit. The splicing unit splices the output of the second convolution unit and the output of the parameter adjustment unit to form joint feature information. The joint feature information is used as the final output of the LITFF module for subsequent detection tasks.

[0067] In the embodiment of the present application, the LITFF module is an efficient decoder head, the core process of which includes extracting features from images and texts respectively and efficiently fusing them to generate accurate open vocabulary target area extraction.

[0068] First, the input image features (feature_map) are processed through independent convolution modules (Conv ModuleList) and normalization operations (normalize). The convolution module performs dimensionality reduction and multi-scale optimization on image features, while the normalization of text features ensures the consistency of cross-modal features in the numerical space, providing a basis for subsequent fusion operations. The image and text data after feature preprocessing are subjected to feature interaction and fusion through the einsum operation. This operation directly aligns the semantic information of the image and text modalities in the feature dimension, so that the semantic guidance in the text can be accurately mapped to the image space. Next, the fused features are further adjusted through the logit_scale and bias parameters to strengthen the correlation between modalities and reduce noise interference. Subsequently, the semantic information of the open vocabulary is spliced ​​with the boundary regression information of the image features to form a joint feature representation. These joint features not only contain the category information of the target, but also can clarify its spatial positioning relationship through boundary features. The spliced ​​features are input into the subsequent network, and combined with the design idea of ​​anchor-free, the real box prediction of the target is directly performed.

[0069] In some embodiments, the inspection head further includes: a Proto network, which realizes efficient generation of segmentation masks for each category, thereby completing the instance segmentation task. The Proto network processes the fused features and uses upsampling operations to restore the spatial resolution and generate a set of potential segmentation masks. These masks are given corresponding semantic information and instance attribution under the guidance of category prediction and target positioning, thereby achieving accurate segmentation effects based on target detection.

[0070] In the embodiment of the present application, a multi-task loss function is used in the network training process, specifically including: Boxes loss for optimizing bounding box regression, DFL (Distribution FocalLoss) loss for improving bounding box prediction accuracy, classification loss for ensuring target classification accuracy, and segmentation loss for generating segmentation masks. The joint design of these loss functions promotes the collaborative optimization between the detection and segmentation modules, thereby improving the overall performance of the model. Thanks to the decoupling of detection and segmentation tasks in the design of the network architecture, the system has a high degree of flexibility and scalability. For detection tasks, the bounding box regression and classification modules can be run separately to achieve target positioning; in segmentation tasks, the segmentation module can be combined to output high-precision segmentation masks; for instance segmentation tasks, through the collaborative operation of the detection and segmentation modules, accurate segmentation masks can be output while generating target boxes.

[0071] In some embodiments, before step S103, the method further includes: Step S1, obtaining a sample data set.

[0072] In the embodiment of the present application, the image resolution in the sample data set is as high as 1920×1080. The data set covers the shooting period from 7 am to 8 pm, and includes images under various weather and lighting conditions. A total of N original pictures are included, which record various road damage conditions in detail. In addition, these images are refined instance segmentation and annotation with the help of the SAM2 model, and text label information is added. The annotated categories are mainly divided into three categories: sidewalk diseases, asphalt pavement diseases, and cement pavement diseases, and are further subdivided into 36 detailed road disease types. In order to protect personal privacy, all facial and license plate information involved in the data set have been mosaiced.

[0073] Step S2, training a multimodal large model based on the sample data set.

[0074] In the embodiment of the present application, a large multimodal model can be trained using a sample data set.

[0075] After training is completed, the multimodal large model can be deployed to detect road hazards.

[0076] The method provided in the embodiment of the present application has a multimodal large model with efficient feature extraction capability, good generalization performance and continuous learning capability. It adopts an adaptive lightweight channel segmentation and rearrangement (ALCSR) module and combines deformable convolution. The ALCSR module optimizes feature extraction through adaptive channel segmentation, enhances information exchange between channels through the channel rearrangement mechanism, and introduces a learnable offset in combination with deformable convolution, thereby improving the model's perception of object shape, size changes and deformed objects. The neck network adopts an efficient hybrid encoder architecture, which not only retains the advantages of convolutional neural network CNN in local feature extraction, but also combines the powerful ability of Transformer in global context modeling, thereby enhancing the generalization performance of the model. The image-text fusion module based on the Transformer architecture combines the self-attention and cross-attention mechanisms to enable text information to effectively guide the optimization of image features. It also provides a lightweight open vocabulary instance segmentation decoder head, which optimizes computational efficiency and enhances the adaptability and accuracy of instance segmentation in a variety of environments.

[0077] It should be understood that the size of the serial numbers of the steps in the above embodiments does not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0078] According to the aforementioned embodiments, the embodiments of the present application provide a road defect detection device based on a multimodal large model. The modules included in the device and the units included in each module can be implemented by a processor in a computer device; of course, they can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU, Central Processing Unit), a microprocessor (MPU, Microprocessor Unit), a digital signal processor (DSP, Digital Signal Processing) or a field programmable gate array (FPGA, Field Programmable Gate Array), etc.

[0079] The present application embodiment provides a road disease detection device based on a multi-modal large model. Figure 8 A schematic diagram of the structure of a road disease detection device based on a multimodal large model provided in an embodiment of the present application is shown in FIG. Figure 8 As shown, the road disease detection 800 based on the multimodal large model includes: The acquisition module 801 is used to acquire the image information of the road and the text information about the disease type in the image information.

[0080] A determination module 802, configured to output the text information to a semantic recognition model to determine semantic information; The detection module 803 is used to input the image information and the semantic information into the multimodal large model to determine the disease information of the road.

[0081] In some embodiments, the multimodal large model includes: a backbone network, a neck network and a detection head, the input of the backbone network is the image information, the backbone network is used to extract multi-dimensional feature information based on the image information, and output the multi-dimensional feature information to the neck network, the input of the neck network is the multi-dimensional feature information and the semantic information, the neck network is used to determine fused feature information based on the multi-dimensional feature information and the semantic information, and output the fused feature information to the detection head, the detection head is used to detect and output the road disease information based on the fused feature information.

[0082] In some embodiments, the backbone network includes: multiple adaptive lightweight channel segmentation and rearrangement ALCSR modules, each ALCSR module is used to adaptively segment the input feature information, obtain the feature information of each channel, and merge the feature information of each channel for output, each channel exchanges information between channels through a channel rearrangement mechanism, the multiple ALCSR modules include: a first ALCSR module, a second ALCSR module, a third ALCSR module, a fourth ALCSR module and a fifth ALCSR module, the backbone network also includes: a first CBS module, a second CBS module, a first CBS module and a fourth CBS module, the first CBS The first CBS module, the second CBS module, the first CBS module, and the fourth CBS module are connected in sequence, the input of the first CBS module is the image information, the output of the fourth CBS module is the input of the first ALCSR module, the input of the second ALCSR module is the output of the first ALCSR module, the output of the second ALCSR module is the input of the third ALCSR module and the neck network, the output of the third ALCSR module is the input of the fourth ALCSR module, the output of the fourth ALCSR module is the input of the fifth ALCSR and the neck network, and the output of the fifth ALCSR is the input of the neck network.

[0083] In some embodiments, the neck network includes: multiple image-text feature alignment TITFA modules based on attention mechanism, each TITFA module is used to fuse the input semantic information and feature information, the multiple TITFA modules include: a first TITFA module, a second TITFA module, and a third TITFA module, the neck network also includes: a semantic information interaction module based on attention mechanism, a first upsampling module, a first feature splicing module, a sixth ALCSR module, a second upsampling module, a second feature splicing module, a seventh ALCSR module, a fifth CBS module, a third feature splicing module, an eighth ALCSR module, a sixth CBS module, a fourth feature splicing module and a ninth ALCSR module, the input of the semantic information interaction module is the output of the fifth ALCSR module, the output of the semantic information interaction module is the input of the first upsampling module, the input of the first feature splicing module is the output of the first upsampling module and the fourth ALCSR module, the output of the first feature splicing module is the input of the sixth ALCSR module, and the output of the sixth ALCSR module is the input of the second upsampling module and the input of the third feature splicing module, the input of the second splicing module is the output of the second ALCSR module and the second upsampling module, the input of the first TITFA module is the input of the second splicing module and the semantic information, the output of the first TITFA module is the input of the seventh ALCSR module, the output of the seventh ALCSR module is the input of the fifth CBS module and the detection head, the output of the fifth CBS module is the input of the third feature splicing module, the input of the second TITFA module is the semantic information and the output of the third feature splicing module, the output of the second TITFA module is the input of the eighth ALCSR module, the output of the eighth ALCSR module is the input of the sixth CBS module and the detection head, the input of the fourth splicing module is the output of the fifth ALCSR module and the sixth CBS module, the input of the third TITFA module is the semantic information and the output of the fourth splicing module, the output of the third TITFA module is the input of the ninth ALCSR module, and the output of the ninth ALCSR module is the input of the detection head.

[0084] In some embodiments, the semantic information interaction module includes: multiple shape change units and multiple residual connection units, the shape change unit is used to change the shape of the feature map, the residual connection unit is used to retain the original information, the multiple shape change units include: a first shape change unit and a second shape change unit, the multiple residual connection units include: a first residual connection unit and a second residual connection unit, the semantic information interaction module also includes: a first multi-head attention unit, a first multi-layer perceptron unit, the input of the first shape change unit is the output of the fifth ALCSR module, the output of the first shape change unit is the input of the first multi-head attention unit, the first multi-head attention unit is used to process the input information based on the multi-head attention mechanism, and output feature information to the first residual connection unit, the first residual connection unit is used to residually connect the information input into the semantic information interaction module with the features output by the first multi-head attention unit, the output of the first residual connection unit is the input of the first multi-layer perceptron unit, the first multi-layer perceptron unit performs nonlinear mapping and outputs feature information to the second residual connection unit, the output of the second residual connection unit is the input of the second shape change unit, and the output of the second shape change unit is the input of the first upsampling module.

[0085] In some embodiments, the TITFA module includes: a convolution unit, a third shape change unit, a second multi-head attention unit, a third residual connection unit, a second multi-layer perceptron unit, a fourth residual connection unit and a fourth shape change unit, wherein the convolution unit is used to convolve the semantic information and output the convolved semantic information to the second multi-head attention unit, the third shape change unit is used to change the shape of the feature information of the image, and output the changed feature information to the second multi-head attention unit, the second multi-head attention unit performs feature fusion based on the multi-head attention mechanism, and outputs the fused feature information to the third residual connection unit, the output of the third residual connection unit is the second multi-layer perceptron unit, the output of the second multi-layer perceptron unit is the fourth residual connection unit, the output of the fourth residual connection unit is the input of the fourth shape change unit, and the fourth shape change unit is used to output the fused feature information.

[0086] In some embodiments, the detection head includes: a lightweight image-text feature fusion LITFF module, the LITFF module includes: multiple convolution units and a normalization unit, the convolution unit is used to perform dimensionality reduction and scale optimization on the image features of the fused features, the normalization unit is used to normalize the features in the numerical space, the multiple convolution units include: a first convolution unit and a second convolution unit, the multiple normalization units include: a first normalization unit and a second normalization unit, the LITFF module also includes: an Einstein summation convention unit, a parameter adjustment unit and a splicing unit, the input of the first convolution unit and the second convolution unit is the fused image information, the output of the first convolution unit is the input of the first normalization unit, the input of the second normalization unit is the semantic information, the input of the Einstein summation convention unit is the output of the first normalization unit and the second normalization unit, the output of the Einstein summation convention unit is the parameter adjustment unit, the input of the splicing unit is the output of the second convolution unit and the parameter adjustment unit, the splicing unit is used to output joint feature information, and perform detection based on the joint feature information.

[0087] in addition, Figure 8 The road damage detection based on the multimodal large model shown can be a software unit, a hardware unit, or a combination of software and hardware units built into existing electronic devices, or can be integrated into electronic devices as an independent pendant, or can exist as an independent terminal device.

[0088] It should be noted that the information interaction, execution process, etc. between the above-mentioned devices / units are based on the same concept as the method embodiment of the present application. Their specific functions and technical effects can be found in the method embodiment part and will not be repeated here.

[0089] The technicians in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional units and modules is used as an example for illustration. In practical applications, the above-mentioned function allocation can be completed by different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiment can be integrated in a processing unit, or each unit can exist physically separately, or two or more units can be integrated in one unit. The above-mentioned integrated unit can be implemented in the form of hardware or in the form of software functional units. In addition, the specific names of the functional units and modules are only for the convenience of distinguishing each other, and are not used to limit the scope of protection of this application. The specific working process of the units and modules in the above-mentioned system can refer to the corresponding process in the aforementioned method embodiment, which will not be repeated here.

[0090] Fig. 9This is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. Fig. 9 As shown, the electronic device 3 of this embodiment may include: at least one processor 30 ( Fig. 9 Only one processor 30 is shown in the figure), a memory 31, and a computer program 32 stored in the memory 31 and executable on at least one processor 30. When the processor 30 executes the computer program 32, the steps in any of the above-mentioned method embodiments are implemented; or, when the processor 30 executes the computer program 32, the functions of the modules / units in the above-mentioned device embodiments are implemented.

[0091] Exemplarily, the computer program 32 may be divided into one or more modules / units, one or more modules / units are stored in the memory 31, and are executed by the processor 30 to complete the present application. One or more modules / units may be a series of computer program 32 instruction segments capable of completing specific functions, and the instruction segments are used to describe the execution process of the computer program 32 in the electronic device 3.

[0092] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program 32. When the computer program 32 is executed by the processor 30, the steps in the above-mentioned method embodiments can be implemented.

[0093] An embodiment of the present application provides a computer program product. When the computer program product runs on an electronic device, the electronic device can implement the steps in the above-mentioned method embodiments when executing the computer program product.

[0094] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. According to this understanding, the present application implements all or part of the processes in the above-mentioned embodiment method, which can be completed by instructing the relevant hardware through a computer program 32. The computer program 32 can be stored in a computer-readable storage medium. When the computer program 32 is executed by the processor 30, the steps of the above-mentioned various method embodiments can be implemented. Among them, the computer program 32 includes computer program code, which can be in source code form, object code form, executable file or some intermediate form. The computer-readable medium can at least include: any entity or device that can carry the computer program code to the terminal, a recording medium, a computer memory, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), an electric carrier signal, a telecommunication signal and a software distribution medium. For example, a USB flash drive, a mobile hard disk, a magnetic disk or an optical disk. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electric carrier signals and telecommunication signals.

[0095] In the above embodiments, the description of each embodiment has its own emphasis. For parts that are not described or recorded in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0096] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0097] In the embodiments provided in the present application, it should be understood that the disclosed devices / network equipment and methods can be implemented in other ways. For example, the device / network equipment embodiments described above are merely schematic. For example, the division of the modules or units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0098] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0099] The embodiments described above are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments may still be modified, or some of the technical features may be replaced by equivalents. Such modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application, and should all be included in the protection scope of the present application.

[0100] The relevant user personal information that may be involved in the various embodiments of this application is strictly in accordance with the requirements of laws and regulations, following the principles of legality, legitimacy and necessity, based on the reasonable purposes of business scenarios, to process the personal information that users actively provide during the use of products / services or generated due to the use of products / services, as well as the personal information obtained with the user's authorization.

[0101] The user personal information processed by the applicant will vary depending on the specific product / service scenario, and shall be based on the specific scenario in which the user uses the product / service, which may involve the user's account information, device information, driving information, vehicle information or other related information. The applicant will treat the user's personal information and its processing with a high degree of diligence.

[0102] The Applicant attaches great importance to the security of user personal information and has adopted reasonable and feasible security protection measures that comply with industry standards to protect user information and prevent personal information from being accessed, disclosed, used, modified, damaged or lost without authorization.

Claims

1. A road disease detection method based on a multimodal large model, characterized in that: include: Acquire image information of the road and text information about the type of damage in the image information; Outputting the text information to a semantic recognition model to determine semantic information; The image information and the semantic information are input into a multimodal large model to determine the road damage information.

2. The method according to claim 1, characterized in that The multimodal large model includes: a backbone network, a neck network and a detection head. The input of the backbone network is the image information. The backbone network is used to extract multi-dimensional feature information based on the image information and output the multi-dimensional feature information to the neck network. The input of the neck network is the multi-dimensional feature information and the semantic information. The neck network is used to determine fused feature information based on the multi-dimensional feature information and the semantic information and output the fused feature information to the detection head. The detection head is used to detect and output the road disease information based on the fused feature information.

3. The method according to claim 2, characterized in that The backbone network includes: multiple deformable convolution adaptive lightweight channel segmentation and rearrangement ALCSR modules, each ALCSR module is used to perform adaptive channel segmentation on the input feature information to obtain the feature information of each channel, and merge the feature information of each channel for output, each channel exchanges information between channels through a channel rearrangement mechanism, each channel is provided with a deformable convolution module, the deformable convolution module is used to change the shape of the feature information in the corresponding channel and then output it, the multiple ALCSR modules include: a first ALCSR module, a second ALCSR module, a third ALCSR module, a fourth ALCSR module and a fifth ALCSR module, the backbone network also includes: a first CBS module, a second CBS module, a third C BS module and the fourth CBS module, the first CBS module, the second CBS module, the third CBS module and the fourth CBS module are connected in sequence, the input of the first CBS module is the image information, the output of the fourth CBS module is the input of the first ALCSR module, the input of the second ALCSR module is the output of the first ALCSR module, the output of the second ALCSR module is the input of the third ALCSR module and the neck network, the output of the third ALCSR module is the input of the fourth ALCSR module, the output of the fourth ALCSR module is the input of the fifth ALCSR module and the neck network, and the output of the fifth ALCSR module is the input of the neck network.

4. The method according to claim 3, characterized in that The neck network includes: multiple image-text feature alignment TITFA modules based on attention mechanism, each TITFA module is used to fuse input semantic information and feature information, multiple TITFA modules include: a first TITFA module, a second TITFA module, and a third TITFA module, the neck network also includes: a semantic information interaction module based on attention mechanism, a first upsampling module, a first feature splicing module, a sixth ALCSR module, a second upsampling module, a second feature splicing module, a seventh ALCSR module, a fifth CBS module, a third feature splicing module, an eighth ALCSR module, a sixth CBS module, a fourth feature splicing module, and a ninth ALCSR module, the input of the semantic information interaction module is the output of the fifth ALCSR module, the output of the semantic information interaction module is the input of the first upsampling module, the input of the first feature splicing module is the output of the first upsampling module and the fourth ALCSR module, the output of the first feature splicing module is the input of the sixth ALCSR module, and the output of the sixth ALCSR module is the output of the second upsampling module and the third feature splicing module. The input of the first TITFA module is the input of the seventh ALCSR module, the output of the seventh ALCSR module is the input of the fifth CBS module and the detection head, the output of the fifth CBS module is the input of the third feature splicing module, the input of the second TITFA module is the semantic information and the output of the third feature splicing module, the output of the second TITFA module is the input of the eighth ALCSR module, the output of the eighth ALCSR module is the input of the sixth CBS module and the detection head, the input of the fourth feature splicing module is the output of the fifth ALCSR module and the sixth CBS module, the input of the third TITFA module is the semantic information and the output of the fourth feature splicing module, the output of the third TITFA module is the input of the ninth ALCSR module, and the output of the ninth ALCSR module is the input of the detection head.

5. The method according to claim 4, characterized in that The semantic information interaction module includes: multiple shape change units and multiple residual connection units, the shape change unit is used to change the shape of the feature map, the residual connection unit is used to retain the original information, the multiple shape change units include: a first shape change unit and a second shape change unit, the multiple residual connection units include: a first residual connection unit and a second residual connection unit, the semantic information interaction module also includes: a first multi-head attention unit, a first multi-layer perceptron unit, the input of the first shape change unit is the output of the fifth ALCSR module, the output of the first shape change unit is the input of the first multi-head attention unit, the first multi-head attention unit is used to process the input information based on the multi-head attention mechanism, and output feature information to the first residual connection unit, the first residual connection unit is used to residually connect the information input into the semantic information interaction module with the features output by the first multi-head attention unit, the output of the first residual connection unit is the input of the first multi-layer perceptron unit, the first multi-layer perceptron unit performs nonlinear mapping and outputs feature information to the second residual connection unit, the output of the second residual connection unit is the input of the second shape change unit, and the output of the second shape change unit is the input of the first upsampling module.

6. The method according to claim 5, characterized in that The TITFA module includes: a convolution unit, a third shape change unit, a second multi-head attention unit, a third residual connection unit, a second multi-layer perceptron unit, a fourth residual connection unit and a fourth shape change unit. The convolution unit is used to convolve the semantic information and output the convolved semantic information to the second multi-head attention unit. The third shape change unit is used to change the shape of the feature information of the image and output the changed feature information to the second multi-head attention unit. The second multi-head attention unit performs feature fusion based on the multi-head attention mechanism and outputs the fused feature information to the third residual connection unit. The output of the third residual connection unit is the second multi-layer perceptron unit, the output of the second multi-layer perceptron unit is the fourth residual connection unit, the output of the fourth residual connection unit is the input of the fourth shape change unit, and the fourth shape change unit is used to output the fused feature information.

7. The method according to claim 2, characterized in that The detection head includes: a lightweight image text feature fusion LITFF module, the LITFF module includes: a plurality of convolution units and a normalization unit, the convolution unit is used to perform dimensionality reduction processing and scale optimization on the image features of the fusion features, the normalization unit is used to normalize the features in the numerical space, the plurality of convolution units include: a first convolution unit and a second convolution unit, the plurality of normalization units include: a first normalization unit and a second normalization unit, the LITFF module also includes: an Einstein sum convention unit, a parameter adjustment unit and a splicing unit, the input of the first convolution unit and the second convolution unit is the fusion image information, the output of the first convolution unit is the input of the first normalization unit, the input of the second normalization unit is the semantic information, the input of the Einstein sum convention unit is the output of the first normalization unit and the second normalization unit, the output of the Einstein sum convention unit is the parameter adjustment unit, the input of the splicing unit is the output of the second convolution unit and the parameter adjustment unit, the splicing unit is used to output joint feature information, and perform detection based on the joint feature information.

8. A road disease detection device based on a multimodal large model, characterized in that: include: An acquisition module, used for acquiring image information of the road and text information about the disease type in the image information; A determination module, used for outputting the text information to a semantic recognition model to determine semantic information; The detection module is used to input the image information and the semantic information into a multimodal large model to determine the disease information of the road.

9. An electronic device, comprising: A memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images