A graph-text reasoning method, system, device and medium for intelligent transportation

Abstract

The application provides a picture-text reasoning method, system, device and medium for intelligent transportation, and belongs to the field of intelligent transportation. The method comprises the following steps: fine-tuning a general language-vision detection model to generate a closed set fine-tuned model, an open set continuous pre-training fine-tuned model and an open vocabulary fine-tuned model; training a router so that the router learns the corresponding relationship between picture-text training data and the closed set fine-tuned model, the open set continuous pre-training fine-tuned model and the open vocabulary fine-tuned model; using the trained router to select a suitable model of picture-text information from the closed set fine-tuned model, the open set continuous pre-training fine-tuned model and the open vocabulary fine-tuned model based on the input picture-text information; and using the suitable model of picture-text information to determine an image feature extractor, a text feature extractor and a detection head for picture-text reasoning, and performing picture-text reasoning on the picture-text information. The application can improve the performance of the roadside perception scene and realize high-robustness road test perception.

Description

Technical Field

[0001] This invention relates to the field of intelligent transportation technology, and more specifically, to a method, system, device, and medium for graph-text reasoning in intelligent transportation. Background Technology

[0002] As a core foundational technology in the fields of intelligent transportation and highway digitalization, roadside perception is undergoing a profound evolution from "seeing" to "understanding" and then to "predicting." By deploying various cameras along the roadside, roadside perception systems can capture the location, speed, type, and behavioral intentions of traffic participants in real time. At the same time, they can comprehensively monitor the status of road infrastructure, weather conditions, and emergencies, building a comprehensive perception network covering "people-vehicles-roads-environment."

[0003] In the practical application of intelligent transportation, roadside perception is no longer limited to traditional traffic flow statistics and video surveillance, but is deeply integrated into cutting-edge scenarios such as vehicle-to-everything (V2X), autonomous driving cloud control platforms, and traffic digital twins. By collaborating with edge computing and cloud intelligence, roadside perception systems can transform raw sensor data into structured traffic semantic information in real time, including target detection and tracking, traffic event recognition, and road condition assessment. This provides intelligent connected vehicles with a "God's-eye view" beyond line of sight, compensating for the limitations of single-vehicle intelligence in perception range and adverse environments.

[0004] Against the backdrop of the digital transformation of highways, roadside perception is becoming the data source for building holographic road networks and digital twin highways. Relying on high-precision, low-latency perception capabilities, roadside perception systems can establish dynamic digital mappings for highway infrastructure, enabling proactive early warning and intelligent decision-making regarding pavement defects, roadside facility damage, and traffic congestion. With the introduction of new technologies such as open-vocabulary detection and cross-modal large models, roadside perception is developing towards "open-set recognition" and "semantic generalization." It can not only accurately identify pre-set traffic signs and signal lights, but also flexibly handle unknown categories in open scenarios such as construction areas, temporary facilities, and abnormal debris, significantly improving the system's robustness and generalization capabilities in complex urban road and highway scenarios.

[0005] Existing roadside perception technologies have been deployed on a large scale in projects such as intelligent connected vehicle demonstration zones, smart highways, and urban holographic intersections. With the deepening of the national strategy of vehicle-road-cloud integration and the increasing maturity of the V2X industry ecosystem, roadside perception will move from technology verification to full-domain coverage, and upgrade from an auxiliary tool to the core engine of intelligent decision-making in transportation systems, providing a solid technological foundation for building a safe, efficient, and green future transportation system.

[0006] Vision-language models such as Grounding-DINO serve as general-purpose object detectors, capable of detecting specific targets based on input text. Roadside perception can detect specific targets in traffic scenarios, such as pedestrians on the road and roadside infrastructure (such as water-filled barriers), while vision-language models (VLMs) such as Grounding-DINO are not specific to roadside perception and therefore lack cognition for fixed roadside perception scenarios.

[0007] Therefore, the general models of existing intelligent transportation graph-text reasoning methods lack optimization for roadside perception scenarios, have poor performance, and lack robustness in roadside perception. Related technical solutions still need to be improved. Summary of the Invention

[0008] To address the aforementioned technical problems, this invention proposes a method, system, device, and medium for intelligent transportation graph-text reasoning, which can improve performance in roadside perception scenarios and achieve highly robust roadside perception.

[0009] The present invention provides a graph-text reasoning method for intelligent transportation in a first aspect, comprising the following steps: Fine-tuning a general language-visual detection model generates a closed-set fine-tuning model, an open-set continuous pre-training fine-tuning model, and an open-vocabulary fine-tuning model. The router is trained to learn the correspondence between the text and image training data and the closed-set fine-tuning model, the open-set continuous pre-training fine-tuning model, and the open-vocabulary fine-tuning model. The trained router selects the appropriate model for the input image and text information from closed-set fine-tuning models, open-set continuous pre-training fine-tuning models, and open-vocabulary fine-tuning models. The appropriate model for the image and text information is used to determine the image feature extractor, text feature extractor, and detection head for image and text reasoning, and to perform image and text reasoning on the image and text information.

[0010] In some embodiments, fine-tuning a general language-visual detection model to generate a closed-set fine-tuning model, an open-set continuous pre-training fine-tuning model, and an open-vocabulary fine-tuning model includes: The detection head of the language-visual detection model is fine-tuned to generate closed-set detection head parameters, and a closed-set fine-tuning model is generated based on the closed-set detection head parameters. The detection head and image feature extractor of the language-visual detection model are fine-tuned to generate open-set continuous pre-training-detection head parameters and open-set continuous pre-training-image feature extractor parameters, and an open-set continuous pre-training fine-tuning model is generated based on the open-set continuous pre-training-detection head parameters and open-set continuous pre-training-image feature extractor parameters. The detection head, image feature extractor, and text feature extractor of the language-visual detection model are fine-tuned to generate open vocabulary-detection head parameters, open vocabulary-image feature extractor parameters, and open vocabulary-text feature extractor parameters. Based on these parameters, an open vocabulary fine-tuning model is generated.

[0011] In some embodiments, enabling the router to learn the correspondence between text-based training data and closed-set fine-tuning models, open-set continuous pre-training fine-tuning models, and open-vocabulary fine-tuning models includes: The image and text training data were respectively input into the closed-set fine-tuning model, the open-set continuous pre-training fine-tuning model, and the open-vocabulary fine-tuning model to obtain three inference results. Three raw scores are obtained based on the intersection-union ratio of each of the three inference results with the truth value. The three raw scores are then normalized to obtain the supervision signal. The graphic training data is input into the router to obtain the original logic value. The router parameters are adjusted through backpropagation to fit the original logic value to the supervision signal in order to establish a correspondence.

[0012] In some embodiments, the applicable model for selecting image-text information based on input image-text information using a trained router from closed-set fine-tuning models, open-set continuous pre-training fine-tuning models, and open-vocabulary fine-tuning models includes: The trained router is used to calculate the inference scores of the closed-set fine-tuning model, the open-set continuous pre-training fine-tuning model, and the open-vocabulary fine-tuning model for text and image information, respectively. The router will select the model with the highest calculated inference score as the applicable model.

[0013] In some embodiments, an image feature extractor, a text feature extractor, and a detection head for performing graph-text reasoning are determined using an applicable model of the graph-text information. Performing graph-text reasoning on the graph-text information includes: In response to the applicable model being a closed-set fine-tuning model, image-text reasoning is performed on image-text information based on a detection head with closed-set-detection head parameters, as well as image feature extractors and text feature extractors in the language-visual detection model. In response to the applicable model being an open-set continuously pre-trained fine-tuning model, image-text reasoning is performed on image-text information based on a detector head with open-set continuously pre-trained-detector parameters, an image feature extractor with open-set continuously pre-trained-image feature extractor parameters, and a text feature extractor in the language-visual detection model. In response to the applicable model being an open vocabulary fine-tuning model, image-text reasoning is performed on image-text information based on a detector with open vocabulary-detector parameters, an image feature extractor with open vocabulary-image feature extractor parameters, and a text feature extractor with open vocabulary-text feature extractor parameters.

[0014] In some embodiments, performing image-text reasoning on image-text information includes: Image features are determined based on the input image in the text and image information using an image feature extractor; Text features are determined based on the input text in the image and text information using a text feature extractor; Feature enhancement is used to determine modality fusion features based on image and text features; The detection head uses modality fusion features to determine the bounding box and category as the result of graph-text reasoning.

[0015] In some embodiments, the feature enhancement neck is a feature enhancement neck in a language-visual detection model.

[0016] A second aspect of the present invention provides a text-image reasoning system for intelligent transportation, comprising: The fine-tuning module fine-tunes a general language-visual detection model to generate closed-set fine-tuning models, open-set continuous pre-trained fine-tuning models, and open-vocabulary fine-tuning models. The training module trains the router, enabling it to learn the correspondence between the text and image training data and the closed-set fine-tuning model, the open-set continuous pre-training fine-tuning model, and the open-vocabulary fine-tuning model. The decision module uses a trained router to select the appropriate model for the input image and text information from three options: closed-set fine-tuning model, open-set continuous pre-training fine-tuning model, and open-vocabulary fine-tuning model. The reasoning module uses a suitable model for the image and text information to determine the image feature extractor, text feature extractor, and detection head for image and text reasoning, and performs image and text reasoning on the image and text information.

[0017] A third aspect of the present invention provides a computing device including a processor and a memory, the memory storing computer instructions executable on the processor, the instructions, when executed by the processor, implementing one or more steps in the aforementioned intelligent transportation graph-text reasoning method.

[0018] A fourth aspect of the present invention provides a computer-readable storage medium storing a computer program, characterized in that, when executed by a processor, the computer program implements one or more steps of the aforementioned intelligent transportation graph-text reasoning method.

[0019] This invention provides a proprietary model by fine-tuning a general visual-language model to meet the requirements of roadside perception scenarios. Multiple fine-tuned models are obtained using different fine-tuning strategies, each with a different focus and suitable for different scenarios or problems. When inference is required, a suitable fine-tuned model is selected based on the given scenario and text, thereby obtaining the optimal inference result without significantly increasing computational and memory overhead. This improves performance for roadside perception scenarios and achieves highly robust roadside perception. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other embodiments can be obtained based on these drawings without creative effort.

[0021] Figure 1 A flowchart of a graphical reasoning method for intelligent transportation provided in an embodiment of the present invention; Figure 2 The diagram shows the structure of the closed-set fine-tuning model of the intelligent transportation graph-text reasoning method provided in the embodiments of the present invention. Figure 3 This is a structural diagram of the open-set continuous pre-training and fine-tuning model of the intelligent transportation graph-text reasoning method provided in the embodiments of the present invention; Figure 4 The structure diagram of the open vocabulary fine-tuning model of the graph-text reasoning method for intelligent transportation provided in the embodiments of the present invention is shown. Figure 5 A schematic diagram illustrating the router selection of the applicable model for the graph-text reasoning method for intelligent transportation provided in this embodiment of the invention; Figure 6 This is a block diagram of a text-image reasoning system for intelligent transportation provided in an embodiment of the present invention. Detailed Implementation

[0022] To make the objectives, technical solutions, and advantages of the present invention clearer, the embodiments of the present invention will be further described in detail below with reference to specific examples and the accompanying drawings.

[0023] These embodiments are provided in this disclosure to make the disclosure thorough and complete, and to fully express the scope of the disclosure to those skilled in the art. It should be noted that, unless otherwise specifically stated, the relative arrangement of components and steps, the composition of materials, numerical expressions and values ​​set forth in these embodiments should be interpreted as exemplary only and not as limiting.

[0024] Furthermore, words such as "include" or "contain" mean that the element preceding the word covers the element listed after the word, and do not exclude the possibility that it may also cover other elements.

[0025] It should also be noted that, in the description of this disclosure, unless otherwise expressly specified and limited, the specific meaning of each term in this disclosure can be understood by those skilled in the art as appropriate. All terms used in this disclosure have the same meaning as understood by those skilled in the art to which this disclosure pertains, unless otherwise specifically defined. It should also be understood that, unless expressly defined herein, terms defined in a general dictionary should be interpreted as having a meaning consistent with their meaning in the context of the relevant art, and not as having an idealized or highly formalized meaning.

[0026] Figure 1 A flowchart illustrating a text-image reasoning method for intelligent transportation provided by an embodiment of the present invention is shown. Figure 1 As shown, the graph-text reasoning method for intelligent transportation includes the following steps: Step S101: Fine-tune the general language-visual detection model to generate a closed-set fine-tuning model, an open-set continuous pre-training fine-tuning model, and an open-vocabulary fine-tuning model.

[0027] In some embodiments, fine-tuning a general language-visual detection model to generate a closed-set fine-tuning model, an open-set continuous pre-training fine-tuning model, and an open-vocabulary fine-tuning model includes: fine-tuning the detection head of the language-visual detection model to generate closed-set detection head parameters, and generating a closed-set fine-tuning model based on the closed-set detection head parameters; fine-tuning the detection head and image feature extractor of the language-visual detection model to generate open-set continuous pre-training detection head parameters and open-set continuous pre-training image feature extractor parameters, and generating an open-set continuous pre-training fine-tuning model based on the open-set continuous pre-training detection head parameters and open-set continuous pre-training image feature extractor parameters; and fine-tuning the detection head, image feature extractor, and text feature extractor of the language-visual detection model to generate open-vocabulary detection head parameters, open-vocabulary image feature extractor parameters, and open-vocabulary text feature extractor parameters, and generating an open-vocabulary fine-tuning model based on the open-vocabulary detection head parameters, open-vocabulary image feature extractor parameters, and open-vocabulary text feature extractor parameters.

[0028] The closed-set fine-tuning model in this embodiment of the invention is as follows: Figure 2 As shown, closed-set fine-tuning follows the traditional paradigm, where the model is trained only on categories included in the labeled dataset. The detection head is active (in... Figure 2(shown as a flame symbol in the image), thus aligning with the target category; the visual backbone and language encoder remain frozen (in...). Figure 2 (Illustrated with a frozen symbol) to preserve pre-trained semantic knowledge. This method achieves high recognition accuracy for known roadside targets (such as traffic signs or traffic lights), but cannot recognize new or evolving facilities.

[0029] Closed-set fine-tuning only requires fine-tuning the detection head, for example, using the Low-Rank Adaptation (LoRA) method, as shown in the following equation.

[0030]

[0031] Therefore, the closed-set fine-tuning model only needs to fine-tune and record the given labeled data. That's all.

[0032] Meanwhile, open-set continuous pre-training and fine-tuning of models, such as Figure 3 As shown, while retaining learned knowledge, the model enhances adaptability through continuous pre-training on an open set, selectively updating the Detection Head and Visual Backbone while keeping the Language Backbone constant. This allows visual features to be progressively refined and adapted to new environments while maintaining semantic foundations. By combining techniques such as negative sampling, the model can distinguish roadside-related areas from background clutter, thereby mitigating the hallucination problem. Therefore, this strategy enhances robustness and domain generalization ability in complex urban scenes.

[0033] Similarly, for open-set continuous pre-training and fine-tuning models, fine-tuning is performed using given labeled data, and the results are recorded. and .

[0034] In addition, open vocabulary fine-tuning models such as Figure 4 As shown, instead of relying on fixed category labels, the detection conditions are built upon natural language cues, enabling the model to associate unseen visual patterns with descriptive queries. The dataset can be divided into base classes (e.g., vehicles, pedestrians) and new classes (e.g., traffic lights, bollards). The model is trained only on the new classes and jointly evaluated on both base and new classes to test its generalization ability. Techniques such as open-vocabulary fine-tuning with controlled learning rates to jointly update the Detection Head, Detection Head, and Visual Backbone mitigate catastrophic forgetting, but base class detection performance degrades.

[0035] Similarly, for open-vocabulary fine-tuning models, fine-tuning is performed using given labeled data, and the results are recorded. , and .

[0036] The three models described above have similar structures, each with its own advantages and disadvantages. Roadside perception scenarios are relatively fixed, therefore, a suitable model can be selected based on the characteristics of the problem and scenario to achieve optimal performance.

[0037] Step S103: Train the router to learn the correspondence between the text and image training data and the closed-set fine-tuning model, the open-set continuous pre-training fine-tuning model, and the open-vocabulary fine-tuning model.

[0038] In some embodiments, enabling the router to learn the correspondence between text-based training data and the closed-set fine-tuning model, the open-set continuous pre-training fine-tuning model, and the open-vocabulary fine-tuning model includes: inputting text-based training data into the closed-set fine-tuning model, the open-set continuous pre-training fine-tuning model, and the open-vocabulary fine-tuning model respectively to obtain three inference results; obtaining three raw scores based on the intersection-union ratio (IUU) of each of the three inference results with the true value; normalizing the three raw scores to obtain a supervision signal; inputting text-based training data into the router to obtain raw logic values; and adjusting the router's parameters through backpropagation to fit the raw logic values ​​to the supervision signal to establish a correspondence.

[0039] The router in this embodiment of the invention is a lightweight model, including CNN, MLP, etc., and the input data is images and text. First, the given image and text problem is input into the three fine-tuning models mentioned above. Scores are then assigned based on the output results of the three models. For example, scores can be based on the Intersection over Union (IoU) between the detection results and the labels. Then, the scores are normalized, for example, through a simple summation normalization using the following formula:

[0040] in , , This represents the scores of the three fine-tuning models for a given text-image pair. During router training, the text-image pair and the given S are used to learn router-related parameters. Therefore, this embodiment of the invention achieves optimal inference results without significantly increasing computational and memory overhead by having the router select a suitable fine-tuning model from multiple models with different focuses and applicable scenarios based on the given scene and text during inference.

[0041] Step S105: Using the trained router, select the appropriate model for the image and text information from the closed-set fine-tuning model, the open-set continuous pre-training fine-tuning model, and the open-vocabulary fine-tuning model based on the input image and text information.

[0042] In some embodiments, selecting an applicable model for the input text-image information from a closed-set fine-tuning model, an open-set continuous pre-training fine-tuning model, and an open-vocabulary fine-tuning model using a trained router includes: calculating the inference scores of the closed-set fine-tuning model, the open-set continuous pre-training fine-tuning model, and the open-vocabulary fine-tuning model for the text-image information using the trained router; and determining the model with the highest calculated inference score as the applicable model.

[0043] In this embodiment of the invention, inference can be performed after the router training has converged (i.e., after the loss on the validation set no longer decreases). For example... Figure 5 As shown, during inference, a new input image-text pair is received, a very fast forward computation is performed, and the output is... Router selection The corresponding model is used as the applicable model (e.g.) If the maximum value is selected, then "open set continuous pre-trained fine-tuning model" is chosen. Only the selected expert model is fed into the input for inference; the other two models are not computed. Because the three fine-tuning models have the same structure, all based on LoRA fine-tuning, the number of parameters that need to be optimized and stored is small. Since the router is a lightweight network, the total increase in parameters is minimal, and the router's computational overhead is also low. Therefore, this embodiment of the invention can quickly determine the optimal fine-tuning model and complete inference based on the input image and text, taking into account the characteristics of different fine-tuning models to obtain the best detection results.

[0044] Step S107: Use the applicable model of the image and text information to determine the image feature extractor, text feature extractor and detection head for image and text reasoning, and perform image and text reasoning on the image and text information.

[0045] In some embodiments, an image feature extractor, a text feature extractor, and a detection head for performing graph-text reasoning are determined using an applicable model for the graph-text information. Performing graph-text reasoning on the graph-text information includes: performing graph-text reasoning on the graph-text information based on a detection head with closed-set-detection head parameters, and an image feature extractor and a text feature extractor in a language-visual detection model, in response to the applicable model being a closed-set continuous pre-training fine-tuning model; performing graph-text reasoning on the graph-text information based on a detection head with open-set continuous pre-training-detection head parameters, an image feature extractor with open-set continuous pre-training-image feature extractor parameters, and a text feature extractor in a language-visual detection model, in response to the applicable model being an open-vocabulary fine-tuning model; and performing graph-text reasoning on the graph-text information based on a detection head with open-vocabulary-detection head parameters, an image feature extractor with open-vocabulary-image feature extractor parameters, and a text feature extractor with open-vocabulary-text feature extractor parameters, in response to the applicable model being an open-vocabulary fine-tuning model.

[0046] In some embodiments, performing graph-text reasoning on graph-text information includes: using an image feature extractor to determine image features based on an input image in the graph-text information; using a text feature extractor to determine text features based on input text in the graph-text information; using a feature enhancement tool to determine modality fusion features based on the image features and text features; and using a detection head to determine a bounding box and category based on the modality fusion features as the result of the graph-text reasoning.

[0047] like Figure 2 , 3 As shown in Figure 4, the closed-set fine-tuning model, open-set continuous pre-training fine-tuning model, and open-vocabulary fine-tuning model of this invention all comprise four parts: a visual feature extractor, a text feature extractor, a feature enhancement neck, and a detection head. For a given image (I) and target text (T), it can be formally represented as follows: Feature I = Visual(I) Feature T = Language(T) FE = FeatureEnhance ([Feature I Feature T ]) [BBox, Cls]=DetectionHead (FE) Wherein, Visual is the image feature extractor, which extracts visual features from the input image I. IVisualization mainly includes structures such as CNN, MLP, and Transformer; Language is a text feature extractor that takes target text T as input and obtains text features. T Language primarily includes structures such as MLP and Transformer; FeatureEnhance is the feature enhancement head, which takes two features as input and obtains the fused features of the two modalities. FeatureEnhance mainly includes structures such as Cross-Attention and FFN to obtain the fused features of the two modalities; DetectionHead is the detection head, which takes the fused features as input and obtains the corresponding bounding box (BBox) and class (Cls). DetectionHead mainly includes structures such as CNN and MLP. The following formula can also be used to optimize each module: Loss = Metric(A, GT) GT stands for ground truth, and metric represents the deviation between the result and GT, including localization loss, contrast loss, cross-entropy loss, etc.

[0048] In some embodiments, the feature enhancement neck is a feature enhancement neck in a general language-visual detection model that has not been fine-tuned by a closed-set fine-tuning model, an open-set continuous pre-training fine-tuning model, or an open-vocabulary fine-tuning model.

[0049] The core function of the feature enhancement neck in this invention is to deeply fuse and align visual and textual features. During the pre-training phase, this neck has already learned how to accurately map "visual pixels" to "semantic words" on large-scale image-text pairs. Updating the neck during fine-tuning would disrupt this robust alignment bridge, leading to semantic shifts. Freezing the neck ensures that the underlying dialogue between the visual and language encoders remains unchanged regardless of subsequent adjustments.

[0050] In summary, this invention addresses the needs of roadside perception scenarios by fine-tuning a general vision-language model to obtain a proprietary model. Multiple fine-tuning models are derived using different fine-tuning strategies, each with a different focus, suitable for different scenarios or problems. Based on a given scenario and text, this invention selects an appropriate fine-tuning model to achieve optimal inference results without significantly increasing computational and memory overhead, while also ensuring overall robustness.

[0051] Figure 6 A block diagram of a text-image reasoning system for intelligent transportation provided by an embodiment of the present invention is shown. Figure 6 As shown, the intelligent transportation graph-text reasoning system includes: Fine-tuning module 61 fine-tunes a general language-visual detection model to generate a closed-set fine-tuning model, an open-set continuous pre-trained fine-tuning model, and an open-vocabulary fine-tuning model. Training module 62 trains the router so that it learns the correspondence between the text and image training data and the closed-set fine-tuning model, the open-set continuous pre-training fine-tuning model, and the open-vocabulary fine-tuning model. The decision module 63 uses a trained router to select the appropriate model for the image and text information from the closed-set fine-tuning model, the open-set continuous pre-training fine-tuning model, and the open-vocabulary fine-tuning model based on the input image and text information. The reasoning module 64 uses an applicable model of the image and text information to determine the image feature extractor, text feature extractor, and detection head for image and text reasoning, and performs image and text reasoning on the image and text information.

[0052] Finally, it should be noted that those skilled in the art will understand that all or part of the processes in the intelligent transportation graph-text reasoning method can be implemented by a computer program instructing related hardware. The program for the intelligent transportation graph-text reasoning method can be stored in a computer-readable storage medium. When executed, the program can include the processes described in the embodiments of the methods above. The storage medium for the program can be a magnetic disk, optical disk, read-only memory (ROM), or random access memory (RAM), etc. The embodiments of the computer program described above can achieve the same or similar effects as any of the corresponding foregoing method embodiments.

[0053] Furthermore, the intelligent transportation graph-text reasoning method disclosed in the embodiments of the present invention can also be implemented as a computer program executed by a processor, which can be stored in a computer-readable storage medium. When the computer program is executed by the processor, it performs the functions defined in the method disclosed in the embodiments of the present invention.

[0054] In one or more exemplary designs, functionality may be implemented in hardware, software, firmware, or any combination thereof. If implemented in software, functionality may be stored as one or more instructions or code on or transmitted via a computer-readable medium. Computer-readable media include computer storage media and communication media, including any medium that facilitates the transfer of a computer program from one location to another. Storage media may be any available medium accessible to a general-purpose or special-purpose computer. By way of example, and not limitation, computer-readable media may include RAM, ROM, EEPROM, CD-ROM or other optical disc storage devices, disk storage devices or other magnetic storage devices, or any other medium that may be used to carry or store the required program code in the form of instructions or data structures and is accessible to a general-purpose or special-purpose computer or a general-purpose or special-purpose processor. Furthermore, any connection may be appropriately referred to as computer-readable media. For example, if software is transmitted from a website, server, or other remote source using coaxial cable, fiber optic cable, twisted pair, digital subscriber line (DSL), or wireless technologies such as infrared, radio, and microwave, then the aforementioned coaxial cable, fiber optic cable, twisted pair, DSL, or wireless technologies such as infrared, radio, and microwave are all included in the definition of media. As used herein, disks and optical discs include compact discs (CDs), laser discs, optical discs, digital multifunction discs (DVDs), floppy disks, and Blu-ray discs, where disks typically reproduce data magnetically, while optical discs reproduce data optically using lasers. Combinations of the above should also be included within the scope of computer-readable media.

[0055] The above are exemplary embodiments disclosed in this invention. However, it should be noted that various changes and modifications can be made without departing from the scope of the embodiments of this invention as defined by the claims. The functions, steps, and / or actions of the methods according to the disclosed embodiments described herein do not need to be performed in any particular order. Furthermore, although the elements disclosed in the embodiments of this invention may be described or claimed individually, they may be understood as multiple unless explicitly limited to a singular number.

[0056] It should be understood that, as used herein, the singular form “a” is intended to include the plural form as well, unless the context clearly supports an exception. It should also be understood that, as used herein, “and / or” refers to any and all possible combinations of one or more of the associated listed items.

[0057] Those skilled in the art should understand that the discussion of any of the above embodiments is merely exemplary and is not intended to imply that the scope of the invention (including the claims) is limited to these examples. Within the framework of the invention, technical features of the above embodiments or different embodiments can be combined, and many other variations of different aspects of the invention exist, which are not provided in the details for the sake of brevity. Therefore, any omissions, modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the invention should be included within the protection scope of the invention.

Claims

1. A graphical reasoning method for intelligent transportation, characterized in that, Perform the following steps: Fine-tuning of a general language-visual detection model to generate closed-set fine-tuning models, open-set continuous pre-training fine-tuning models, and open-vocabulary fine-tuning models; The router is trained to learn the correspondence between the text and image training data and the closed-set fine-tuning model, the open-set continuous pre-training fine-tuning model, and the open-vocabulary fine-tuning model; The trained router selects the appropriate model for the input image and text information from the closed-set fine-tuning model, the open-set continuous pre-training fine-tuning model, and the open-vocabulary fine-tuning model. The applicable model for the image and text information is used to determine the image feature extractor, text feature extractor, and detection head for image and text reasoning, and to perform image and text reasoning on the image and text information.

2. The method according to claim 1, characterized in that, Fine-tuning a general language-visual detection model to generate closed-set fine-tuning models, open-set continuous pre-training fine-tuning models, and open-vocabulary fine-tuning models includes: The language-visual detection model is fine-tuned to generate closed-set detection head parameters, and the closed-set fine-tuning model is further generated. The language-visual detection model is fine-tuned to generate open-set continuous pre-training-detection head parameters and open-set continuous pre-training-image feature extractor parameters, and the open-set continuous pre-training fine-tuning model is further generated. The language-visual detection model is fine-tuned to generate open vocabulary-detection head parameters, open vocabulary-image feature extractor parameters, and open vocabulary-text feature extractor parameters, and the open vocabulary fine-tuning model is further generated.

3. The method according to claim 1, characterized in that, The correspondence between the router learning the text and image training data and the closed-set fine-tuning model, the open-set continuous pre-training fine-tuning model, and the open-vocabulary fine-tuning model includes: Based on the intersection-union ratio (IUU) of the three inference results obtained from the image and text training data through three different models with the true value, three raw scores are obtained and normalized to obtain the supervision signal; The image and text training data is input into the router to obtain the original logical value. The parameters of the router are adjusted through backpropagation so that the original logical value fits the supervision signal.

4. The method according to claim 3, characterized in that, Using the trained router, the applicable model for selecting the text-image information from the closed-set fine-tuning model, the open-set continuous pre-training fine-tuning model, and the open-vocabulary fine-tuning model based on the input text-image information includes: The trained router is used to calculate the inference scores of the closed-set fine-tuning model, the open-set continuous pre-training fine-tuning model, and the open-vocabulary fine-tuning model for the text and image information, respectively. The router then identifies the model with the highest calculated inference score as the applicable model.

5. The method according to claim 2, characterized in that, Using the applicable model of the image and text information, an image feature extractor, a text feature extractor, and a detection head are determined for image and text reasoning. Image and text reasoning of the image and text information includes: In response to the applicable model being the closed-set fine-tuning model, text-image reasoning is performed based on the detection head having the closed-set-detection head parameters; In response to the applicable model being the open-set continuous pre-training fine-tuning model, image-text reasoning is performed based on the detector head having the open-set continuous pre-training-detector parameters and the image feature extractor having the open-set continuous pre-training-image feature extractor parameters; In response to the applicable model being the open vocabulary fine-tuning model, image-text reasoning is performed based on the detector head having the open vocabulary-detector parameters, the image feature extractor having the open vocabulary-image feature extractor parameters, and the text feature extractor having the open vocabulary-text feature extractor parameters.

6. The method according to claim 1, characterized in that, The image-text reasoning process for the aforementioned image-text information includes: The image feature extractor is used to determine image features based on the input image in the image and text information; The text feature extractor is used to determine text features based on the input text in the image and text information; The feature enhancement method is used to determine modality fusion features based on the image features and the text features; The detection head is used to determine the bounding box and category based on the modality fusion features as the result of graph-text reasoning.

7. The method according to claim 6, characterized in that, The feature enhancement neck is the feature enhancement neck in the language-visual detection model.

8. A text-image reasoning system for intelligent transportation, characterized in that, include: The fine-tuning module fine-tunes a general language-visual detection model to generate closed-set fine-tuning models, open-set continuous pre-trained fine-tuning models, and open-vocabulary fine-tuning models. The training module trains the router so that the router learns the correspondence between the text and image training data and the closed-set fine-tuning model, the open-set continuous pre-training fine-tuning model, and the open-vocabulary fine-tuning model. The determination module uses the trained router to select the appropriate model for the image and text information from the closed-set fine-tuning model, the open-set continuous pre-training fine-tuning model, and the open-vocabulary fine-tuning model based on the input image and text information. The inference module uses the applicable model of the image and text information to determine an image feature extractor, a text feature extractor, and a detection head for image and text inference, and performs image and text inference on the image and text information.

9. A computing device, characterized in that, It includes a processor and a memory, the memory storing computer instructions executable on the processor, the instructions, when executed by the processor, implementing one or more steps of the intelligent transportation graph-text reasoning method according to any one of claims 1-7.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by the processor, it implements one or more steps of the intelligent transportation graph-text reasoning method according to any one of claims 1-7.

Images