Vehicle information identification method and device based on large model

Abstract

The invention relates to the technical field of computer vision, in particular to a vehicle information identification method and device based on a large model. Obtaining position information of each vehicle included in each to-be-detected image frame in the video, and determining an identifier corresponding to each vehicle according to the position information and a target tracking algorithm; for each identifier, determining that the video comprises a target image frame of a target vehicle corresponding to the identifier; and the target image frame is analyzed based on the first multi-modal large model, an identification result corresponding to each piece of preset vehicle information is determined, and the preset vehicle information is a vehicle color, a vehicle type, a driving direction, a license plate color, a license plate number or a vehicle brand. According to the invention, the vehicle in the to-be-detected image frame of the video is detected and tracked by using the small model, and then the preset vehicle information of each identified vehicle is identified by using the multi-modal large model, so that the identification efficiency is improved on the premise of ensuring the vehicle information identification effect.

Description

Technical Field

[0001] The present application relates to the field of computer vision technology, and in particular to a vehicle information recognition method and device based on a large model. Background Art

[0002] In recent years, with the development of social economy and the continuous advancement of urbanization, the number of motor vehicles and traffic monitoring has increased dramatically. Faced with the massive amount of traffic monitoring data in cities, it is no longer possible to efficiently collect, analyze and apply vehicle data using manpower alone. Using computer vision technology to analyze video data has become the current mainstream. Extracting vehicle information, that is, structured information such as vehicle color, type, license plate color, and license plate number, can effectively analyze the flow and distribution of different types of vehicles, helping urban planners to formulate more scientific traffic planning and infrastructure construction plans; it can quickly lock and track the vehicles involved in the case, improve the efficiency of solving the case, and enhance urban public safety; it can assist in the identification and handling of traffic accident responsibilities by extracting vehicle information at the scene of the accident, and provide strong evidence support for the determination of accident responsibilities.

[0003] At present, a single traditional small model cannot complete the extraction of all vehicle information. A small model can often only extract one type of vehicle information. Some complex information requires two or even more small models to process. For example, license plate number recognition requires the use of both target detection and optical character recognition (OCR) technologies. If a small model is used for vehicle information recognition, many small models need to be connected in parallel to achieve it. On the other hand, the generalization performance and robustness of the small model are poor, and the recognition accuracy is easily affected by the environment and noise, resulting in the model outputting incorrect results.

[0004] Therefore, how to improve the efficiency and accuracy of vehicle information recognition has become an urgent problem to be solved. Summary of the invention

[0005] The embodiments of the present application provide a vehicle information recognition method and device based on a large model to solve the problem of low efficiency and accuracy of vehicle information recognition in the prior art.

[0006] The present application provides a vehicle information recognition method based on a large model, the method comprising:

[0007] Obtaining the position information of each vehicle included in each image frame to be detected in the video, and determining the identification corresponding to each vehicle according to the position information and the target tracking algorithm;

[0008] For each logo, determine a target image frame of the target vehicle corresponding to the logo in the video; analyze the target image frame based on the first multimodal large model to determine the recognition result corresponding to each preset vehicle information, where the preset vehicle information is vehicle color, vehicle type, driving direction, license plate color, license plate number or vehicle brand.

[0009] The above technical solution obtains the position information of each vehicle included in each image frame to be detected in the video, and determines the identification corresponding to each vehicle based on the position information and the target tracking algorithm; for each identification, determines the target image frame of the target vehicle corresponding to the identification in the video; analyzes the target image frame based on the first multimodal large model to determine the recognition result corresponding to each preset vehicle information, and the preset vehicle information is vehicle color, vehicle type, driving direction, license plate color, license plate number or vehicle brand. First, the small model is used to detect and track the vehicles in the image frames to be detected in the video, and then the multimodal large model is used to identify the preset vehicle information of each identified vehicle, thereby improving the recognition efficiency while ensuring the vehicle information recognition effect.

[0010] Furthermore, the process of determining each image frame to be detected in the video includes:

[0011] In the video, image frames at intervals of a preset time length are determined as image frames to be detected.

[0012] The above technical solution takes frames at intervals in the video according to a preset time length. Under the premise of ensuring the recognition effect, it reduces the number of image frames to be detected by the small model and effectively improves the efficiency of vehicle information recognition.

[0013] Furthermore, after acquiring the position information of each vehicle included in each image frame to be detected in the video, and before determining the identifier corresponding to each target according to the position information and the target tracking algorithm, the method further includes:

[0014] Determine, according to the position information of each vehicle, a first area occupied by each vehicle in a corresponding image frame to be detected;

[0015] If the first area does not meet the preset area threshold requirement, the position information of the corresponding vehicle is deleted in the corresponding image frame to be detected.

[0016] In the above technical solution, since the vehicle details of a vehicle with a smaller area in the image are not very clear, even if the vehicle information of the vehicle with a smaller area is identified, the accuracy of the identification result will not be very high. Therefore, in an embodiment of the present application, the position information of the vehicle whose first area does not meet the preset area threshold requirement is deleted, and the vehicle is not detected subsequently, which can effectively improve the efficiency and accuracy of vehicle information recognition.

[0017] Furthermore, after determining the identifier corresponding to each target, and before determining, for each identifier, a target image frame including the target vehicle corresponding to the identifier in each image frame to be detected, the method further includes:

[0018] If the vehicle corresponding to any logo is not recognized again within a set time length after being recognized for the first time, the logo will be deleted.

[0019] In the above technical solution, if the vehicle corresponding to any identification is not recognized again within the set time length after being recognized for the first time, it is considered that the vehicle has driven out of the monitoring area, and the identification is deleted. The vehicle corresponding to the identification will no longer be detected subsequently, which releases the cache and saves memory and computing power consumption during the operation of the algorithm.

[0020] Furthermore, the target image frame includes the first image frame and the last image frame in which the target vehicle appears in each of the image frames to be detected;

[0021] The analyzing the target image frame based on the first multimodal large model to determine the recognition result corresponding to each preset vehicle information includes:

[0022] Determining a first prompt text according to the preset vehicle information and the preset prompt word template;

[0023] Inputting the first prompt text and the first image frame into the first multimodal large model, obtaining a recognition result corresponding to each preset vehicle information output by the first multimodal large model, and determining the recognition result corresponding to the driving direction in the preset vehicle information as the recognition result of the entry direction;

[0024] The second prompt text and the last image frame are input into the first multimodal large model to obtain a recognition result corresponding to the driving direction output by the first multimodal large model, and the recognition result is determined as the recognition result of the exit direction, wherein the second prompt text is at least used to prompt the first multimodal large model to determine the driving direction of the target vehicle.

[0025] The above technical solution uses the first multimodal large model to process the first image frame in which the target vehicle appears, obtains the recognition result corresponding to each preset vehicle information, and determines the recognition result corresponding to the driving direction as the recognition result of the entry direction, and then uses the first multimodal large model to process the last image frame in which the target vehicle appears, obtains the recognition result corresponding to the driving direction, and determines the recognition result corresponding to the driving direction as the recognition result of the exit direction. Even if the target vehicle changes its driving direction in the monitoring area, the entry direction and the exit direction can be accurately identified, thereby accurately tracking the target vehicle.

[0026] Furthermore, after obtaining the recognition result corresponding to each preset vehicle information, the method further includes:

[0027] If there is preset vehicle information for which the first multimodal large model has not obtained a recognition result, the preset vehicle information is determined as a first re-inspection element, and a third prompt text is determined according to the first re-inspection element and the preset prompt word template;

[0028] Determine any image frame to be detected that includes the target vehicle and is located between the first image frame and the last image frame as a re-inspection image frame;

[0029] The third prompt text and the re-inspection image frame are input into the first multimodal large model to obtain a recognition result corresponding to the first re-inspection element.

[0030] The above technical scheme determines the preset vehicle information that has not obtained the recognition result as the first re-inspection factor, and dynamically determines the third prompt text according to the first re-inspection factor and the preset prompt word template, so as to use the first multimodal large model to re-inspect the intermediate image frame where the target vehicle appears, so as to ensure that all the preset vehicle information of the target vehicle is recognized as much as possible, thereby improving the accuracy of target vehicle recognition.

[0031] Furthermore, before inputting the second prompt text and the last image frame into the first multimodal large model, the method further includes:

[0032] If there is currently preset vehicle information for which no recognition result has been obtained, the preset vehicle information for which no recognition result has been obtained is determined as the second re-inspection element;

[0033] Determining the second prompt text according to the second re-inspection element, the driving direction and the preset prompt word template;

[0034] The step of inputting the second prompt text and the last image frame into the first multimodal large model to obtain a recognition result corresponding to the driving direction includes:

[0035] The second prompt text and the last image frame are input into the first multimodal large model to obtain a recognition result corresponding to the driving direction and a recognition result corresponding to the second re-inspection element.

[0036] The above technical solution uses the first multimodal large model to identify the last image frame, obtains the recognition result corresponding to the driving direction and the second re-inspection element, and further ensures that all preset vehicle information of the target vehicle information is recognized.

[0037] Furthermore, the fine-tuning training process of the first multimodal large model includes:

[0038] Acquire a sample image and a standard result corresponding to each preset vehicle information of a sample vehicle in the sample image;

[0039] Randomly selecting at least one target vehicle information from each preset vehicle information, and determining a fourth prompt text according to the target vehicle information and a preset prompt word template;

[0040] Determine a sample answer based on the standard result corresponding to the target vehicle information and a preset answer template;

[0041] The pre-trained first multimodal large model is fine-tuned based on the sample image, the fourth prompt text, and the sample answer to obtain the first multimodal large model.

[0042] The above technical solution builds a diverse, input-output formatted instruction fine-tuning dataset, and enhances the detection capability and the ability to output in a set format through LoRA fine-tuning

[0043] Further, each preset vehicle information includes simple elements and complex elements, the simple elements include vehicle color, driving direction, license plate color and license plate number, and the complex elements include vehicle type and vehicle brand;

[0044] The process of determining the standard result corresponding to each preset vehicle information of the sample vehicle in the sample image includes:

[0045] Identify a sample vehicle in the sample image based on a small target detection model;

[0046] For each simple element, based on the small recognition model saved for the simple element, determine a standard result corresponding to the simple element of the sample vehicle in the sample image;

[0047] The fifth prompt text and the sample image are input into the second multimodal model to obtain a standard result corresponding to the complex element.

[0048] The above technical solution adopts a combination of large and small models to pre-label data with models and algorithms, thereby improving labeling efficiency.

[0049] The present application provides a vehicle information recognition device based on a large model, the device comprising:

[0050] A determination module, used to obtain the position information of each vehicle included in each image frame to be detected in the video, and determine the identification corresponding to each vehicle according to the position information and the target tracking algorithm;

[0051] The recognition module is used to determine, for each identification, a target image frame of a target vehicle corresponding to the identification in the video; analyze the target image frame based on the first multimodal large model to determine a recognition result corresponding to each preset vehicle information, wherein the preset vehicle information is vehicle color, vehicle type, driving direction, license plate color, license plate number or vehicle brand.

[0052] The present application also provides an electronic device, which includes a processor, and the processor is used to implement the steps of any of the above-mentioned large model-based vehicle information recognition methods when executing a computer program stored in a memory.

[0053] The present application also provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any of the above-mentioned large-model-based vehicle information recognition methods. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] In order to more clearly illustrate the technical solution of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0055] Figure 1 A schematic diagram of a process flow of a vehicle information recognition process based on a large model provided in an embodiment of the present application;

[0056] Figure 2 A schematic diagram of constructing a fourth prompt text provided in an embodiment of the present application;

[0057] Figure 3 An overall schematic diagram of a vehicle information identification method provided in an embodiment of the present application;

[0058] Figure 4 A schematic diagram of a vehicle information recognition process based on a large model provided in an embodiment of the present application;

[0059] Figure 5 A schematic diagram of vehicle information display provided in an embodiment of the present application;

[0060] Figure 6 A schematic diagram of the structure of a vehicle information recognition device based on a large model provided in an embodiment of the present application;

[0061] Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the embodiment of the present application will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiment is only a part of the embodiment of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field belong to the scope of protection of this application.

[0063] In view of the problems that the vehicle traffic element detection method based on small models in smart traffic scenarios requires parallel processing of multiple small models, poor algorithm robustness and low detection effect, the embodiments of the present application provide a vehicle information recognition method and device based on a large model. In this method, the position information of each vehicle included in each image frame to be detected in the video is obtained, and the identification corresponding to each vehicle is determined according to each position information and the target tracking algorithm; for each identification, the target image frame of the target vehicle corresponding to the identification is determined in the video; based on the first multimodal large model, the target image frame is analyzed to determine the recognition result corresponding to each preset vehicle information, and the preset vehicle information is vehicle color, vehicle type, driving direction, license plate color, license plate number or vehicle brand.

[0064] Figure 1 A flow chart of a vehicle information recognition process based on a large model provided in an embodiment of the present application is shown as follows: Figure 1 As shown, the process includes the following steps:

[0065] S101: Acquire the position information of each vehicle included in each image frame to be detected in the video, and determine the identification corresponding to each vehicle according to the position information and the target tracking algorithm.

[0066] The large model-based vehicle information recognition method provided in the present application is applied to an electronic device, which may be a server, a PC, a mobile terminal, etc.

[0067] At present, although the multimodal large model has superior video image understanding ability and semantic understanding ability, and can accurately detect the traffic element information of the vehicle according to the input prompt, the use of the multimodal large model alone to identify the vehicle information cannot meet the requirements of real-time processing. Because the multimodal large model needs to perform extremely high calculations when performing tasks, and the processing task time is relatively long. When the multimodal large model processes the video, it requires a longer processing time. Therefore, in order to accurately and efficiently identify the vehicle information, in an embodiment of the present application, the fast detection ability of the small model and the precise detection ability of the multimodal large model are combined to identify the vehicle in the received video. In an embodiment of the present application, the preset vehicle information to be identified can be saved, and the recognition result corresponding to the preset vehicle information is identified when the identification is performed later. Among them, the preset vehicle information can be vehicle color, vehicle type, driving direction, license plate color, license plate number or vehicle brand. When performing vehicle information identification, one or more preset vehicle information can be identified at one time. Those skilled in the art can also configure the preset vehicle information to be identified as needed.

[0068] When identifying vehicle information, a video can be obtained, which can be a video containing a vehicle collected by an image acquisition device installed at any position. The video can be an offline video or a real-time video. The offline video can be historical data collected and saved by the image acquisition device, for example, video data collected from the 10th to the 20th of last month. The real-time video can be a video being collected by the image acquisition device. In an embodiment of the present application, each image frame in the video can be determined as an image frame to be detected, and the position information of each vehicle included in each image frame to be detected is obtained. In other words, the vehicle included in each image frame to be detected is detected. Among them, the position information can be the coordinate information of the detection frame of the corresponding vehicle. In an embodiment of the present application, the position of each vehicle can be marked in the corresponding image frame to be detected using a detection frame. When identifying the position information of each vehicle, a small model can be used to process each image frame to be detected. Exemplarily, YOLOV10 can be used to obtain the position information of all vehicles appearing in each image frame to be detected.

[0069] In order to facilitate the analysis of each vehicle, in an embodiment of the present application, the identification corresponding to each vehicle can be determined. In an embodiment of the present application, when determining the identification corresponding to each vehicle, the motion trajectory of each vehicle in the image frame to be detected can be determined based on the target tracking algorithm. In other words, it is determined which vehicles in different image frames belong to the same vehicle. Optionally, the target tracking algorithm used to determine the motion trajectory can be a BoT-SORT algorithm.

[0070] After obtaining the motion trajectory of each target, it is possible to determine which targets in different image frames belong to the same target, and to assign an identifier to each running trajectory, which can be used as the identifier of the corresponding vehicle. In other words, the identifier of the same vehicle in different image frames is the same, and the identifier can be used to uniquely identify the corresponding vehicle.

[0071] S102: For each logo, determine a target image frame of a target vehicle corresponding to the logo in the video; analyze the target image frame based on the first multimodal large model to determine a recognition result corresponding to each preset vehicle information, wherein the preset vehicle information is vehicle color, vehicle type, driving direction, license plate color, license plate number or vehicle brand.

[0072] After obtaining the identification corresponding to each vehicle in each image frame to be detected, it is determined which vehicle in different image frames to be detected belongs to the same vehicle. In an embodiment of the present application, for each identification, a target image frame including the target vehicle corresponding to the identification in the video can be determined. Specifically, a set number of image frames including the target vehicle corresponding to the identification can be selected in the video, and the selected image frames are referred to as target image frames. The set number can be any positive integer, for example, 1, 2, 5, etc. In other words, the number of target image frames determined can be one or more.

[0073] In order to be able to identify multiple vehicle information of the target vehicle at one time, in an embodiment of the present application, the target image frame can be analyzed based on the first multimodal large model to determine the recognition result corresponding to each preset vehicle information. Among them, the preset vehicle information can be vehicle color, vehicle type, driving direction, license plate color, license plate number or vehicle brand. The first multimodal large model can recognize one or more preset vehicle information at a time. Exemplarily, the prompt text and the target image frame can be input into the first multimodal large model and recognized by the first multimodal large model. In order to make the first multimodal large model clear which vehicle is the target vehicle, in an embodiment of the present application, a labeling box can be used to mark the target vehicle. Exemplarily, the prompt text can be "Describe the following information of the vehicle in the picture: color, type, driving direction, license plate color, license plate number, brand. The results are output in a list format.".

[0074] In a possible implementation, in order to facilitate the subsequent viewing of the recognition results corresponding to each preset vehicle information corresponding to the target vehicle, in the embodiment of the present application, a traffic element dictionary with the content of [vehicle color, type, driving direction, license plate color, license plate number, brand] can be established for each identification, and all are initialized to unknown. After obtaining the recognition result corresponding to any preset vehicle information, the corresponding recognition result can be used to update the unknown of the corresponding element in the traffic element dictionary. In other words, the recognition result is saved in the traffic element dictionary.

[0075] It should be noted that since different image acquisition devices have different installation directions, in order to enable the first multimodal large model to accurately identify the driving direction of the target vehicle, the installation direction of the image acquisition device that captures the video can be input into the first multimodal large model, so that the first multimodal large model can accurately determine the driving direction of the target vehicle based on the installation direction.

[0076] In the embodiment of the present application, the position information of each vehicle included in each image frame to be detected in the video is obtained, and the identification corresponding to each vehicle is determined based on the position information and the target tracking algorithm; for each identification, the target image frame of the target vehicle corresponding to the identification is determined in the video; the target image frame is analyzed based on the first multimodal large model to determine the recognition result corresponding to each preset vehicle information, and the preset vehicle information is vehicle color, vehicle type, driving direction, license plate color, license plate number or vehicle brand. First, the small model is used to detect and track the vehicles in the image frames to be detected in the video, and then the multimodal large model is used to identify the preset vehicle information of each identified vehicle, thereby improving the recognition efficiency while ensuring the vehicle information recognition effect.

[0077] In order to further improve the efficiency of vehicle information recognition, based on the above embodiment, in the embodiment of the present application, the determination process of each image frame to be detected in the video includes:

[0078] In the video, image frames at intervals of a preset time length are determined as image frames to be detected.

[0079] Since the video acquisition frequency is relatively high, the posture change of the vehicle in adjacent image frames is not large. If each image frame in the video is identified, it will take up more computing power and the efficiency of vehicle information identification is low. Therefore, in order to further improve the efficiency of vehicle information identification, in the embodiment of the present application, when determining each image frame to be detected, the image frames with a preset time length in the video can be determined as the image frames to be detected.

[0080] Specifically, frames can be taken at 2-second intervals in the video. That is, in the video, image frames at 2-second intervals are determined as image frames to be detected. It should be noted that the preset time length can be a time length of any value, for example, 0.8 seconds, 2 seconds, 3 seconds, 3.4 seconds, etc., and those skilled in the art can configure it as needed.

[0081] In an embodiment of the present application, frames are taken at intervals in the video according to a preset time length. Under the premise of ensuring the recognition effect, the number of image frames to be detected by the small model is reduced, thereby effectively improving the efficiency of vehicle information recognition.

[0082] In order to further improve the accuracy of vehicle information recognition, on the basis of the above embodiments, in the embodiment of the present application, after obtaining the position information of each vehicle included in each image frame to be detected in the video, and before determining the identifier corresponding to each target according to the position information and the target tracking algorithm, the method further includes:

[0083] Determine, according to the position information of each vehicle, a first area occupied by each vehicle in a corresponding image frame to be detected;

[0084] If the first area does not meet the preset area threshold requirement, the position information of the corresponding vehicle is deleted in the corresponding image frame to be detected.

[0085] Since the collection range of the image acquisition device is limited, the accuracy of the images collected for vehicles that are far away from the image acquisition device is not very high. Therefore, in order to further improve the accuracy of vehicle information recognition, after determining the location information of each vehicle and before determining the corresponding identification of each vehicle, vehicles that are far away from the image acquisition device can be filtered.

[0086] Since the area occupied by vehicles closer to the image acquisition device is relatively large, while the area occupied by vehicles farther from the image acquisition device is relatively small, in the embodiment of the present application, the first area occupied by each vehicle in the corresponding image frame to be detected can be determined based on the position information of each vehicle. Thus, it is determined whether the corresponding vehicle is farther from the image acquisition device based on whether the first area meets the preset area threshold requirement.

[0087] Specifically, it can be determined whether the corresponding vehicle is far away from the image acquisition device according to whether the ratio of the first area to the total area of ​​the corresponding image frame to be detected is less than a preset threshold. If it is determined that the ratio is less than the preset threshold, it means that the corresponding vehicle is far away from the image acquisition device, and the position information of the corresponding vehicle can be deleted in the corresponding image frame to be detected. The preset threshold can be any decimal greater than 0 and less than 1, for example, the preset threshold is 0.4, 3%, etc.

[0088] Since the vehicle details of a vehicle with a smaller area in the image are not very clear, even if the vehicle information of the vehicle with a smaller area is identified, the accuracy of the identification result will not be very high. Therefore, in an embodiment of the present application, the position information of the vehicle whose first area does not meet the preset area threshold requirement is deleted, and the vehicle is not detected subsequently, which can effectively improve the efficiency and accuracy of vehicle information recognition.

[0089] In order to save resources, based on the above embodiments, in the embodiment of the present application, after determining the identifier corresponding to each target, and before determining, for each identifier, a target image frame including the target vehicle corresponding to the identifier in each image frame to be detected, the method further includes:

[0090] If the vehicle corresponding to any logo is not recognized again within a set time length after being recognized for the first time, the logo will be deleted.

[0091] In order to save memory and computing power during the operation of the algorithm, if a vehicle is not detected again within a set time length after being detected and tracked for the first time, it can be considered that the vehicle has left the monitoring area and no longer needs to be detected and tracked.

[0092] In the embodiment of the present application, after determining the identification corresponding to each target and before determining the target image frame corresponding to each identification, it can be determined whether the vehicle corresponding to each identification is recognized again within the set time length after being recognized for the first time. If not, it means that the vehicle may have driven out of the monitoring area of ​​the image acquisition device that shot the video, and the identification can be deleted. The set time length can be 3 minutes, 5 minutes, 3 minutes and 50 seconds, etc., and can be configured by those skilled in the art as needed.

[0093] In an embodiment of the present application, if a vehicle corresponding to any identification is not recognized again within a set time length after being first recognized, it is considered that the vehicle has left the monitoring area, and the identification is deleted. The vehicle corresponding to the identification will no longer be detected subsequently, which releases the cache and saves memory and computing power consumption during the algorithm operation.

[0094] In order to further improve the accuracy of vehicle information recognition, on the basis of the above embodiments, in the embodiment of the present application, the target image frame includes the first image frame and the last image frame in which the target vehicle appears in each image frame to be detected;

[0095] The analyzing the target image frame based on the first multimodal large model to determine the recognition result corresponding to each preset vehicle information includes:

[0096] Determining a first prompt text according to the preset vehicle information and the preset prompt word template;

[0097] Inputting the first prompt text and the first image frame into the first multimodal large model, obtaining a recognition result corresponding to each preset vehicle information output by the first multimodal large model, and determining the recognition result corresponding to the driving direction in the preset vehicle information as the recognition result of the entry direction;

[0098] The second prompt text and the last image frame are input into the first multimodal large model to obtain a recognition result corresponding to the driving direction output by the first multimodal large model, and the recognition result is determined as the recognition result of the exit direction, wherein the second prompt text is at least used to prompt the first multimodal large model to determine the driving direction of the target vehicle.

[0099] Since the direction of a vehicle's travel within a monitoring area may change, in order to know in which direction the vehicle continues to travel after leaving the monitoring area, so as to facilitate subsequent tracking of the vehicle, in an embodiment of the present application, when analyzing the target image frame based on the first multimodal large model and determining the recognition result corresponding to each preset vehicle information, the first image frame and the last image frame in which the target vehicle appears can be analyzed respectively. In an embodiment of the present application, the target image frame may include the first image frame and the last image frame in which the target vehicle appears in all the image frames to be detected.

[0100] In order to facilitate the determination of the prompt words input into the first multimodal large model, in an embodiment of the present application, a preset prompt word template may be saved. Subsequently, the preset vehicle information to be identified may be filled into the corresponding position of the preset prompt word template to obtain the prompt word text. Exemplarily, the preset prompt word template may be "Describe the following information of the vehicle in the picture: [area to be filled]. The result is output in a list format.".

[0101] When the first image frame is processed using the first multimodal large model, the preset vehicle information that needs to be identified can be filled into the to-be-filled area of ​​the prompt word template. Exemplarily, at this time, the first prompt text can be "Describe the following information of the vehicle in the picture: color, type, driving direction, license plate color, license plate number, brand. The results are output in a list format.". After obtaining the first prompt text, the first prompt text and the first image frame can be input into the first multimodal large model to obtain the recognition results corresponding to each preset vehicle information output by the first multimodal large model. Since this is the first image frame in which the target vehicle appears, the driving direction of the target vehicle at this time can be considered to be the input direction of the target vehicle. Therefore, in this embodiment of the present application, the recognition result corresponding to the driving direction in the preset vehicle information can be determined as the recognition result of the entry direction.

[0102] In order to determine the driving direction of the target vehicle when it leaves the monitoring area, in an embodiment of the present application, the first multimodal large model can be used to process the last image frame. Since the first multimodal large model is called to process the last image frame at this time in order to identify the driving direction of the target vehicle at this time. Therefore, in an embodiment of the present application, the "driving direction" can be determined as the vehicle information that currently needs to be identified, and the "driving direction" can be filled into the preset position of the preset prompt word template to obtain a second prompt text. The second prompt text is at least used to prompt the first multimodal large model to determine the driving direction of the target vehicle. After obtaining the second prompt text, the second prompt text and the last image frame are input into the first multimodal large model to obtain the recognition result corresponding to the driving direction output by the first multimodal large model, and the recognition result is determined as the recognition result of the exit direction.

[0103] In an embodiment of the present application, the first multimodal large model is used to process the first image frame in which the target vehicle appears, and the recognition result corresponding to each preset vehicle information is obtained, and the recognition result corresponding to the driving direction is determined as the recognition result of the entry direction. The first multimodal large model is then used to process the last image frame in which the target vehicle appears, and the recognition result corresponding to the driving direction is obtained, and the recognition result corresponding to the driving direction is determined as the recognition result of the exit direction. Even if the target vehicle changes its driving direction within the monitoring area, the entry direction and the exit direction can be accurately identified, thereby accurately tracking the target vehicle.

[0104] In order to fully identify the vehicle information, based on the above embodiments, in the embodiment of the present application, after obtaining the identification result corresponding to each preset vehicle information, the method further includes:

[0105] If there is preset vehicle information for which the first multimodal large model has not obtained a recognition result, the preset vehicle information is determined as a first re-inspection element, and a third prompt text is determined according to the first re-inspection element and the preset prompt word template;

[0106] Determine any image frame to be detected that includes the target vehicle and is located between the first image frame and the last image frame as a re-inspection image frame;

[0107] The third prompt text and the re-inspection image frame are input into the first multimodal large model to obtain a recognition result corresponding to the first re-inspection element.

[0108] Due to the posture of the target vehicle in a certain image frame to be detected, it may be impossible to identify all the preset vehicle information based on only one image frame to be detected. Therefore, in order to fully identify each vehicle information, in an embodiment of the present application, after obtaining the recognition result corresponding to each preset vehicle information recognized by the first multimodal large model, it can be determined whether there is preset vehicle information in the obtained recognition result that the first multimodal large model has not obtained the recognition result. If it exists, it means that the first multimodal large model has not recognized all the preset vehicle information, and it is necessary to recheck the preset vehicle information that has not obtained the recognition result. In an embodiment of the present application, the preset vehicle information that has not obtained the recognition result can be determined as the first recheck element. And the first recheck element is filled into the preset prompt word template to obtain the third prompt text.

[0109] In order to obtain target vehicles with different postures as much as possible, when identifying the first re-inspection element, any image frame to be detected that includes the target vehicle and is located between the first image frame and the last image frame can be determined as a re-inspection image frame. After obtaining the third prompt text and the re-inspection image frame, the third prompt text and the re-inspection image frame can be input into the first multimodal large model to obtain the recognition result corresponding to the first re-inspection element. After obtaining the recognition result corresponding to the first re-inspection element, the latest recognition result can be updated to the traffic element dictionary.

[0110] Specifically, the middle image frame to be detected including the target vehicle located between the first image frame and the last image frame can be determined as the re-inspection image frame. For example, there are 7 image frames to be detected including the target vehicle, and the 7 image frames to be detected are sorted according to the order of the 7 image frames to be detected in the video. The first image frame to be detected in the sequence is the first image frame, and the seventh image frame to be detected is the last image frame. Then, the fourth image frame to be detected in the sequence can be determined as the re-inspection image frame.

[0111] In an embodiment of the present application, the preset vehicle information that has not obtained an identification result is determined as the first re-inspection element, and the third prompt text is dynamically determined based on the first re-inspection element and the preset prompt word template, so that the intermediate image frame where the target vehicle appears is re-inspected using the first multimodal large model, so as to ensure that all the preset vehicle information of the target vehicle is identified as much as possible, thereby improving the accuracy of target vehicle identification.

[0112] In order to further ensure that the vehicle information is fully identified, based on the above embodiments, in the embodiment of the present application, before the second prompt text and the last image frame are input into the first multimodal large model, the method further includes:

[0113] If there is currently preset vehicle information for which no recognition result has been obtained, the preset vehicle information for which no recognition result has been obtained is determined as the second re-inspection element;

[0114] Determining the second prompt text according to the second re-inspection element, the driving direction and the preset prompt word template;

[0115] The step of inputting the second prompt text and the last image frame into the first multimodal large model to obtain a recognition result corresponding to the driving direction includes:

[0116] The second prompt text and the last image frame are input into the first multimodal large model to obtain a recognition result corresponding to the driving direction and a recognition result corresponding to the second re-inspection element.

[0117] If there is still preset vehicle information that has not obtained a recognition result when the last image frame is processed using the first multimodal large model, it is necessary to further identify the unrecognized preset vehicle information. If the recognition result of each preset vehicle information is obtained, then it is only necessary to identify the driving direction of the target vehicle in the last image frame. Therefore, in an embodiment of the present application, before the second prompt text and the last image frame are input into the first multimodal large model, it can also be determined whether there is preset vehicle information that has not obtained a recognition result. If so, the preset vehicle information that has not obtained a recognition result can be determined as the second re-inspection element.

[0118] In order to enable the first multimodal large model to identify the second re-check element and the driving direction, in an embodiment of the present application, the second prompt text can be determined based on the second re-check element, the driving direction and a preset prompt word template.

[0119] After obtaining the second prompt text, the second prompt text and the last image frame can be input into the first multimodal large model to obtain the recognition result corresponding to the driving direction and the recognition result corresponding to the second recheck element. In other words, the first multimodal large model is used to recognize the recognition result corresponding to the second recheck element and the driving direction at one time.

[0120] In the embodiment of the present application, the first multimodal large model is used to identify the last image frame to obtain the recognition result corresponding to the driving direction and the second re-inspection element, which further ensures that all preset vehicle information of the target vehicle information is recognized.

[0121] In order to further improve the accuracy of vehicle information recognition, based on the above embodiments, in the embodiment of the present application, the fine-tuning training process of the first multimodal large model includes:

[0122] Acquire a sample image and a standard result corresponding to each preset vehicle information of a sample vehicle in the sample image;

[0123] Randomly selecting at least one target vehicle information from each preset vehicle information, and determining a fourth prompt text according to the target vehicle information and a preset prompt word template;

[0124] Determine a sample answer based on the standard result corresponding to the target vehicle information and a preset answer template;

[0125] The pre-trained first multimodal large model is fine-tuned based on the sample image, the fourth prompt text, and the sample answer to obtain the first multimodal large model.

[0126] In order to further improve the accuracy of vehicle information recognition, when identifying vehicle information, the first multimodal large model that has been fine-tuned can be used for recognition. In an embodiment of the present application, the pre-trained first multimodal large model can be fine-tuned using sample data to obtain a first multimodal large model dedicated to vehicle information recognition, which can more accurately identify vehicle information.

[0127] During fine-tuning training, sample images and standard results corresponding to each preset vehicle information of sample vehicles in the sample images may be obtained. The sample images may be collected by relevant staff through various channels. The sample vehicles may be selected and annotated by relevant personnel in the sample images, and the standard results corresponding to each preset vehicle information may also be annotated by relevant personnel.

[0128] In order to make the first multimodal large model obtained by fine-tuning training more robust, when fine-tuning the pre-trained first multimodal large model, the pre-trained first multimodal large model can be made to process different amounts of preset vehicle information each time. In an embodiment of the present application, one or more target vehicle information can be randomly selected from each preset vehicle information, and the fourth prompt text can be determined based on the target vehicle information and the preset prompt word template. In other words, the randomly selected target vehicle information is filled into the preset position of the preset prompt word template to obtain the fourth prompt text.

[0129] Specifically, the preset prompt word template may be "describe the following information of the vehicle in the picture: {}, {}, ..., {}. The result is output in a list format." The preset vehicle information may include: color, type, driving direction, license plate color, license plate number, and brand. When constructing the fourth prompt text, 1 to N preset vehicle information may be randomly selected from the above preset vehicle information as the target vehicle information. After obtaining the target vehicle information, the target vehicle information is filled into "{}" of the above preset prompt word template to obtain the fourth prompt text.

[0130] Since the selection of target vehicle information is random and the order in which the target vehicle information is filled into the preset prompt word template is also uncertain, in order to match the target vehicle information with the corresponding standard results, in an embodiment of the present application, the corresponding standard results can be filled into the preset answer template in the order in which the target vehicle information is filled in the fourth prompt text to obtain a sample answer.

[0131] Specifically, the preset answer template may be “[{}, {}, …, {}]”, and the corresponding standard results are filled into the preset answer template in the order of the target vehicle information filled into the preset prompt word template to obtain a sample answer.

[0132] After obtaining the fourth prompt text and the sample answer, the pre-trained first multimodal large model can be fine-tuned based on the sample image, the fourth prompt text, and the sample answer to obtain the first multimodal large model. During the fine-tuning training process, a bypass can be added to the pre-trained first multimodal large model, and the pre-trained first multimodal large model is fine-tuned by the LoRA (Low-Rank Adaptation of Large Language Models) fine-tuning method in which the weights of the pre-trained first multimodal large model are frozen and remain unchanged, and only the bypass weights are updated, so as to enhance its ability to detect traffic elements and follow format answers.

[0133] The process of fine-tuning training is described below with reference to a specific embodiment. Figure 2 A schematic diagram of constructing a fourth prompt text provided for an embodiment of the present application. In the preliminary preparation process, after the vehicle information in the sample image is labeled, the fourth prompt text and sample answer corresponding to the sample image can be generated, and the multimodal large model can be fine-tuned using the LoRA method. In order to further enhance the multimodal large model's ability to recognize six vehicle information and adapt to the dynamic detection capabilities of large and small models, in the process of constructing the fourth prompt text, 1 to 6 elements can be randomly selected from the six traffic elements to construct the fourth prompt text and the corresponding sample answers, so that it can accurately answer any number of traffic element questions. In addition, in order to improve the response speed of the multimodal large model and the speed of parsing the results, the sample answers are designed to have only the corresponding traffic element information, and are constrained to a list format. Figure 2 As shown, Figure 2The picture on the middle left (left and right in the diagram) side is a sample image. The standard result based on the sample image is: the color of the sample vehicle in the sample image is white, the type is a sedan, the driving direction is south, the license plate color is blue, the license plate number is Lu F8****, and the brand is unknown. Among them, the brand is unknown, which means that the standard result corresponding to the vehicle information was not identified during the labeling process. Then, when constructing the fourth prompt text, the number N of standard results that are not unknown and the corresponding index INDEX can be obtained, and only these N valid labels are used to construct a question-answer pair, where the question-answer pair includes the fourth prompt text and the sample answer. Figure 2 As shown, the index INDEX corresponding to "color" can be determined as 0, the index INDEX corresponding to "type" can be determined as 1, the index INDEX corresponding to "form direction" can be determined as 2, and so on. Since the standard result corresponding to "brand" is unknown, when constructing the question-answer pair corresponding to the sample image, N=5, the index INDEX is 0, 1, 2, 3, 4 respectively. When constructing the fourth prompt text, a set number of index INDEX can be randomly selected, for example, 0, 1, 4 are randomly selected. According to the correspondence between the index INDEX and the vehicle information, it can be determined that the color, type, and license plate number are determined as the target vehicle information, and the target vehicle information is filled into the preset prompt word template, and the fourth prompt text is obtained as "Describe the following information of the vehicle in the picture: color, type, license plate number. The result is output in the format of a list.". In the order of color, type, and license plate number, the corresponding standard results are filled into the preset answer template, and the sample answer obtained is "[white, sedan, Lu F8****]".

[0134] In order to improve the efficiency of model fine-tuning training, on the basis of the above embodiments, in the embodiment of the present application, each preset vehicle information includes simple elements and complex elements, the simple elements include vehicle color, driving direction, license plate color and license plate number, and the complex elements include vehicle type and vehicle brand;

[0135] The process of determining the standard result corresponding to each preset vehicle information of the sample vehicle in the sample image includes:

[0136] Identify a sample vehicle in the sample image based on a small target detection model;

[0137] For each simple element, based on the small recognition model saved for the simple element, determine a standard result corresponding to the simple element of the sample vehicle in the sample image;

[0138] The fifth prompt text and the sample image are input into the second multimodal model to obtain a standard result corresponding to the complex element.

[0139] In order to improve the efficiency of model fine-tuning training, in the embodiment of the present application, the model can be used to determine the standard result corresponding to each preset vehicle information of the sample vehicle in the sample image. In the embodiment of the present application, the vehicle data can be efficiently annotated by using a labeling process of pre-labeling the sample data with a large and small model and manually revising the labeling twice, so as to provide a data basis for model training.

[0140] When determining the sample vehicle in the sample image, in an embodiment of the present application, the small target detection model YOLOV10 can be used to obtain all vehicles appearing in the sample image, and the area threshold is set to 3%. When the ratio of the vehicle detection frame area to the sample image area is less than the area threshold, the vehicle is considered to be too small to be suitable as training data and needs to be filtered out. The remaining vehicles in the sample image are determined as sample vehicles. Subsequently, the standard results of each preset element of the sample vehicle are identified based on the large model and the small model.

[0141] In an embodiment of the present application, the preset vehicle information can be divided into simple elements and complex elements according to the complexity and difficulty of element recognition, where simple elements may include vehicle color, driving direction, license plate color and license plate number, and complex elements may include vehicle type and vehicle brand.

[0142] When determining the standard result corresponding to each preset vehicle information, the sample image may be processed for each simple element based on the recognition small model saved for the simple element to obtain the standard result corresponding to the simple element. In other words, the simple element may be pre-labeled using the small model.

[0143] Specifically, for the recognition of vehicle color, a classification model based on Convolutional Neural Networks (CNN) can be used to obtain the color information of the vehicle. In an embodiment of the present application, the classification model can output the probability corresponding to each preset color, and determine the color with the highest probability as the vehicle color. In a possible implementation, in order to prevent ambiguity caused by the vehicle being composed of multiple colors, the color that occupies the largest surface area of ​​the vehicle can be used as the vehicle color.

[0144] For the recognition of driving direction, the driving direction can be divided into eight directions: east, west, south, north, northeast, southeast, northwest and southwest based on the vehicle's front direction and the surveillance shooting direction, and a CNN-based classification model can be used to obtain the vehicle's driving direction information.

[0145] For the recognition of vehicle color and license plate number, you can use the target detection algorithm YOLOV10 to obtain the license plate detection frame of the current vehicle, and then use the CNN-based classification model to obtain the license plate color information, and use the PaddleOCR algorithm to obtain the license plate number information. When the color or number cannot be recognized due to occlusion, viewing angle, lighting and other factors, the corresponding standard result is set to unknown.

[0146] The complex elements can be annotated using the second multimodal large model, for example, using the GPT4o application programming interface (Application Programming Interface, API) for pre-annotation. In an embodiment of the present application, the fifth prompt text and the sample image can be input into the second multimodal model to obtain the standard result corresponding to the complex elements.

[0147] Specifically, the fifth prompt text may be “Please describe the type and brand of the vehicle in the picture.” The GPT4o API is called to obtain the type and brand information of the vehicle, and the standard result corresponding to the unrecognizable elements is set to unknown.

[0148] After obtaining the standard results corresponding to each preset vehicle information of the sample vehicle, the standard results corresponding to each preset vehicle information can be manually corrected for the second time and arranged in the order of vehicle color, vehicle type, driving direction, license plate color, license plate number, and vehicle brand.

[0149] The vehicle information recognition process based on the large model is described below with reference to a specific embodiment. Figure 3 An overall schematic diagram of a vehicle information identification method provided in an embodiment of the present application is shown as follows: Figure 3As shown, the vehicle information recognition method adopts a framework combining large and small models, which mainly includes three parts: semi-automatic annotation of vehicle information, fine-tuning of multimodal large model LoRA, and dynamic detection of large and small models. In this application, six vehicle information, namely vehicle color, type, driving direction, license plate color, license plate number, and vehicle brand, are selected for detection according to the frequency of use and application value. The semi-automatic annotation part of vehicle information divides the six vehicle information into two categories, simple elements and complex elements, according to the complexity and difficulty of detection. The simple elements are pre-annotated using a small model, and the complex elements are pre-annotated using the API of GPT4o, and then the pre-annotations are manually corrected for a second time. In the fine-tuning part of the multimodal large model LoRA, when the question-answer pair is constructed, 1 to 6 elements are randomly selected from the six elements to construct the prompt text, and the output format is constrained, allowing only the required traffic elements to be output to improve the model response speed and result parsing speed. After the question-answer pair is constructed, the pre-trained multimodal large model is fine-tuned by LoRA. The dynamic detection part of the large and small models is based on the detection and tracking algorithm. It dynamically adjusts the prompt text of the multimodal large model according to the progress of vehicle information recognition to reduce the waste of computing resources. Specifically, the dynamic detection part of the large and small models jointly detects and tracks the small model and the multimodal large model to realize the dynamic detection of the six traffic elements of the vehicle, including vehicle detection and tracking and multimodal large model dynamic detection. Vehicle detection and tracking is mainly responsible for obtaining the location information of the vehicle in the video frame, and tracking the detected vehicle and assigning a unique identification (ID). The dynamic detection part of the multimodal large model is responsible for dynamically detecting the vehicle information of each tracking ID. The dynamic detection is divided into three steps (each tracked vehicle is detected up to three times). At different stages, the output of the multimodal large model is combined to adjust the number and category of specific sub-problems for constructing the prompt text.

[0150] The vehicle information recognition process based on the large model is described below in conjunction with another specific embodiment. Figure 4 A schematic diagram of a vehicle information recognition process based on a large model is provided in an embodiment of the present application, such as Figure 4 As shown, when performing vehicle information identification, the following steps are mainly included.

[0151] First, obtain the surveillance video. Take frames of the surveillance video stream at intervals of 2 seconds, use the target detection algorithm YOLOV10 to obtain the location information of all vehicles appearing in the current image frame to be detected, and filter out vehicles whose ratio of the detection frame area to the total area of ​​the image frame to be detected is not greater than 3%.

[0152] For vehicles with a total area ratio greater than 3%, the identification corresponding to each vehicle is determined based on the target tracking algorithm. If a vehicle is not detected again within five minutes after the first detection and tracking, it is considered that the vehicle has left the monitoring area and will not be detected and tracked.

[0153] For the remaining vehicles that need to be tracked, the first multimodal large model is used to perform detection up to three times respectively.

[0154] First detection: Use all preset vehicle information prompts - "Describe the following information of the vehicle in the image: color, type, driving direction, license plate color, license plate number, brand. The results are output in a list format." During the first detection process, the first image frame where each ID appears is detected, and the detection results are stored in the traffic element dictionary. If all vehicle information is identified, that is, only the value of the exit direction in the traffic element dictionary is unknown, the second detection is skipped and the tracking intermediate frame is not rechecked to reduce computing resource consumption.

[0155] Second detection: Only the first re-check element corresponding to the vehicle element that has not been identified is used to construct a prompt, and the intermediate frame of the vehicle tracking process of the ID is detected once, and the traffic element dictionary is updated. If there is still vehicle information that has not been identified at this time, it is considered that the vehicle has problems such as occlusion and the detection is suspended.

[0156] The third detection: To address the issue of changing driving direction, we build prompts based on the first and second detection results, and detect the last image frame that appears during the tracking process of the ID vehicle:

[0157] ① If all vehicle information needs to be identified at this time, that is, only the identification result of the exit direction in the traffic element dictionary is unknown, then only the driving direction is used to construct the prompt for detection, and the traffic element dictionary is updated; ② If there are still vehicles that have not been identified at this time, the unidentified vehicle element is determined as the second re-inspection element, and the driving direction and the second re-inspection element are used to construct the prompt for detection, and the traffic element dictionary is updated.

[0158] Figure 5 A schematic diagram of a vehicle information display provided in an embodiment of the present application, such as Figure 5 As shown, after completing the vehicle information recognition, any image frame containing the target vehicle can be displayed on the left side (left and right in the figure) of the display screen. In an embodiment of the present application, the image frame to be detected with the largest first area containing the target vehicle can be displayed. In order to facilitate the user to understand the detection result, the prompt text and the recognition result can be displayed on the right side (left and right in the figure) of the display screen, so that the user can understand the identified vehicle information and the corresponding recognition result.

[0159] The large-model-based vehicle information recognition method provided in this application selects six attributes of the vehicle: color, type, driving direction, license plate color, license plate number, and brand, and makes improvements in three aspects: data labeling, model training, and algorithm design. This method can effectively improve the detection accuracy and speed of vehicle traffic elements in smart traffic scenarios.

[0160] In terms of data labeling, according to the complexity and difficulty of vehicle information recognition, a small model and GPT4o's API are used for pre-labeling respectively. The labeling process of using models and algorithms to pre-label data and manually revising the labeling is used to improve the labeling efficiency; in terms of model training, a batch of diverse, formatted, high-quality vehicle traffic element data is constructed, and then the multimodal large model is fine-tuned on this basis to further improve the model's detection accuracy and ability to follow the set format to answer; in terms of algorithm design, the detection and tracking algorithm is used to detect and track all vehicles under monitoring, and the multimodal large model is used to identify traffic information of detected vehicles, and the prompt text input into the multimodal large model is dynamically adjusted according to the progress of vehicle information recognition.

[0161] Based on the same inventive concept, the embodiment of the present application also provides a vehicle information recognition device based on a large model, Figure 6 A schematic diagram of the structure of a vehicle information recognition device based on a large model provided in an embodiment of the present application, the device comprises:

[0162] The determination module 601 is used to obtain the position information of each vehicle included in each image frame to be detected in the video, and determine the identification corresponding to each vehicle according to the position information and the target tracking algorithm;

[0163] The recognition module 602 is used to determine, for each identification, a target image frame of a target vehicle corresponding to the identification in the video; analyze the target image frame based on the first multimodal large model to determine a recognition result corresponding to each preset vehicle information, wherein the preset vehicle information is vehicle color, vehicle type, driving direction, license plate color, license plate number or vehicle brand.

[0164] In a possible implementation manner, the determination module 601 is further configured to determine, in the video, image frames at intervals of a preset time length as image frames to be detected.

[0165] In a possible implementation, the determination module 601 is also used to determine the first area occupied by each vehicle in the corresponding image frame to be detected based on the position information of each vehicle; if the first area does not meet the preset area threshold requirement, the position information of the corresponding vehicle is deleted in the corresponding image frame to be detected.

[0166] In a possible implementation manner, the identification module 602 is further configured to delete any identification if the vehicle corresponding to the identification is not identified again within a set time length after being identified for the first time.

[0167] In a possible implementation manner, the target image frame includes the first image frame and the last image frame in which the target vehicle appears in each of the image frames to be detected;

[0168] The identification module 602 is specifically configured to determine a first prompt text according to the preset vehicle information and the preset prompt word template;

[0169] The first prompt text and the first image frame are input into the first multimodal large model to obtain the recognition result corresponding to each preset vehicle information output by the first multimodal large model, and the recognition result corresponding to the driving direction in the preset vehicle information is determined as the recognition result of the approaching direction; the second prompt text and the last image frame are input into the first multimodal large model to obtain the recognition result corresponding to the driving direction output by the first multimodal large model, and the recognition result is determined as the recognition result of the exiting direction, wherein the second prompt text is at least used to prompt the first multimodal large model to determine the driving direction of the target vehicle.

[0170] In one possible implementation, the recognition module 602 is specifically used to determine the preset vehicle information as the first re-inspection element if there is preset vehicle information for which the first multimodal large model has not obtained an identification result, and determine the third prompt text based on the first re-inspection element and the preset prompt word template; determine any image frame to be detected that includes the target vehicle and is located between the first image frame and the last image frame as a re-inspection image frame; input the third prompt text and the re-inspection image frame into the first multimodal large model to obtain the recognition result corresponding to the first re-inspection element.

[0171] In a possible implementation, the identification module 602 is further used to, if there is preset vehicle information for which no identification result has been obtained, determine the preset vehicle information for which no identification result has been obtained as a second re-inspection element; determine the second prompt text according to the second re-inspection element, the driving direction and the preset prompt word template; and input the second prompt text and the last image frame into the first multimodal large model to obtain a recognition result corresponding to the driving direction, including: inputting the second prompt text and the last image frame into the first multimodal large model to obtain a recognition result corresponding to the driving direction, and a recognition result corresponding to the second re-inspection element.

[0172] In a possible implementation, the device further includes:

[0173] The training module 603 is used to obtain a sample image and a standard result corresponding to each preset vehicle information of a sample vehicle in the sample image; randomly select at least one target vehicle information from each preset vehicle information, and determine a fourth prompt text based on the target vehicle information and a preset prompt word template; determine a sample answer based on the standard result corresponding to the target vehicle information and a preset answer template; and fine-tune the pre-trained first multimodal large model based on the sample image, the fourth prompt text, and the sample answer to obtain the first multimodal large model.

[0174] In a possible implementation manner, each preset vehicle information includes simple elements and complex elements, wherein the simple elements include vehicle color, driving direction, license plate color and license plate number, and the complex elements include vehicle type and vehicle brand;

[0175] The device also includes:

[0176] A labeling module 604, configured to identify a sample vehicle in the sample image based on a small target detection model;

[0177] For each simple element, based on the small recognition model saved for the simple element, determine the standard result corresponding to the simple element of the sample vehicle in the sample image; input the fifth prompt text and the sample image into the second multimodal model to obtain the standard result corresponding to the complex element.

[0178] Based on the same inventive concept, an embodiment of the present application further provides an electronic device, Figure 7 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application is shown in FIG. Figure 7 As shown, it includes: a processor 701, a communication interface 702, a memory 703 and a communication bus 704, wherein the processor 701, the communication interface 702, and the memory 703 communicate with each other through the communication bus 704;

[0179] The memory 703 stores a computer program. When the program is executed by the processor 701, the processor 701 performs the following steps:

[0180] Obtaining the position information of each vehicle included in each image frame to be detected in the video, and determining the identification corresponding to each vehicle according to the position information and the target tracking algorithm;

[0181] For each logo, determine a target image frame of the target vehicle corresponding to the logo in the video; analyze the target image frame based on the first multimodal large model to determine the recognition result corresponding to each preset vehicle information, where the preset vehicle information is vehicle color, vehicle type, driving direction, license plate color, license plate number or vehicle brand.

[0182] In a possible implementation manner, the process of determining each image frame to be detected in the video includes:

[0183] In the video, image frames at intervals of a preset time length are determined as image frames to be detected.

[0184] In a possible implementation, after acquiring the position information of each vehicle included in each image frame to be detected in the video, and before determining the identifier corresponding to each target according to the position information and the target tracking algorithm, the method further includes:

[0185] Determine, according to the position information of each vehicle, a first area occupied by each vehicle in a corresponding image frame to be detected;

[0186] If the first area does not meet the preset area threshold requirement, the position information of the corresponding vehicle is deleted in the corresponding image frame to be detected.

[0187] In a possible implementation, after determining the identifier corresponding to each target and before determining, for each identifier, a target image frame including the target vehicle corresponding to the identifier in each image frame to be detected, the method further includes:

[0188] If the vehicle corresponding to any logo is not recognized again within a set time length after being recognized for the first time, the logo will be deleted.

[0189] In a possible implementation manner, the target image frame includes the first image frame and the last image frame in which the target vehicle appears in each of the image frames to be detected;

[0190] The analyzing the target image frame based on the first multimodal large model to determine the recognition result corresponding to each preset vehicle information includes:

[0191] Determining a first prompt text according to the preset vehicle information and the preset prompt word template;

[0192] Inputting the first prompt text and the first image frame into the first multimodal large model, obtaining a recognition result corresponding to each preset vehicle information output by the first multimodal large model, and determining the recognition result corresponding to the driving direction in the preset vehicle information as the recognition result of the entry direction;

[0193] The second prompt text and the last image frame are input into the first multimodal large model to obtain a recognition result corresponding to the driving direction output by the first multimodal large model, and the recognition result is determined as the recognition result of the exit direction, wherein the second prompt text is at least used to prompt the first multimodal large model to determine the driving direction of the target vehicle.

[0194] In a possible implementation manner, after obtaining the recognition result corresponding to each preset vehicle information, the method further includes:

[0195] If there is preset vehicle information for which the first multimodal large model has not obtained a recognition result, the preset vehicle information is determined as a first re-inspection element, and a third prompt text is determined according to the first re-inspection element and the preset prompt word template;

[0196] Determine any image frame to be detected that includes the target vehicle and is located between the first image frame and the last image frame as a re-inspection image frame;

[0197] The third prompt text and the re-inspection image frame are input into the first multimodal large model to obtain a recognition result corresponding to the first re-inspection element.

[0198] In a possible implementation, before inputting the second prompt text and the last image frame into the first multimodal large model, the method further includes:

[0199] If there is currently preset vehicle information for which no recognition result has been obtained, the preset vehicle information for which no recognition result has been obtained is determined as the second re-inspection element;

[0200] Determining the second prompt text according to the second re-inspection element, the driving direction and the preset prompt word template;

[0201] The step of inputting the second prompt text and the last image frame into the first multimodal large model to obtain a recognition result corresponding to the driving direction includes:

[0202] The second prompt text and the last image frame are input into the first multimodal large model to obtain a recognition result corresponding to the driving direction and a recognition result corresponding to the second re-inspection element.

[0203] In a possible implementation, the fine-tuning training process of the first multimodal large model includes:

[0204] Acquire a sample image and a standard result corresponding to each preset vehicle information of a sample vehicle in the sample image;

[0205] Randomly selecting at least one target vehicle information from each preset vehicle information, and determining a fourth prompt text according to the target vehicle information and a preset prompt word template;

[0206] Determine a sample answer based on the standard result corresponding to the target vehicle information and a preset answer template;

[0207] The pre-trained first multimodal large model is fine-tuned based on the sample image, the fourth prompt text, and the sample answer to obtain the first multimodal large model.

[0208] In a possible implementation manner, each preset vehicle information includes simple elements and complex elements, wherein the simple elements include vehicle color, driving direction, license plate color and license plate number, and the complex elements include vehicle type and vehicle brand;

[0209] The process of determining the standard result corresponding to each preset vehicle information of the sample vehicle in the sample image includes:

[0210] Identify a sample vehicle in the sample image based on a small target detection model;

[0211] For each simple element, based on the small recognition model saved for the simple element, determine a standard result corresponding to the simple element of the sample vehicle in the sample image;

[0212] The fifth prompt text and the sample image are input into the second multimodal model to obtain a standard result corresponding to the complex element.

[0213] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus. The communication interface 702 is used for communication between the above electronic device and other devices. The memory may include a random access memory (RAM) and may also include a non-volatile memory (NVM), such as at least one disk storage. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0214] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0215] On the basis of the above embodiments, an embodiment of the present invention further provides a computer-readable storage medium, which stores a computer program executable by a processor. When the program runs on the processor, the processor implements the steps of the large model-based vehicle information identification method described in the above embodiments.

[0216] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0217] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0218] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0219] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 A step that specifies a function in one or more boxes.

[0220] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A vehicle information recognition method based on a large model, characterized in that: The method comprises: Obtaining the position information of each vehicle included in each image frame to be detected in the video, and determining the identification corresponding to each vehicle according to the position information and the target tracking algorithm; For each logo, determine a target image frame of the target vehicle corresponding to the logo in the video; analyze the target image frame based on the first multimodal large model to determine the recognition result corresponding to each preset vehicle information, where the preset vehicle information is vehicle color, vehicle type, driving direction, license plate color, license plate number or vehicle brand.

2. The method according to claim 1, characterized in that The process of determining each image frame to be detected in the video includes: In the video, image frames at intervals of a preset time length are determined as image frames to be detected.

3. The method according to claim 1, characterized in that After acquiring the position information of each vehicle included in each image frame to be detected in the video, and before determining the identifier corresponding to each target according to the position information and the target tracking algorithm, the method further includes: Determine, according to the position information of each vehicle, a first area occupied by each vehicle in a corresponding image frame to be detected; If the first area does not meet the preset area threshold requirement, the position information of the corresponding vehicle is deleted in the corresponding image frame to be detected.

4. The method according to claim 1, characterized in that: After determining the identifier corresponding to each target, and before determining, for each identifier, a target image frame including the target vehicle corresponding to the identifier in each image frame to be detected, the method further includes: If the vehicle corresponding to any logo is not recognized again within a set time length after being recognized for the first time, the logo will be deleted.

5. The method according to claim 4, characterized in that The target image frame includes the first image frame and the last image frame in which the target vehicle appears in each of the image frames to be detected; The analyzing the target image frame based on the first multimodal large model to determine the recognition result corresponding to each preset vehicle information includes: Determining a first prompt text according to the preset vehicle information and the preset prompt word template; Inputting the first prompt text and the first image frame into the first multimodal large model, obtaining a recognition result corresponding to each preset vehicle information output by the first multimodal large model, and determining the recognition result corresponding to the driving direction in the preset vehicle information as the recognition result of the entry direction; The second prompt text and the last image frame are input into the first multimodal large model to obtain a recognition result corresponding to the driving direction output by the first multimodal large model, and the recognition result is determined as the recognition result of the exit direction, wherein the second prompt text is at least used to prompt the first multimodal large model to determine the driving direction of the target vehicle.

6. The method according to claim 5, characterized in that After obtaining the recognition result corresponding to each preset vehicle information, the method further includes: If there is preset vehicle information for which the first multimodal large model has not obtained a recognition result, the preset vehicle information is determined as a first re-inspection element, and a third prompt text is determined according to the first re-inspection element and the preset prompt word template; Determine any image frame to be detected that includes the target vehicle and is located between the first image frame and the last image frame as a re-inspection image frame; The third prompt text and the re-inspection image frame are input into the first multimodal large model to obtain a recognition result corresponding to the first re-inspection element.

7. The method according to claim 5 or 6, characterized in that: Before inputting the second prompt text and the last image frame into the first multimodal large model, the method further includes: If there is currently preset vehicle information for which no recognition result has been obtained, the preset vehicle information for which no recognition result has been obtained is determined as the second re-inspection element; Determining the second prompt text according to the second re-inspection element, the driving direction and the preset prompt word template; The step of inputting the second prompt text and the last image frame into the first multimodal large model to obtain a recognition result corresponding to the driving direction includes: The second prompt text and the last image frame are input into the first multimodal large model to obtain a recognition result corresponding to the driving direction and a recognition result corresponding to the second re-inspection element.

8. The method according to claim 1, characterized in that The fine-tuning training process of the first multimodal large model includes: Acquire a sample image and a standard result corresponding to each preset vehicle information of a sample vehicle in the sample image; Randomly selecting at least one target vehicle information from each preset vehicle information, and determining a fourth prompt text according to the target vehicle information and a preset prompt word template; Determine a sample answer based on the standard result corresponding to the target vehicle information and a preset answer template; The pre-trained first multimodal large model is fine-tuned based on the sample image, the fourth prompt text, and the sample answer to obtain the first multimodal large model.

9. The method according to claim 8, characterized in that Each preset vehicle information includes simple elements and complex elements, wherein the simple elements include vehicle color, driving direction, license plate color and license plate number, and the complex elements include vehicle type and vehicle brand; The process of determining the standard result corresponding to each preset vehicle information of the sample vehicle in the sample image includes: Identify a sample vehicle in the sample image based on a small target detection model; For each simple element, based on the small recognition model saved for the simple element, determine a standard result corresponding to the simple element of the sample vehicle in the sample image; The fifth prompt text and the sample image are input into the second multimodal model to obtain a standard result corresponding to the complex element.

10. An electronic device, characterized in that: The electronic device comprises a processor, and the processor is used to implement the steps of the vehicle information recognition method based on a large model as claimed in any one of claims 1 to 9 when executing a computer program stored in a memory.

Images