Model deployment method, end-side device, and storage medium

Abstract

The present disclosure relates to the technical field of target detection, and particularly relates to a model deployment method, an end-side device and a storage medium, which are used for solving the problem in the related art of the accuracy of a deployed model being low. The method comprises: performing target detection on a video frame image input into a first model, and acquiring a first target detection result and a first confidence; if the first confidence is greater than or equal to a first confidence threshold value, recording the video frame image and the first target detection result as samples in a training set; if the first confidence is less than the first confidence threshold value, performing target detection on the video frame image on the basis of a second model, and recording the video frame image and an acquired second target detection result as samples in the training set; and training the first model on the basis of the training set, and replacing the current first model with a trained first model for subsequent target detection. In this way, the accuracy and model generalization capability of a first model are improved.

Description

A model deployment method, end-side device, and storage medium

[0001] Cross-references to related applications

[0002] This disclosure claims priority to Chinese applications filed on May 8, 2024, application number 202410558417.3; filed on May 31, 2024, application number 202410702591.0; filed on June 12, 2024, application number 202410749810.0; and filed on July 2, 2024, application number 202410881385.0, the entire contents of which are incorporated herein by reference. Technical Field

[0003] This disclosure relates to the field of target detection technology, and in particular to a model deployment method, an end-side device, and a storage medium. Background Technology

[0004] Edge devices refer to devices or nodes located close to the data source and user, such as mobile devices and Internet of Things (IoT) devices. Some data processing and storage functions are placed on these devices to improve response speed and reduce network load. Currently, the models deployed on edge devices are typically lightweight models, also known as small models. These small models are usually trained specifically for a particular business need or a single scenario. Because of this, small models offer high real-time object detection performance in practical applications, but suffer from low accuracy and weak generalization ability in complex scenarios, requiring continuous iterative optimization.

[0005] However, in related technologies, the iterative optimization of small models usually relies on the cloud. The edge device feeds the collected data back to the cloud, which then uses the fed-back data to obtain scenario data for a specific business requirement and iteratively optimizes the small model based on the scenario data. The optimized model parameters are then sent back to the edge device to achieve iterative optimization of the small model already deployed on the edge device. However, due to privacy and security concerns, the data feed back in related technologies carries the risk of privacy leakage. Therefore, data containing user privacy is usually not allowed to be transmitted externally. This results in the cloud being unable to obtain scenario data, making it difficult to continue iteratively optimizing the small model already deployed on the edge device, thus failing to improve recognition accuracy and enhance model generalization ability.

[0006] Summary of the Invention

[0007] This disclosure provides a model deployment method, an edge device, and a storage medium to improve the accuracy of deployed models and enhance their generalization ability.

[0008] In a first aspect, embodiments of this disclosure provide a model deployment method, the method comprising:

[0009] For the video frame images input to the first model, perform the following operations to construct the training set:

[0010] Based on the first model, target detection is performed on the video frame image to obtain a first target detection result and a first confidence level of the first target detection result; the first confidence level is used to characterize the accuracy of the first target detection result; the first model is used for target detection under a first business requirement;

[0011] If the first confidence level is greater than or equal to the first confidence threshold, then the video frame image and the first target detection result are recorded as samples in the training set;

[0012] If the first confidence level is less than the first confidence level threshold, then target detection is performed on the video frame image based on the second model to obtain the second target detection result of the video frame image and the second confidence level of the second target detection result. The video frame image and the second target detection result are recorded as samples in the training set. The second confidence level is used to characterize the accuracy of the second target detection result. The second model is used for target detection under various business requirements, including the first business requirement.

[0013] The first model is trained based on the training set, and the trained first model is used to replace the current first model for subsequent object detection.

[0014] Secondly, embodiments of this disclosure provide a model deployment method, the method comprising:

[0015] Acquire a multimodal large model and a target video; wherein the multimodal large model includes a fusion structure, a language large model, and at least two video encoders;

[0016] The target video is input into the at least two video encoders respectively, and each initial video feature matrix output by the at least two video encoders is obtained;

[0017] Each initial video feature matrix is ​​input into the fusion structure, which concatenates each initial video feature matrix to obtain a candidate video feature matrix. Based on a preset weight matrix, the candidate video feature matrix is ​​mapped to a target video feature matrix with the same number of rows and columns as the weight matrix.

[0018] The target video feature matrix and instruction prompts are input into a language model to obtain the target text output by the language model.

[0019] Thirdly, embodiments of this disclosure provide a model deployment method, the method comprising:

[0020] The target image to be detected and a preset number of other images adjacent to the acquisition time of the target image are input into the accident detection mini-model, so that the accident detection mini-model determines the motion trajectory corresponding to each target, and determines and outputs the target confidence that there is a traffic accident in the target image based on the motion trajectory; wherein, the accident detection mini-model includes at least a target detection sub-model, a target tracking sub-model and an accident detection sub-model;

[0021] If the target confidence level falls within a preset confidence level range, the target image is input into a multimodal large model for detecting traffic accidents, and the detection result of whether a traffic accident exists in the target image is obtained from the output of the multimodal large model.

[0022] Fourthly, embodiments of this disclosure provide a model deployment method, the method comprising:

[0023] A small model is used to identify key regions of the target image to be identified and determine the coordinate information corresponding to the key regions.

[0024] Based on the coordinate information, determine the target sub-image corresponding to the key region in the target image;

[0025] Based on the multimodal large model and the target sub-image, the recognition result corresponding to the target sub-image is determined.

[0026] Fifthly, embodiments of this disclosure provide an end-side device, including:

[0027] Image acquisition device, used to acquire video streams;

[0028] Front-end display device for displaying images and / or alarms;

[0029] Memory is used to store computer programs or instructions;

[0030] A processor for executing a computer program or instructions in the memory such that the method described in any of the first to fourth aspects above is performed.

[0031] In a sixth aspect, embodiments of this disclosure provide a computer-readable storage medium that, when instructions in the storage medium are executed by a processor, enables the processor to perform the method described in any one of the first to fourth aspects.

[0032] In a seventh aspect, embodiments of this disclosure provide a computer program product comprising: computer program code, which, when executed on a computer, causes the computer to perform the method described in any one of the first to fourth aspects. Attached Figure Description

[0033] Figure 1 is a schematic diagram of the application scenario of the model deployment method in the embodiments of this disclosure;

[0034] Figure 2 is a schematic diagram of the architecture of the deployment model in the end-side device in the embodiments of this disclosure;

[0035] Figure 3 is a flowchart illustrating a model deployment method according to an embodiment of this disclosure;

[0036] Figure 4 is a schematic diagram of the training process of a first model in an embodiment of this disclosure;

[0037] Figure 5 is a schematic diagram of the update process of a first model in an embodiment of this disclosure;

[0038] Figure 6 is a schematic diagram of the fine-tuning process of a second model in an embodiment of this disclosure;

[0039] Figure 7 is another flowchart illustrating a model deployment method in an embodiment of this disclosure;

[0040] Figure 8 is a schematic diagram of the track scene in an embodiment of this disclosure;

[0041] Figure 9 is another flowchart illustrating a model deployment method in an embodiment of this disclosure;

[0042] Figure 10 is a schematic diagram of a multimodal large model used for video understanding in an embodiment of this disclosure;

[0043] Figure 11 is another flowchart illustrating a model deployment method according to an embodiment of this disclosure;

[0044] Figure 12 is a schematic diagram of the application of the multimodal large model in the embodiments of this disclosure;

[0045] Figure 13 is a flowchart illustrating the process of determining the target video feature matrix in an embodiment of this disclosure;

[0046] Figure 14 is a schematic diagram of the application of the grouped attention fusion structure in the embodiments of this disclosure;

[0047] Figure 15 is a schematic diagram of the process of updating the sub-video feature matrix in an embodiment of this disclosure;

[0048] Figure 16 is another flowchart illustrating the determination of the target video feature matrix in an embodiment of this disclosure;

[0049] Figure 17 is a schematic diagram of the video processing flow based on a multimodal large model in an embodiment of this disclosure;

[0050] Figure 18 is another flowchart illustrating a model deployment method according to an embodiment of this disclosure;

[0051] Figure 19 is a schematic diagram comparing traditional convolution and dilated convolution in the embodiments of this disclosure;

[0052] Figure 20 is a flowchart of the application of the visual encoder in an embodiment of this disclosure;

[0053] Figure 21 is a schematic diagram of the dataset construction process in an embodiment of this disclosure;

[0054] Figure 22 is a schematic diagram of a sample image in an embodiment of this disclosure;

[0055] Figure 23 is a schematic diagram of the output of the sample image after the region description pre-training model in the embodiment of this disclosure;

[0056] Figure 24 is a schematic diagram of the dataset construction process in an embodiment of this disclosure;

[0057] Figure 25 is a schematic diagram of the traffic accident detection process based on the size model decision in an embodiment of this disclosure;

[0058] Figure 26 is a schematic diagram of a snow accumulation recognition scenario according to an embodiment of this disclosure;

[0059] Figure 27 is a schematic diagram of a railway foreign object identification scenario according to an embodiment of this disclosure;

[0060] Figure 28 is a schematic diagram of a sleeping-on-duty scenario according to an embodiment of this disclosure;

[0061] Figure 29 is another flowchart illustrating a model deployment method according to an embodiment of this disclosure;

[0062] Figure 30 is a schematic diagram of the snow recognition scene after masking in an embodiment of this disclosure;

[0063] Figure 31 is a schematic diagram of railway foreign object identification mask in an embodiment of this disclosure;

[0064] Figure 32 is a schematic diagram of the construction process of the sample dataset corresponding to the snow scene in the embodiments of this disclosure;

[0065] Figure 33 is a schematic diagram of the scene recognition process in an embodiment of this disclosure;

[0066] Figure 34 is a schematic diagram of the application of multimodal large model recognition in the embodiments of this disclosure;

[0067] Figure 35 is a schematic diagram of the physical architecture of the end-side device in an embodiment of this disclosure. Detailed Implementation

[0068] The technical solutions of the embodiments of this disclosure will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this disclosure, and not all embodiments. Based on the embodiments of this disclosure, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this disclosure.

[0069] It should be noted that the terms "first," "second," "third," etc., used in the specification, claims, and accompanying drawings of this disclosure are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this disclosure described herein can be implemented in sequences other than those illustrated or described herein.

[0070] The design concept of the embodiments of this disclosure will be briefly introduced below.

[0071] This disclosure relates to the field of target detection technology, and mainly to a model deployment method, an end-side device, and a storage medium.

[0072] On-device devices typically deploy target detection models (also known as lightweight models or small models) tailored to a specific business requirement to achieve target detection within that requirement. In practice, small models often suffer from low accuracy and weak generalization ability in complex scenarios. Solutions to these problems often require continuous iterative optimization of the small model.

[0073] Under relevant technologies, iterative optimization of small models usually relies on the cloud. The cloud obtains scenario data for a specific business requirement based on the data fed back from the edge device, then iteratively optimizes the small model based on the scenario data, and then sends the iteratively optimized model parameters to the edge device to achieve iterative optimization of the small model already deployed on the edge device.

[0074] However, as users become more aware of privacy protection, they are usually not allowed to transmit user data to external systems. As a result, the cloud cannot obtain the aforementioned backflow data, and consequently, the cloud cannot obtain scene data. Consequently, the cloud cannot continue to iterate and optimize the small models already deployed on the edge devices, thus failing to improve accuracy and enhance model generalization ability.

[0075] To alleviate the above problems, this disclosure provides a model deployment method. In this embodiment, a first model and a second model are deployed on an edge device. The first model is used for target detection under a first business requirement, and the second model is used for target detection under multiple business requirements, including the first business requirement. The method includes: performing the following operations on a video frame image input to the first model: performing target detection on the video frame image based on the first model, obtaining a first target detection result and a first confidence level of the first target detection result; if the first confidence level is greater than or equal to a first confidence level threshold, recording the video frame image and the first target detection result as samples in the training set; if the first confidence level is less than the first confidence level threshold, performing target detection on the video frame image based on the second model, obtaining a second target detection result and a second confidence level of the second target detection result, and recording the video frame image and the second target detection result as samples in the training set; and then training the first model based on the training set.

[0076] In this way, a training set can be built on the edge device, and it is built in conjunction with the second model during the implementation of the first model. Because each sample in the training set is collected and processed during implementation, scene data is preserved. Training the first model based on this training set can improve the accuracy of the first model. Deploying the second model on the edge device, which is used for target detection under various business needs, also improves the model's generalization ability. Thus, the embodiments of this disclosure have robustness, generalizability, reproducibility, and controllability, and meet the trust characteristics. Furthermore, the training set used to train the first model comes from local data on the edge device, and the data is not transmitted externally, which also ensures user privacy and security.

[0077] The model deployment method provided in this disclosure is applicable to edge devices in application fields such as smart cities, intelligent transportation, intelligent manufacturing, energy, railway, and education. It is also applicable to scenarios where the aforementioned method can be used for event detection, such as interaction events related to people or vehicles, target detection, and product defect detection.

[0078] Figure 1 is a schematic diagram of an application scenario of a model deployment method provided in an embodiment of this disclosure. Referring to Figure 1, this application scenario may include an end-side device 10, which may include a front-end display device 101, an image acquisition device 102, and a back-end server 103.

[0079] The front-end display device 10 is used to display real-time video footage and / or alarms, etc.; in some embodiments, the front-end display device may be a smart terminal, a smart mobile terminal, a tablet computer, a laptop computer, a smart handheld device, a personal computer (PC), a computer, a display device, a smart screen, an in-vehicle terminal, a personal digital assistant (PDA), etc.

[0080] Image acquisition device 102 is used to acquire video streams; in some embodiments, the image acquisition device may be a bullet camera, a dome camera, an infrared camera, etc.

[0081] The backend server 103 deploys a first model and a second model. The first model is used for target detection under a first business requirement, and the second model is used for target detection under multiple business requirements, including the first business requirement. The backend server 103 is used to execute the model deployment method provided in this embodiment of the disclosure, construct a training set based on the first model and the second model, as well as video frame images in the input model, and train the first model based on the training set to improve the accuracy and generalization ability of the model.

[0082] In some embodiments, the backend server can be an independent physical server, a server cluster composed of multiple physical servers, or a distributed system. It can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, content delivery networks (CDN), and big data and artificial intelligence platforms. If the user has no requirements for user privacy, a local server or a cloud server can be used. If the user has requirements for user privacy, only a local server, such as an enterprise internal cloud server or a local area cloud server, will be used.

[0083] In some embodiments, the front-end display device 101 and the back-end server 103, the image acquisition device 102 and the back-end server 103, and the front-end display device 101 and the image acquisition device 102 can all be connected via wired or wireless means.

[0084] Of course, the method provided in this disclosure is not limited to the application scenario shown in FIG1, and can also be used in other possible application scenarios. This disclosure does not limit the application scenario.

[0085] It should be noted that the edge device of this disclosure can deploy multiple small models, each of which is used for target detection under the corresponding business requirements. For ease of description, this disclosure only uses one small model (denoted as the first model) as an example for detailed description. It is understood that this disclosure does not limit the specific number of small models deployed in the edge device, and the number of small models deployed can be determined based on the number of business requirements.

[0086] The end-side device disclosed herein can also deploy multiple large models. For ease of description, this disclosure describes a single large model (referred to as the second model) as an example. It is understood that this disclosure does not limit the specific number of large models deployed in the end-side device, and the number of large models can be determined according to the actual business complexity.

[0087] Referring to Figure 2, the edge device deploys N small models and one large model for N service requirements. The corresponding video stream is input into the small model corresponding to the service requirement to achieve target detection for the N service requirements. The large model is used for re-judgment of video frame images for any service requirement. As shown in Figure 2, the process includes the following steps:

[0088] S201: Determine the video stream required for business needs;

[0089] S202: Input the video stream into the small model;

[0090] S203: Input the video stream from the small model into the large model for re-judgment.

[0091] For example, video stream 1 for business requirement 1 is input into small model 1, video stream 2 for business requirement 2 is input into small model 2, and so on, with video stream N for business requirement N being input into small model N. Video streams 1, 2, 3, ..., N may include some video streams acquired by the same image acquisition device, or they may be acquired by their own independent image acquisition devices; this disclosure does not impose specific limitations. It is understood that the N small models deployed in the end-side device shown in Figure 2 are merely examples, and the embodiments of this disclosure do not impose specific limitations.

[0092] After introducing the application scenarios of the embodiments of this disclosure, the preferred embodiments of this disclosure will be further described in detail below with reference to the accompanying drawings. It should be understood that the preferred embodiments described herein are only for illustration and explanation of this disclosure and are not intended to limit this disclosure. Furthermore, the embodiments of this disclosure and the features in the embodiments can be combined with each other without conflict.

[0093] In some embodiments, the model deployment method provided in this disclosure is applied to an end-side device. The end-side device deploys a first model and a second model. The first model is used for target detection under a first business requirement, and the second model is used for target detection under multiple business requirements, including the first business requirement.

[0094] Referring again to Figure 1, the image acquisition device, such as a camera, acquires a real-time video stream and transmits the video stream to the backend server. The backend server first performs video decoding on the input video stream to obtain an image sequence. Then, according to a preset frame extraction period, it extracts video frame images from the image sequence and inputs the extracted video frame images into the first model deployed on the backend server.

[0095] Taking the first model, the smoking model in a track scene, as an example, for the input video stream, the Region of Interest (ROI) can be extracted from the image according to the set detection area, resulting in the ROI sequence img1, img2, img3, ..., imgn, ... In some embodiments, assuming that detection is performed at intervals, the final sequence of images to be detected can be image1 (denoted as video frame image 1), image2 (denoted as video frame image 2), ..., imagen (denoted as video frame image n), ...

[0096] In some embodiments, the image sequence may also be preprocessed, such as image enhancement or deblurring. This disclosure does not specifically limit this process, and the specific settings can be made according to actual needs.

[0097] In some embodiments, the first model may be a machine learning model, a convolutional neural network (CNN) or a deep learning model based on the Transformer architecture, etc. It should be noted that this disclosure does not limit the model structure or model type.

[0098] In some embodiments, after performing frame extraction processing on the acquired video stream, video frame images are obtained. Further, the process of the above-described model deployment method can be executed to achieve local iterative optimization of the first model. The specific process of the above method may include, but is not limited to, some or all of the following steps:

[0099] Figure 3 illustrates the overall flow of a model deployment method provided in an embodiment of this disclosure, including the following steps:

[0100] S301: Perform target detection on the video frame image based on the first model, and obtain the first target detection result of the video frame image and the first confidence level of the first target detection result, wherein the first confidence level is used to characterize the accuracy of the first target detection result.

[0101] In some embodiments, the first model is used for target detection under a first business requirement. After the aforementioned frame extraction process is performed to obtain a video frame image, the video frame image is input into the first model, and step S301 is executed. The first model performs real-time detection on the input video frame image and outputs the first target detection result of the video frame image and the first confidence level of the first target detection result.

[0102] Taking the first model as an example of the smoking model in the track scene, the first model can detect and analyze scene data (i.e., video frame image data input to the first model), such as Intersection over Union (IOU), to determine whether there is smoking behavior (whether there is a smoking event) and whether an alarm is needed, and output the first confidence level of the judgment.

[0103] In some embodiments, the first confidence level can be obtained based on the confidence level of the target box detected by the first model, or based on the post-processing judgment logic of the first model.

[0104] In some embodiments, when the first confidence level of the first target detection result of the first model is greater than or equal to the first confidence threshold T1, and there is smoking behavior (smoking event), i.e., the preset alarm condition is met, an alarm is issued to the user; when the first confidence level of the first target detection result of the first model is greater than or equal to the threshold T1, and there is no smoking behavior (smoking event), i.e., the alarm condition is not met, no alarm is required, and the target detection process for the next video frame image is entered; conversely, when the first confidence level of the first target detection result output by the first model is less than the first confidence threshold T1, the target detection of the video frame image is performed by the second model to achieve detection re-judgment.

[0105] Assume the first model is a YOLOv8 model for smoking detection based on deep learning. In specific implementation, when executing step S301, the YOLOv8 model performs object detection on the input video frame image, and obtains the first object detection result and the first confidence level of the first object detection result.

[0106] S302: If the first confidence level is greater than or equal to the first confidence level threshold, then the video frame image and the first target detection result are recorded as samples in the training set.

[0107] In some embodiments, if the first confidence level is greater than or equal to the first confidence threshold, then step S302 is executed, recording the video frame image and the first target detection result of the video frame image as samples in the training set. These samples can be recorded as positive samples in the training set.

[0108] In some embodiments, after obtaining the first target detection result and the first confidence level of the first target detection result from the video frame image, a first alarm may be issued to the user when it is determined that the first confidence level is greater than or equal to the first confidence level threshold. The first alarm is used to prompt the user to confirm whether an event associated with the first target detection result has occurred. As exemplified above, this event could be a smoking event.

[0109] In some embodiments, the edge device can display the first alarm through a human-machine interface (HMI) for the user. After receiving the first alarm through the HMI, the user can check the displayed screen or conduct an on-site inspection to confirm whether the event is related to the first target detection result. For example, as described above, the user can check the on-site screen or conduct an on-site inspection to determine whether a smoking event has occurred in the track scene.

[0110] In some embodiments, after on-site confirmation and / or processing, the user may or may not respond to the first alarm in the human-computer interaction interface.

[0111] In some embodiments, if a user responds to a first alarm in the human-computer interaction interface, the edge device responds to the user's operation and generates a response instruction, which includes a confirmation instruction and a false alarm instruction. Upon receiving the user's confirmation instruction for the first alarm, the edge device sets a manually labeled tag for the first target detection result in the training set, where this sample is recorded as a positive sample in the training set. Upon receiving the user's false alarm instruction for the first alarm, the edge device updates the first target detection result in the training set and sets a manually labeled tag for the updated first target detection result, where this sample is recorded as a positive sample in the training set.

[0112] In this way, through the above process, manually labeled samples can be collected in the training set, which enriches the source of labeled data in the training set and facilitates the subsequent training of the first model based on the training set, thereby improving the detection accuracy of the model.

[0113] S303: If the first confidence level is less than the first confidence level threshold, then target detection is performed on the video frame image based on the second model, the second target detection result of the video frame image and the second confidence level of the second target detection result are obtained, and the video frame image and the second target detection result are recorded as samples in the training set, wherein the second confidence level is used to characterize the accuracy of the second target detection result.

[0114] In some embodiments, if the first confidence level is less than the first confidence threshold, then step S303 is executed, the video frame image is input into the second model, the second model is used to perform target detection on the video frame image, the second target detection result of the video frame image and the second confidence level of the second target detection result are obtained, and the video frame image and the second target detection result are recorded as samples in the training set.

[0115] In some embodiments, if the second target detection result of the sample indicates that a target event exists in the video frame image, it is recorded as a positive sample in the training set; if the second target detection result of the sample indicates that a target event does not exist in the video frame image, it is recorded as a negative sample in the training set.

[0116] In some embodiments, after obtaining the second target detection result and the second confidence level of the second target detection result from the video frame image, the second confidence level is compared with a second confidence threshold. If the second confidence level is greater than or equal to the second confidence threshold, a second alarm is issued to the user. The second alarm is used to prompt the user to confirm whether an event associated with the second target detection result has occurred. As exemplified above, the event could be a smoking event.

[0117] In some embodiments, the edge device can display the second alarm through a human-machine interface (HMI) for the user. After receiving the second alarm through the HMI, the user can confirm whether it is an event related to the second target detection result by viewing the screen displayed on the interface or by conducting an on-site inspection. Continuing with the previous example, similar to the scenario of the first alarm, the user can determine whether a smoking event has occurred in the track scene by viewing the on-site screen or conducting an on-site inspection.

[0118] In some embodiments, after on-site confirmation and / or processing, the user may or may not respond to the second alarm in the human-computer interaction interface.

[0119] If the user responds to the second alarm in the human-computer interaction interface, the edge device, upon receiving the user's confirmation instruction for the second alarm, sets a manually labeled tag for the second target detection result in the training set, where the sample is recorded as a positive sample in the training set; upon receiving the user's false alarm instruction for the second alarm, it updates the second target detection result in the training set and sets a manually labeled tag for the updated second target detection result, where the sample is recorded as a negative sample in the training set.

[0120] In this way, through the above process, manually labeled samples can also be collected in the training set, which enriches the source of labeled data in the training set and facilitates the subsequent training of the first model based on the training set, thereby improving the model's detection accuracy.

[0121] S304: Train the first model based on the training set, and replace the current first model with the trained first model for subsequent object detection.

[0122] The specific process of training the first model based on the training set is shown in Figure 4, including the following steps:

[0123] S401: Based on the preset sample balancing rules, each sample is obtained from the training set.

[0124] In some embodiments, the sample balancing rule includes some or all of the following rules:

[0125] Rule 1: The ratio of the number of samples with manual labels to the total number of samples in the model training is greater than or equal to the first ratio.

[0126] Rule 2: The proportion of samples from different data sources is the first proportion;

[0127] Rule 3: The proportion of samples from different target categories is the second proportion;

[0128] Rule 4: The proportion of samples with different attribute information is the third proportion. The attribute information is obtained by the second model to identify the attributes of each sample in the training set. The attribute information includes part or all of the image illumination information and the weather information of the image acquisition environment.

[0129] In some embodiments, when a sample is recorded in the training set, the detection model information of that sample is also recorded as a data source (which can be used as a data label for that sample).

[0130] As a specific implementation, after the number of samples in the training set is greater than the preset total number of samples for model training, firstly, based on the data label of each sample in the training set, the number of samples with manually labeled data is counted, and / or, the number of samples from different data sources is counted, and / or, the number of samples from different target categories is counted, and based on the counted quantities, samples that meet the sample balance rules are obtained from the training set.

[0131] In some embodiments, the number of samples with manually labeled data (positive samples) can be counted first, and it can be determined whether the number of manually labeled samples meets the model training requirements. For example, if this number is half of the total number of samples for model training, then samples obtained from the training set by the second model for object detection (negative samples) are obtained, and it is ensured that the positive samples obtained from the training set are balanced across categories. For example, if the target categories to be detected are excavators, bulldozers, and cement mixers, then the number of samples of these three target categories in the samples obtained from the training set needs to be the same or similar.

[0132] In some embodiments, if the number of samples with manually labeled data does not meet the model training requirements, for example, if the number is less than one-third of the total number of samples for model training, then samples for object detection by the second model are obtained from the training set. In this case, the samples include both positive and negative samples, and each sample obtained must satisfy the sample balance rule.

[0133] In other embodiments, before acquiring each sample from the training set, attribute information of each sample in the training set is obtained based on the second model. In the end-side device of this disclosure, some prompt commands are pre-set to trigger the second model to perform attribute recognition on each sample in the training set and obtain the attribute information of each sample. This attribute information includes, but is not limited to, image illumination information and weather information of the image acquisition environment, such as the weather conditions in the sample image—sunny, rainy, snowy, or foggy. This disclosure does not limit the type and content of the prompt commands.

[0134] In other embodiments, at least one third model can be pre-deployed in the edge device. Each third model is used for target detection under various business requirements. Then, when obtaining the attribute information of each sample in the training set, attribute recognition can be performed on each sample in the training set based on at least one third model deployed in the edge device to obtain the attribute information of each sample.

[0135] In addition, in order to retain the existing capabilities of the first model, each sample obtained from the training set must include a portion of the samples that were used for object detection by the first model.

[0136] For example, when recording samples in the training set, the training set can be subdivided. Specifically, it can be subdivided into a first subset where the first model performs object detection, a second subset where the second model performs object detection, and a third subset with manually labeled data. Alternatively, it can be further subdivided based on differences in the detection results or target locations of the first and second models to obtain subsets of incorrect samples from the first model, subsets of correct samples from the first model, and so on. In this way, each sample can be obtained more conveniently when executing step S401.

[0137] S402: Train the first model based on each sample.

[0138] In some embodiments, the timing for training the first model can be determined based on actual business conditions, such as the availability of backend server resources. If resources are more abundant at night, the first model can be trained at night to optimize the model.

[0139] In some other embodiments, referring to Figure 5, after training the first model based on the training set, the following steps may also be performed to verify the capabilities of the trained first model and to determine whether the current first model should be replaced:

[0140] S501: Obtain multiple samples from the training set respectively.

[0141] In some embodiments, after training the first model, in order to confirm the superiority or inferiority of the trained first model and the current first model, a validation set or test set is usually used to evaluate and compare the two models; then, step S501 is executed to obtain multiple samples from the training set to verify the capabilities of the two models.

[0142] S502: Evaluate and compare the first model after training and the current first model based on multiple samples to obtain the first parameter value. The first parameter value represents the degree of difference between the evaluation index values ​​of the first model after training and the current first model. The evaluation index values ​​include some or all of the accuracy, recall, mean squared error and log loss.

[0143] In some embodiments, a model evaluation method in related technologies is used to evaluate the trained first model and the current first model based on multiple samples to obtain their respective evaluation index values. Then, the evaluation index values ​​of the two models are compared to obtain a first parameter value that characterizes the degree of difference between the two models.

[0144] In other embodiments, all samples in the training set can be used to evaluate the trained first model and the current first model to better obtain the degree of difference between the two models.

[0145] S503: If the first parameter value meets the first replacement condition, the trained first model replaces the current first model for subsequent target detection. The first replacement condition includes the first parameter value being greater than the first parameter threshold, or the first parameter value being greater than the first parameter threshold and the detection time of the current first model reaching the first preset time.

[0146] In some embodiments, the current first model can be replaced with the trained first model after determining that the first reference value is greater than the first parameter threshold, or it can be replaced after sufficient validation with more samples, such as after the first parameter value is greater than the first parameter threshold and the detection time of the current first model reaches the first preset time.

[0147] In some embodiments, if the first parameter value does not meet the first replacement condition, the aforementioned steps S301 to S303 can be continued to be executed to continue collecting samples and training the first model based on the training set, so as to obtain a trained first model whose evaluation index value representation ability is better than the current first model.

[0148] Through the above process, samples can be collected and local data accumulated simultaneously during the execution of the target detection task by the first model. This enables automatic iterative optimization of the first model already deployed on the edge device based on the obtained training set. Compared with related technologies where iterative optimization of the deployed first model can only be achieved by relying on the cloud, the model deployment method in this embodiment can guide the first model to perform automatic iterative training locally based on the second model, which has scene understanding and recognition capabilities for target detection under various business needs, as well as its strong generalization ability, thereby improving the accuracy and recall of the first model.

[0149] In some embodiments, manually labeled samples can be collected through the above alarm process. Then, the second model can be fine-tuned and updated using samples with manually labeled tags in the training set to improve the accuracy and recall of the second model.

[0150] In some embodiments, the second model can be fine-tuned and updated by performing some or all of the following steps. It should be noted that the process shown in Figure 6 is only one specific implementation:

[0151] S600: Select multiple target samples from the training set that have been manually labeled.

[0152] S610: Fine-tune the second model based on multiple target samples.

[0153] S620: Obtain multiple samples from the training set respectively, and evaluate and compare the fine-tuned second model and the current second model based on the multiple samples to obtain the second parameter value. The second parameter value represents the degree of difference between the evaluation index values ​​of the fine-tuned second model and the second model. The evaluation index values ​​include some or all of the accuracy, recall, mean squared error and log loss.

[0154] S630: After the second parameter value meets the second replacement condition, the current second model is replaced with the fine-tuned second model for subsequent target detection. The second replacement condition includes the second parameter value being greater than the second parameter threshold, or the second parameter value being greater than the second parameter threshold and the detection time of the current second model reaching the second preset time.

[0155] Steps S600 to S630 are similar to the aforementioned process of verifying and replacing the trained first model and the current first model, as detailed above, and will not be repeated here.

[0156] Through the above process, samples can be collected and local data accumulated simultaneously during the execution of the target detection task by the first model. While enabling automatic iterative optimization of the first model already deployed on the edge device based on the obtained training set, the second model can also be automatically fine-tuned based on samples with manually labeled samples in the training set, thereby achieving iterative updates of the second model and improving its accuracy and recall.

[0157] The automatic iterative optimization method for the end-side device has been described in detail above. The implementation and operation process of deploying the first model and the second model is described in detail below. This process is another model deployment method in the embodiments of this disclosure and is implemented based on the aforementioned end-side device structure.

[0158] In some embodiments, a first model is used to perform target detection on the input video frame image to obtain a first target detection result and a first confidence level of the first target detection result. It is then determined whether the first confidence level is greater than a first confidence threshold. If the first confidence level is greater than or equal to the first confidence threshold, a first alarm is issued to the user. The first alarm is also used to prompt the user to confirm and / or process events associated with the first target detection result. If the first confidence level is less than the first confidence threshold, a second model is used to perform target detection on the video frame image to achieve re-judgment of the detection of the video frame image, thereby obtaining a third target detection result and a third confidence level of the third target detection result.

[0159] Following the foregoing examples, in some embodiments, the first target detection result further includes multiple sub-detection information and / or at least one IOU value. Each sub-detection information corresponds to a target and is used to describe the target category of the corresponding target and the image region within the video frame image. Each IOU value is the degree of overlap between any two sub-detection information. The first confidence level also includes the confidence level of multiple sub-detection information.

[0160] Therefore, in some embodiments, if the first confidence level is less than the first confidence threshold, then when performing target detection on the video frame image based on the second model, part or all of the process shown in Figure 7 can be executed; it should be noted that the process shown in Figure 7 is only one specific implementation:

[0161] Next, as shown in Figure 7, which is another flowchart of a model deployment method in an embodiment of this disclosure, including the following steps:

[0162] S700: Based on the confidence of multiple sub-detection information, select target sub-detection information with a confidence of less than the sub-confidence threshold from the multiple sub-detection information, and / or, based on the relationship between any two sub-detection information, select target sub-detection information from the multiple sub-detection information, wherein the relationship includes the IOU value and / or coordinate position relationship between the two associated sub-detection information.

[0163] The coordinate positional relationship includes, but is not limited to, coordinate distance and coordinate distribution.

[0164] Taking the first model as an example of a smoking model in a track scene, the first model usually detects core targets such as heads (head and shoulders), cigarettes, and hands, and obtains sub-detection information for multiple targets, namely head bounding boxes, cigarette bounding boxes, and hand bounding boxes. Then, based on the coordinate positions of the cigarette bounding box, hand bounding box, and head bounding box, as well as the confidence of each sub-detection information, the IOU of the cigarette bounding box, hand bounding box, and head bounding box is analyzed to determine whether smoking behavior exists (whether a smoking event exists).

[0165] S710: Based on the selected target sub-detection information, generate a first prompt instruction, wherein the first prompt instruction is used to guide the second model to perform target detection on the image region where the target described by the target sub-detection information is located within the video frame image.

[0166] For example, if the first model detects a head, a hand, and a cigarette, but the confidence level of the cigarette target bounding box (i.e., the aforementioned sub-detection information) is low, then step S700 is executed. The target sub-detection information selected from multiple sub-detection information includes the cigarette target bounding box. When step S710 is executed, a first prompt instruction is generated based on the selected target sub-detection information, as described in the prompt description, to guide the second model to perform target detection on the cigarette target bounding box and determine whether the hand target bounding box contains a cigarette.

[0167] For example, if the first model detects a head, a hand, and a cigarette, but the IOU value between the cigarette target box and the head target box is less than the IOU threshold, meaning there is a certain distance between the cigarette and the head, then step S700 is executed. The target sub-detection information selected from multiple sub-detection information includes the cigarette target box and the hand target box. When step S710 is executed, the first prompt instruction generated based on the selected target sub-detection information is used to guide the second model to perform target detection on the cigarette target box and the hand target box to determine whether the target pedestrian is holding a cigarette and about to smoke.

[0168] For example, if the first model only detects a head and a cigarette, then step S700 is executed, where the target sub-detection information selected from multiple sub-detection information includes the head target bounding box and the cigarette target bounding box; when step S710 is executed, the first prompt instruction generated based on the target sub-detection information is used to guide the second model to perform target detection on the head target bounding box and the cigarette target bounding box to determine whether the target pedestrian has a cigarette in their mouth and is smoking.

[0169] Figure 8 is a schematic diagram of the track scene in an embodiment of this disclosure. Referring to Figure 8, it is assumed that the first target detection information detected by the first model includes two sub-detection information, such as head detection information, as shown in Figure 8 (head target box); cigarette detection information, as shown in Figure 8 (cigarette target box); and the confidence level of the head detection information is 0.91, as shown in Figure 8; the confidence level of the cigarette detection information is 0.47.

[0170] For example, in Figure 8, the head bounding box and its confidence score, as well as the cigarette bounding box and its confidence score, represent what the first model considers a person and a cigarette that might be smoking. Based on these head and cigarette bounding boxes, a first prompt is generated to guide the second model in performing target detection on these boxes. This prompt determines whether the person (large box) at coordinates (x1, y1) and (x2, y2) in Figure 8 has a cigarette in their mouth, and whether the target (small box) at coordinates (x3, y3) and (x4, y4) in Figure 8 is a cigarette.

[0171] S720: Input the video frame image into the second model, trigger the second model to perform target detection according to the first prompt instruction, obtain the third target detection result and the third confidence level of the third target detection result, wherein the third confidence level is used to characterize the accuracy of the third target detection result.

[0172] In some embodiments, after generating the first prompt instruction, during step S720, the first prompt instruction and the video frame image can be input into the second model, triggering the second model to perform target detection on the video frame image according to the first prompt instruction, and obtain a third target detection result and a third confidence level of the third target detection result. Using the above process can effectively improve the accuracy of the second model in detection and re-judgment.

[0173] In other embodiments, when performing step S710, multiple first sub-hint instructions can be generated by combining technologies such as thought chain, and the third sub-target detection result and the third confidence level of the third sub-target detection result obtained by the second model according to the multiple first sub-hint instructions can be used to obtain the final third target detection result and the third confidence level of the third target detection result, thereby further improving the accuracy of large model re-judgment.

[0174] S730: When the third confidence level is greater than or equal to the second confidence level threshold, a third alarm is sent to the user, wherein the third alarm is used to prompt the user to confirm and / or process the event associated with the third target detection result.

[0175] In some embodiments, when performing step S303, steps S700 to S720 can also be used to obtain the second target detection result and the second confidence level of the second target detection result. That is, before inputting the video frame image into the second model for target detection, a more specific and targeted prompt description (such as a first prompt instruction) is generated based on the first target detection result of the first model, which improves the accuracy of the second model for the video frame image and also improves the detection speed of the second model for target detection of the video frame image. The specific method for generating the prompt description is similar to the description of steps S700 to S720, and will not be repeated here.

[0176] In real-world scenarios, users may request the creation of a new detection model for a new business requirement, or for a business requirement for which a corresponding detection model has already been deployed but whose performance is poor. In such cases, based on the aforementioned end-side device structure, another model deployment method can be adopted to match the detection model to the above business requirement.

[0177] In some embodiments, the aforementioned multiple business requirements include second business requirements, such as the aforementioned new business requirements or business requirements that require setting up a new detection model. Based on the second business requirements, when executing a model deployment method provided in this disclosure embodiment, part or all of the process shown in FIG9 may also be executed; it should be noted that the process shown in FIG9 is only one specific implementation method:

[0178] Next, as shown in Figure 9, which is another flowchart of a model deployment method in an embodiment of this disclosure, the method includes the following steps:

[0179] S900: Receive a second prompt instruction, wherein the second prompt instruction is used to instruct the second model to perform target detection under the second business requirement based on each sample included in the validation set.

[0180] In practical use, users may request target detection for a second business requirement for which no relevant detection model has been deployed. Since a second model has been deployed on the edge device, and the second model has target detection capabilities for various business requirements, corresponding prompt content, i.e., a second prompt instruction, can be designed for the second business requirement to trigger the second model to perform target detection for the second business requirement based on each sample included in the validation set, in order to verify the accuracy of the second model for the second business requirement.

[0181] S910: Execute the second prompt instruction on the second model to obtain the accuracy of the second model under the second business requirement.

[0182] In some embodiments, when performing step S910, the second model can be triggered to perform target detection on each sample included in the validation set for the second business requirement based on the second prompt instruction, and the accuracy of the second model under the second business requirement can be obtained based on the preset model evaluation index.

[0183] S920: If the accuracy of the second model is less than the model accuracy threshold, then train the second model until the accuracy of the second model is greater than or equal to the model accuracy threshold.

[0184] In some embodiments, after obtaining the accuracy of the second model under the second business requirement, it is compared with the model accuracy threshold. If the accuracy of the second model is greater than or equal to the model accuracy threshold, the second model is used as the model for performing target detection under the second business requirement.

[0185] In other embodiments, if the accuracy of the second model is less than the model accuracy threshold, it is necessary to collect some scenario data under the second business requirement, and train the second model based on the collected scenario data, so as to fine-tune the second model into a target detection model under the second business requirement with an accuracy greater than or equal to the model accuracy threshold.

[0186] S930: Replace the current second model with the trained second model to perform object detection for the second business requirement.

[0187] In some embodiments, after obtaining the target detection model for the second business requirement, the current second model can be directly replaced by the trained second model, or a model replacement operation can be performed according to the actual situation, thereby achieving the goal of quickly responding to user business requirements and quickly launching algorithms / models.

[0188] In other embodiments, the recall rate of the second model can also be considered when validating and training the second model to ensure that the accuracy and recall rate of the final online model can achieve good results. The specific methods are as described above and will not be repeated here.

[0189] As mentioned above, the first model in this embodiment is used for target detection under a first business requirement, while the second model is used for target detection under multiple business requirements that include the first business requirement. The model structure should be set according to the specific business requirements in the actual scenario. For example, the first model can be a target detection model for target detection, such as the YOLO series, RNN series, etc., or the first model can be an image segmentation model, such as MaxViT, HRNet, etc.

[0190] Since the second model is used to handle target detection under various business requirements, a multimodal large model that covers multiple modalities such as text, image, and audio can be considered in actual deployment.

[0191] Based on this, the model deployment method provided in this disclosure can also acquire a multimodal large model and a target video. The multimodal large model includes a fusion structure, a language large model, and at least two video encoders. Initial video features of the target video are extracted using at least two video encoders, and the initial video features from multiple video encoders are fused using the fusion structure to obtain the target video features. The "multimodal large model" in the above process can be the second model deployed on the edge device mentioned earlier. Through the above process, refined feature extraction of the video can be achieved, thereby enhancing the video detail description capability of the multimodal large model.

[0192] Figure 10 is a schematic diagram of a multimodal large model for video understanding in related technologies. The video understanding multimodal large model is a visual-language multimodal large model composed of a video encoder and a large language model, which can be used to process video understanding tasks such as video description, video question answering, video classification, video retrieval, and video temporal localization.

[0193] The operation flow of the model can be shown in Figure 10, including the following steps:

[0194] S1001, Video Input, inputs video into the video encoder;

[0195] S1002 maps the video to a video feature vector and inputs the video feature vector into the large language model;

[0196] S1003, input the word embedding vectors of the text prompts into the large language model;

[0197] S1004, a large language model, combines the input word embedding vectors and video feature vectors to output the answer to the text question.

[0198] The video input is mapped to a set of feature vectors by a video encoder. Then, the word embedding vectors of the text prompt input and the video feature vectors are input together into a large language model. Finally, the corresponding text is input according to different tasks. For example, for video description, a detailed description text is input, and for video question answering, the answer to the text question is output.

[0199] Although the industry has made some progress in using multimodal large models for video understanding, many problems still exist in practical applications. The most typical problem is that the output text of multimodal large models is very brief and difficult to describe more detailed information.

[0200] Taking the video above as an example: the text prompt input is "Please describe the video content in detail", and the model output is "The video is a busy road with many vehicles driving"; while the actual desired output of the multimodal large model is "This is a video taken by a surveillance camera, showing a city street scene. (1) Traffic conditions: There are 6 cars and 1 pedestrian motorcycle in the left lane, and 1 car in the right lane...; (2) Roads and facilities: The road is a two-way four-lane road, separated by double yellow lines...; (3) Traffic signs: There is a traffic light in the video, which is green...; (4) Environment and weather: The weather in the video is green, which may be a summer scene....

[0201] One important reason for the aforementioned problems in multimodal large models when performing video understanding is that the video encoder is not powerful enough to encode various detailed features in the video, which in turn makes it difficult for large language models to understand the detailed content.

[0202] Figure 11 is another flowchart illustrating a model deployment method provided in this embodiment of the present disclosure, including the following steps:

[0203] S1101, acquire a multimodal large model and a target video; wherein, the multimodal large model includes a fusion structure, a language large model, and at least two video encoders;

[0204] S1102: Input the target video into the at least two video encoders respectively, and obtain each initial video feature matrix output by the at least two video encoders.

[0205] Multimodal large-scale models for video understanding consist of video encoders and large language models, and can be used to handle video understanding tasks such as video description, video question answering, video classification, video retrieval, and video temporal localization. Existing multimodal large-scale models for video understanding mainly use a single video encoder to encode the input video, which makes it difficult to accurately encode video information and limits the video detail description capabilities of the large-scale model. Therefore, this disclosure employs multiple video encoders to extract features in the above process and performs feature fusion through multiple video encoders to achieve refined feature extraction from the video, thereby enhancing the video detail description capabilities of the multimodal large-scale model.

[0206] In some embodiments, the multimodal large model comprises at least two video encoders, a fusion structure, and a language large model. The edge device inputs the target video into at least two video encoders of the multimodal large model, and the at least two video encoders respectively extract features from the target video to determine and output an initial video feature matrix corresponding to the target video. The edge device acquires each initial video feature matrix output by each video encoder.

[0207] Specifically, the function of a video encoder is to process the input video sequence V = {I1, I2, ..., I...} l}, mapped to the initial video feature matrix f∈R n×d , where I k Let k = 1, 2, ..., l represent the k-th frame in the video, where k is a positive integer, l represent the length of the video, n represent the number of row vectors in the initial video feature matrix (which can also be called video tokens), and d represent the dimension of each row vector, i.e., the dimension of each video token. Let the video encoder function be VE, then a single video encoder can be expressed by the formula: f1 = VE1(I1, I2, ..., I... l f2=VE2(I1,I2,…,I) l ) … f N =VE N (I1,I2,…,I l )

[0208] Among them, VE k k = 1, 2, ..., N represents N video encoders, f1, f2, ..., f N This represents the initial video feature matrix extracted by the corresponding video encoder.

[0209] S1103: Input each initial video feature matrix into the fusion structure, so that the fusion structure concatenates each initial video feature matrix to obtain a candidate video feature matrix, and maps the candidate video feature matrix to a target video feature matrix with the same number of rows and columns as the weight matrix according to a preset weight matrix.

[0210] In some embodiments, the multimodal large model also includes a fusion structure that fuses the initial video feature matrices output by each video encoder to obtain a target video feature matrix.

[0211] Specifically, each initial video feature matrix output by each video encoder is input into the fusion structure, which concatenates each initial video feature matrix to obtain a candidate video feature matrix. Based on a preset weight matrix, the candidate video feature matrix is ​​mapped to a target video feature matrix with the same number of rows and columns as the weight matrix.

[0212] The purpose of the fusion structure is to combine the initial video feature matrices f1, f2, ..., f from multiple video encoders. N Fusion is performed to extract the advantageous features of each encoder, thereby improving the feature representation capability of the entire video encoder. Let the fusion structure be Fuse, then the formulaic expression is F = fuse(f1, f2, ..., f...). N )

[0213] Where F∈R L×d Here, L represents the final number of video tokens (row vectors), and d represents the dimension of each video token.

[0214] In some embodiments, the number of rows and columns of the target video feature matrix depends on the number of rows and columns of the weight matrix. The number of rows and columns of the weight matrix may or may not be the same as the number of rows and columns of the initial video feature matrix, and no restriction is imposed here.

[0215] Furthermore, when the fusion structure concatenates each initial video feature matrix, it does so based on the row vectors of the initial video feature matrix. For example, if each initial video feature matrix contains 3 row vectors and there are a total of 3 initial video feature matrices, then the concatenated candidate video feature matrix will have 9 rows, while the number of columns remains unchanged.

[0216] S1104: Input the target video feature matrix and instruction prompts into the language big model to obtain the target text output by the language big model.

[0217] In some embodiments, after determining the target video feature matrix of the target video through the fusion structure, the target video feature matrix and the instruction prompt word are input into the language big model. The language big model processes the target video feature matrix according to the instruction prompt word, determines and outputs the target text, and the edge device obtains the target text output by the language big model.

[0218] Figure 12 is a schematic diagram of the application of the multimodal large model of the multi-video encoder provided in the embodiment of this disclosure. As shown in Figure 12, the following steps may be included:

[0219] S1201, Video Input;

[0220] S1202, the video input is mapped to multiple initial video features by multiple video encoders, and the multiple initial video features are input into the fusion structure;

[0221] S1203, based on the fusion structure, fuses each initial video feature matrix into a target video feature matrix and inputs it into a large language model;

[0222] S1204, input the instruction prompt words from the text prompt into the large language model;

[0223] S1205, a large language model, combines the target video feature matrix and instruction prompts to output the answer to the text question.

[0224] The video input is mapped to multiple initial video features by multiple video encoders. Then, based on the fusion structure, each initial video feature matrix is ​​fused into a target video feature matrix. The instruction prompts and the target video feature matrix are then input together into a large language model. Finally, the corresponding text is input according to different tasks, such as inputting detailed description text for video description, and outputting the answer to the text question for video question answering, etc.

[0225] In addition, the edge device can also determine the text vector corresponding to the instruction prompt word, and input the text vector and the target video feature matrix together into the large language model to obtain the target text output by the large language model.

[0226] In some embodiments, multiple video encoders are used to extract initial video features of the target video, and the initial video features of the multiple video encoders are fused to obtain the target video features, thereby achieving refined extraction of video features and enhancing the ability to describe the details of multimodal large model videos.

[0227] To improve the quality of text output by a multimodal large model during video understanding and enhance video understanding capabilities, when mapping the candidate video feature matrix to a target video feature matrix with the same number of rows and columns as the weight matrix according to a preset weight matrix, a first product matrix of the transpose of the weight matrix and the candidate video feature matrix can be determined. The ratio of the first product matrix to a preset value is determined as the attention score matrix. The attention score matrix is ​​normalized to determine the probability distribution matrix. A second product matrix of the probability distribution matrix and the candidate video feature matrix is ​​determined, and the second product matrix is ​​determined as the target video feature matrix.

[0228] In some embodiments, the fusion structure in a multimodal large model can be a cross-attention structure, which pre-stores a weight matrix. The fusion structure performs cross-attention processing on the candidate video feature matrix according to the weight matrix, mapping the candidate video feature matrix to a target video feature matrix with the same number of rows and columns as the weight matrix.

[0229] In some embodiments, the fusion structure determines a first product matrix of the weight matrix and the transpose of the candidate video feature matrix, and determines the ratio of the first product matrix to a preset value as the attention score matrix; the attention score matrix is ​​normalized to determine the probability distribution matrix; a second product matrix of the probability distribution matrix and the candidate video feature matrix is ​​determined, and the second product matrix is ​​determined as the target video feature matrix.

[0230] The preset value is the dimension of each row vector in the candidate video feature matrix, that is, the number of columns in the candidate video feature matrix.

[0231] The fusion structure can be determined using the following formula to define the target video feature matrix: F t =concat(f1,f2,…,f N K = V = F t

[0232] Where, Q∈R L×d Here, is the weight matrix, are learnable parameters, `concat` represents the concatenation operation, and `softmax` is the normalized softmax operation. This structure allows for the processing of any number of video terms F. t The output is transformed into a fixed number of L video words, each with a dimension of d.

[0233] Figure 13 is a schematic diagram of a process for determining the target video feature matrix using a fusion structure according to an embodiment of this disclosure, including the following steps:

[0234] S1301: Determine the first product matrix of the transpose of the weight matrix and the candidate video feature matrix, and determine the ratio of the first product matrix to a preset value as the attention score matrix.

[0235] S1302: Normalize the attention score matrix to determine the probability distribution matrix.

[0236] S1303: Determine the second product matrix of the probability distribution matrix and the candidate video feature matrix, and determine the second product matrix as the target video feature matrix.

[0237] To improve the quality of text output by multimodal large models during video understanding and enhance video understanding capabilities, based on the above embodiments, before mapping the candidate video feature matrix to a target video feature matrix with the same number of rows and columns as the weight matrix according to a preset weight matrix, the method further includes:

[0238] Each initial video feature matrix is ​​split into a predetermined number of sub-video feature matrices;

[0239] For each sub-video feature matrix, determine each product of each row vector of the sub-video feature matrix with each pre-configured weight column vector; determine each product vector of each row vector and its corresponding product, and update the sub-video feature matrix using the sum of the product vectors.

[0240] Based on the updated feature matrix of each sub-video, the updated candidate video feature matrix is ​​determined.

[0241] In some embodiments, since the initial video feature matrix output by each video encoder contains a large number of video terms, that is, each initial video feature matrix contains many row vectors, if each initial video feature matrix is ​​directly fused, the computational load of the fusion structure will be very large, resulting in low video processing efficiency.

[0242] Based on this, before concatenating the initial video feature matrices, the fusion structure first reduces the number of video terms contained in each initial video feature matrix, and then concatenates the updated initial video feature matrices to obtain the candidate video feature matrices.

[0243] Specifically, the fusion structure splits each initial video feature matrix into a preset number of sub-video feature matrices; for each sub-video feature matrix, it determines the product of each row vector of the sub-video feature matrix with each pre-configured weight column vector; it determines each product vector of each row vector and its corresponding product, and updates the sub-video feature matrix using the sum of each product vector; and it determines the updated candidate video feature matrix based on each updated sub-video feature matrix.

[0244] That is, for each row vector, the fusion structure determines the product of the row vector and the pre-configured weight column vector, uses the product as the weight of the row vector, and performs a weighted summation of the row vectors in each group with the corresponding products (weights). The calculated sum vector is then used to update the sub-video feature matrix corresponding to the group, so that the updated sub-video feature matrix of each group is a row vector, thus reducing the amount of data.

[0245] In some embodiments, the number of rows in the updated candidate video feature matrix is ​​the number of groups.

[0246] In addition, when the edge device groups the initial video feature vector matrix, it groups according to the row vectors. That is, the fusion structure has a preset number of groups. The fusion structure performs a rounding operation on the total number of rows of the initial video feature vector matrix according to the preset grouping, and determines the number of row vectors contained in each group by the final rounding result.

[0247] To improve the quality of the text output by the multimodal large model during video understanding and enhance the video understanding capability, a normalization operation can be performed on each product after determining each row vector of the sub-video feature matrix and each pre-configured weight column vector.

[0248] In some embodiments, in order to further improve the quality of the text output by the multimodal large model during video understanding and enhance the video understanding capability, the fusion structure performs a normalization operation on each product after determining each row vector of the sub-video feature matrix and each pre-configured weight column vector.

[0249] The fusion structure can reduce the number of video terms in each initial video feature matrix using the following formula: f i1 ,f i2 ,…,f ic =split(f i ) s ij =softmax(f ij w i ) f' ij =∑s ij f ij f' i =concat(f' i1 ,f' i2 ,…,f' ic )

[0250] Among them, split(f i ), i = 1, 2, ..., k represents the rounding operation on the initial video feature matrix, dividing the original n row vectors into c groups, with each group containing int(c / n) row vectors (video words), f ij For video terms, w i Let f' be the weight column vector. ij For the updated sub-video feature matrix, f' i This represents the updated candidate video feature matrix.

[0251] Figure 14 is a schematic diagram of the application of the grouped attention fusion structure provided in the embodiments of this disclosure. As shown in Figure 14, the following steps are included:

[0252] (1) First, the initial video feature matrix obtained by each video encoder is grouped to obtain a sub-video feature matrix f with a fixed number of rows c. i ;

[0253] (2) Then, adaptive weight calculation is performed for each group, that is, the row vector (f) in each sub-video feature matrix. i1 f i2 ... f ic The vector product of (S) and the weight column vector (w) i1 S i2 ... S ic ), and then processed by softmax normalization fusion.

[0254] (3) Next, the normalized product is multiplied and summed with the row vectors in each sub-video feature matrix to obtain the fused sum vector, which represents the features of the group, and then the candidate video feature matrix of the video encoder is determined.

[0255] (4) Finally, the candidate video feature matrices of multiple video encoders are fused using cross-attention to obtain the final target video feature matrix.

[0256] Figure 15 is a schematic diagram of the process for updating the sub-video feature matrix provided in an embodiment of this disclosure, including the following steps:

[0257] S1501: Determine the product of each row vector of the sub-video feature matrix with each pre-configured weight column vector.

[0258] S1502: Perform normalization operation on each product separately.

[0259] S1503: Determine each product vector of each row vector and its corresponding product, and update the sub-video feature matrix using the sum vector of each product vector.

[0260] Figure 16 is a schematic diagram of another fusion structure for determining the target video feature matrix provided in an embodiment of this disclosure. The process includes:

[0261] S1601: Split each initial video feature matrix into a preset number of sub-video feature matrices.

[0262] S1602: For each sub-video feature matrix, update the sub-video feature matrix according to each row vector of the sub-video feature matrix and the pre-configured weight column vector.

[0263] S1603: Determine the updated candidate video feature matrix based on the updated feature matrix of each sub-video.

[0264] S1604: Determine the first product matrix of the weight matrix and the transpose of the updated candidate video feature matrix, and determine the ratio of the first product matrix to a preset value as the attention score matrix.

[0265] S1605: Normalize the attention score matrix to determine the probability distribution matrix.

[0266] S1606: Determine the second product matrix of the probability distribution matrix and the updated candidate video feature matrix, and determine the second product matrix as the target video feature matrix.

[0267] To improve the quality of the text output by the multimodal large model during video understanding and enhance video understanding capabilities, the multimodal large model can be trained in the following ways:

[0268] Freeze the parameters in the first parameter group of the at least two encoders and the parameters in the second parameter group of the language big model, and set the parameters in the third parameter group of the fusion structure to their initial values;

[0269] Obtain the first sample video and its corresponding first sample text from the dataset;

[0270] The first sample video is processed using the multimodal large model to obtain the first predicted sample text output by the multimodal large model;

[0271] Based on the first predicted sample text and the first sample text, the parameters in the third parameter group of the fusion result of the multimodal large model are adjusted.

[0272] In some embodiments, since the multimodal large model includes multiple video encoders, a fusion structure, and a language large model, wherein the multiple video encoders are pre-configured, the parameters of the fusion structure and the language large model can be adjusted separately when training the multimodal large model.

[0273] Specifically, the edge device freezes the parameters in the first parameter group of at least two encoders and the parameters in the second parameter group of the language large model, and sets the parameters in the third parameter group of the fusion structure to the initial values, so that when the edge device trains the multimodal large model, it will only adjust the parameters in the third parameter group of the fusion structure.

[0274] The edge device acquires the first sample video and the corresponding first sample text carried in the dataset; it uses a multimodal large model to process the first sample video and obtains the first predicted sample text output by the multimodal large model; based on the first predicted sample text and the first sample text, it adjusts the parameters in the third parameter group in the fusion result of the multimodal large model.

[0275] In some embodiments, the parameters in the third parameter group of the fusion structure include, but are not limited to, the weight matrix and weight column vector in the above embodiments.

[0276] To improve the quality of the text output by the multimodal large model during video understanding and enhance video understanding capabilities, the parameters in the second parameter group of the language large model can be unfrozen; the second sample video and its corresponding second sample text carried in the dataset can be obtained; the second sample video can be processed using the multimodal large model to obtain the second predicted sample text output by the multimodal large model; based on the second predicted sample text and the second sample text, the parameters in the third parameter group of the fusion result of the multimodal large model and the parameters in the second parameter group of the language large model can be adjusted.

[0277] In some embodiments, after adjusting the parameters of the third parameter group in the fusion structure, the edge device can also adjust the parameters of the second parameter group of the language big model.

[0278] Specifically, the edge device unfreezes the parameters in the second parameter group of the language model, so that this training can adjust the parameters of both the third parameter group of the fusion structure and the second parameter group of the language model.

[0279] The edge device acquires the second sample video and the corresponding second sample text carried in the dataset; it uses a multimodal large model to process the second sample video and obtains the second predicted sample text output by the multimodal large model; based on the second predicted sample text and the second sample text, it adjusts the parameters in the third parameter group of the fusion result of the multimodal large model and the parameters in the second parameter group of the language large model.

[0280] The multimodal large model provided in this disclosure can perform multiple computer vision tasks with a single model, such as pedestrian detection, vehicle detection, video event recognition, video description, and video question answering. The following uses a traffic scenario as an example to illustrate how to use the video understanding multimodal large model.

[0281] The specific processing steps are as follows:

[0282] Step 1: Given a trained multimodal large model, input a command prompt and the video to be processed.

[0283] Instruction prompts include phrases such as "Detect all pedestrians in the video", "Describe the video content", and "Did a car accident occur in the video? Please answer with yes or no".

[0284] Step 2: The multimodal large model encodes and processes the instruction prompts and video, and outputs the text answer.

[0285] To improve the quality of the text output by the multimodal large model during video understanding and enhance the video understanding capability, based on the above embodiments, at least two video encoders may be selected from at least two of Video BERT, Act BERT, Video MAE, Vi VIT, and Slow Fast.

[0286] In some embodiments, the initial video feature matrix can be extracted using various video encoders, such as Video BERT, Act BERT, Video MAE, Vi VIT, and Slow Fast. It is worth noting that when using these video encoders, the last feature map layer corresponding to a specific category is removed, retaining only the feature map of the penultimate layer, which is then mapped to a d-dimensional vector through a fully connected layer or convolutional layer to obtain the initial video feature matrix.

[0287] Figure 17 is a schematic diagram of the video processing flow based on a multimodal large model provided in this embodiment of the present disclosure, including the following steps:

[0288] S1701: Input the target video into at least two video encoders respectively, and obtain each initial video feature matrix output by at least two video encoders.

[0289] S1702: Input each initial video feature matrix into the fusion structure, so that the fusion structure concatenates each initial video feature matrix to obtain a candidate video feature matrix.

[0290] S1703: The fusion structure splits each initial video feature matrix into a preset number of sub-video feature matrices, and updates each sub-video feature matrix according to each row vector of each sub-video feature matrix and the pre-configured weight column vector.

[0291] S1704: The fusion structure determines the updated candidate video feature matrix based on the updated feature matrix of each sub-video.

[0292] S1705: The fusion structure maps the candidate video feature matrix to the target video feature matrix with the same number of rows and columns as the weight matrix, according to the preset weight matrix.

[0293] S1706: Input the target video feature matrix and instruction prompts into the language model to obtain the target text output by the language model.

[0294] The model deployment method provided in this embodiment can also input the target image to be detected and a preset number of other images adjacent to the acquisition time of the target image into a small accident detection model, and obtain the target confidence score of the target image output by the small accident detection model to indicate the existence of a traffic accident; if the target confidence score belongs to a preset confidence score range, then the target image is input into a multimodal large model for detecting traffic accidents, and the detection result of whether there is a traffic accident in the target image output by the multimodal large model is obtained. This alleviates the problem that the current traffic accident detection mainly relies on manually defined accidents, resulting in poor detection accuracy and difficulty in meeting the needs of a wide range of scenarios.

[0295] The "small accident detection model" in the above process refers to the first model deployed on the edge device, and the "large multimodal model for detecting traffic accidents" refers to the second model deployed on the edge device. In accident detection scenarios, the first and second models deployed on the edge device can be processed in the above way to achieve traffic accident detection by combining the large and small models, thereby improving the accuracy of accident detection.

[0296] Figure 18 is another flowchart illustrating a model deployment method provided in an embodiment of this disclosure. As shown in Figure 18, it includes the following steps:

[0297] S1801: Input the target image to be detected and a preset number of other images adjacent to the acquisition time of the target image into the accident detection mini-model, and obtain the target confidence that a traffic accident exists in the target image output by the accident detection mini-model.

[0298] In practice, vehicles in the image are first detected, and then the Kalman filter algorithm is used to predict the vehicle's position in the next frame. The optimal match for the target vehicle is then obtained using the Hungarian matching algorithm, thus enabling vehicle tracking. When a traffic accident occurs, the vehicle stops, and the location and status of the vehicle, as well as whether anyone got out to check, are used to determine if a traffic accident has occurred. However, due to the diverse types and varying circumstances of traffic accidents, the approach of object detection + tracking + business logic results in poor accident identification accuracy and is time-consuming and labor-intensive.

[0299] Because multimodal large models have a strong ability to understand images, they can directly obtain whether a traffic accident has occurred in an image through command-and-response methods and have high detection accuracy. However, multimodal large models have a large number of parameters, high resource consumption, and long detection time, making it impossible to process each frame of the image in real time.

[0300] Based on this, the end-side device of this embodiment can first use a small model to determine the confidence level of the target image to be detected as containing a traffic accident, and then further determine whether a multimodal large model is needed for accident detection based on the target confidence level.

[0301] Specifically, the edge device acquires a video stream, decodes it, and extracts frames according to a set algorithm at a set frame rate to obtain multiple images. Based on the input, the edge device can determine the target image to be detected from these multiple video frames, and also determine a preset number of other images whose acquisition time is adjacent to that of the target image. These other images are used to track the target in the target image and determine its motion trajectory.

[0302] The edge device inputs the target image and a preset number of other images into the accident detection mini-model. The accident detection mini-model performs target detection on the target image, identifies each target contained in the target image, determines the motion trajectory corresponding to each target based on the other images, and determines the target confidence that a traffic accident exists in the target image based on the motion estimation corresponding to each target. Each target contained in the target image can be a person, a vehicle, etc.

[0303] S1802: If the target confidence level belongs to a preset confidence level range, the target image is input into a multimodal large model for detecting traffic accidents, and the detection result of whether a traffic accident exists in the target image output by the multimodal large model is obtained.

[0304] In some embodiments, the edge device can determine whether a traffic accident exists in a target image based on the target confidence score output by the accident detection mini-model. Specifically, the edge device stores a confidence score range. If the edge device determines that the target confidence score is less than the minimum value of the confidence score range, then the edge device determines that there is no traffic accident in the target image. If the edge device determines that the target confidence score is greater than the maximum value of the confidence score range, then the edge device determines that there is a traffic accident in the target image.

[0305] In some embodiments, if the edge device determines that the target confidence score output by the accident detection mini-model falls within a preset confidence score range, the edge device determines that the accident detection mini-model cannot accurately identify whether a traffic accident exists in the target image. Based on this, the edge device uses a multimodal large model for detecting traffic accidents to further detect the target image.

[0306] Specifically, if the edge device determines that the target confidence score output by the accident detection mini-model falls within a preset confidence score range, then the edge device inputs the target image into a multimodal large-scale model for detecting traffic accidents. The multimodal large-scale model then identifies the target image to determine whether a traffic accident exists in the target image, and outputs a detection result indicating whether a traffic accident exists in the target image. The edge device then acquires the detection result output by the multimodal large-scale model.

[0307] In some embodiments, the edge device inputs the target image to be detected and a preset number of other images acquired at a time adjacent to that of the target image into a small accident detection model, and obtains the target confidence score output by the small accident detection model indicating the presence of a traffic accident in the target image. If the target confidence score falls within a preset confidence score range, the target image is then input into a large multimodal model for detecting traffic accidents, and the detection result of whether a traffic accident exists in the target image is obtained from the output of the large multimodal model. In this embodiment of the disclosure, the combination of the large and small models achieves traffic accident detection and improves the accuracy of traffic accident detection.

[0308] In some embodiments, for images of the same traffic accident scene, accident-related events can still be detected even in different operating environments, which meets the reproducibility characteristic of the reliability feature; furthermore, the degree of accident detection can be determined by user-set hyperparameters (confidence range), and the degree of importance and adoption of the event detection results can be intervened by the user, which meets the controllability characteristic of the reliability feature. In addition to having the above two reliability features, the embodiments of this disclosure can greatly improve the detection efficiency and accuracy of traffic accident events.

[0309] To achieve traffic accident detection and improve its accuracy, the multimodal large model of this disclosure may include a visual encoder, a position-aware visual-language adapter, and a language large model. When obtaining the detection result of whether a traffic accident exists in the target image output by the multimodal large model, the visual encoder encodes the target image to determine the feature code corresponding to the target image; the position-aware visual-language adapter compresses the feature code to determine the compressed feature code; and the language large model determines and outputs the detection result of whether a traffic accident exists in the target image based on the compressed feature code.

[0310] In some embodiments, the multimodal large model includes a visual encoder, a position-aware visual-language adapter, and a language large model. The visual encoder is used to crop and feature-encode the input target image; the position-aware visual-language adapter is used for feature alignment and compression of the feature codes output by the visual encoder; and the language large model is used to output a detection result indicating whether a traffic accident exists in the target image based on the compressed feature codes.

[0311] For example, a multimodal large model could be the Qwen-VL model, which is a Transformer-based multimodal large model with a network structure consisting of three parts: the first part is a visual encoder, which uses VIT-bigG as the pre-trained weight initialization to crop the input image and encode features; the second part is a feature alignment module called a position-aware visual language adapter, which includes a single-layer cross-attention module and is mainly used to encode compressed image features; the third part is a language large model, which uses Qwen-7b as the pre-trained weight initialization to provide the multimodal large model with language generation capabilities.

[0312] In order to detect traffic accidents and improve the accuracy of traffic accident detection, the visual encoder of this disclosure includes a convolutional layer, a linear projection layer and a Transformer layer;

[0313] Encoding the target image to determine the feature code corresponding to the target image includes:

[0314] The convolutional layer uses dilated convolution to extract features from the target image, determines the feature map corresponding to the target image, and segments the feature map to obtain each sub-feature map; wherein, the parameters of the dilated convolution are the parameters learned during model training;

[0315] The linear projection layer sorts each sub-feature map according to the segmentation order of the feature map, and performs linear encoding on each sorted sub-feature map to determine the image features corresponding to the feature map;

[0316] The Transformer layer extracts features from the image and determines the feature encoding.

[0317] In some embodiments, a visual encoder may include convolutional layers, linear projection layers, and Transformer layers. Traditional convolutional layers use conventional convolution methods to extract features from the input image and then segment the image into local small images. This approach only performs simple segmentation at a single image level, while images are rich in features and the details of objects at different locations vary. Therefore, this simple segmentation method will fragment the positional information in the image.

[0318] Based on this, dilated convolution can be used to extract features from the target image. Since dilated convolution can increase the receptive field of the image, more image features can be input into the linear projection layer.

[0319] Figure 19 is a schematic diagram comparing traditional convolution and dilated convolution provided in the embodiments of this disclosure. As shown in Figure 19, the black part in a is the ordinary convolution kernel, and b and c are dilated convolutions. It can be seen that different holes result in different image features. Among them, a obtains the fewest image features, b obtains the next few, and c obtains the most.

[0320] In some embodiments, the parameter of dilated convolution is the dilation value of the dilated convolution. Traditionally, the dilation value is determined manually by setting different probabilities for the dilation value d, such as a probability of 0.24 for d equal to 1 and a probability of 0.17 for d equal to 2. As d increases, the probability gradually decreases. However, manually determining the value of d involves a degree of subjectivity and often fails to effectively optimize the model. Therefore, in this embodiment, it is proposed to set the dilation value of the dilated convolution as a learnable parameter, allowing the network model to automatically adjust the dilation value based on the error loss function and the optimizer.

[0321] The cross-entropy loss function is as follows: Loss=-[y*log e (p)+(1-y)*log e (1-p)]

[0322] Where y is the actual detection result of the sample image, and p is the predicted detection result.

[0323] The Adam optimizer is as follows: m t =β1m t-1 +(1-β1)g t

[0324] Where, m t For first-order momentum, v t For second-order momentum, m′ tand v′ t For averaging optimization, θ t+1 Update the weights.

[0325] Based on this, after the visual encoder is trained, in practical applications, the convolutional layers of the visual encoder use dilated convolution to extract features from the target image, determine the corresponding feature map, and segment the feature map to obtain each sub-feature map; the parameters of this dilated convolution are the dilation values, which are learned during model training. The linear projection layer of the visual encoder sorts each sub-feature map according to the segmentation order of the feature map, and performs linear encoding on each sorted sub-feature map to determine the image features corresponding to the feature map; the Transformer layer of the visual encoder extracts features from the image features to determine the feature encoding.

[0326] Figure 20 is an application flowchart of the visual encoder provided in the embodiments of this disclosure. As shown in Figure 20, it includes the following steps:

[0327] S2001, Input the target image, which is a 1920×1080, 3-channel color image, and initialize the dilation value d of the dilated convolution.

[0328] S2002, perform dilated convolution on the target image to obtain a feature map of size 224×224.

[0329] S2003: Divide the length and width of the feature image by 16 to form 196 image blocks of size 16×16, each image block having a dimension of 768. Arrange the image blocks into a sequence and input them into the linear projection layer to obtain the features of a single image block.

[0330] S2004 takes the * sign as the category token and is a learnable tensor of size 1×68. At the same time, a learnable positional code of size 1×768 is added. The positional code and the linearly encoded image features are added bit by bit and concatenated with the category token before being input into the Transformer layer.

[0331] The Transformer layer consists of multiple coding layers, each of which comprises a multi-head attention layer and a multi-layer perceptron, for a total of L coding layers.

[0332] In step S2005, the 197×768 dimension data obtained in step S2004 is normalized and input into the multi-head attention for calculation, resulting in a feature code of size 197×768. After normalization, the obtained 197×768 feature code is input into a multilayer perceptron layer. First, the input feature code is magnified by a factor of 4, becoming 197×3072, and then passed through a linear projection layer to obtain a feature code of size 197×768.

[0333] S2006, Output image features based on the feature encoding obtained in step S2005.

[0334] To achieve traffic accident detection and improve its accuracy, the accident detection sub-model of this disclosure may include a target detection sub-model, a target tracking sub-model, and an accident detection sub-model. When obtaining the target confidence score of a traffic accident in the target image output by the accident detection sub-model, the target detection sub-model performs target detection on the target image and other images respectively, identifies each target contained in the target image and other images, and labels each target in the target image and other images; wherein, the target is a vehicle or a pedestrian; the target tracking sub-model determines the motion trajectory of each target based on the labeled target image and other images; the accident detection sub-model performs accident detection based on the motion trajectory of each target, determines and outputs the target confidence score of a traffic accident in the target image.

[0335] In some embodiments, the accident detection mini-model is obtained by integrating multiple sub-models, which include at least a target detection sub-model, a target tracking sub-model, and an accident detection sub-model. The target detection sub-model is used to identify each target contained in the target image and other images; the target tracking sub-model is used to determine the motion trajectory of each target; and the accident detection sub-model is used to determine the confidence level of the target in the target image as indicating a traffic accident based on the motion trajectory.

[0336] Specifically, the target detection sub-model performs target detection on the target image and other images separately, identifying each target contained in the target image and other images, and annotating each target in the target image and other images. The target tracking sub-model determines the motion trajectory of each target based on the annotated target image and other images. The accident detection sub-model performs accident detection based on the motion trajectory of each target, determines and outputs the confidence score of targets in the target image that have traffic accidents.

[0337] The target detection sub-model can be a model using the Yolov5s algorithm, and the target tracking sub-model can be a model using the DeepSort tracking algorithm.

[0338] In some embodiments, the accident detection sub-model can determine the target confidence level of a traffic accident in a target image based on the following conditions:

[0339] Condition 1: The vehicles are side by side and there is an obstruction relationship, and the obstruction time exceeds the time threshold of 3 minutes;

[0340] Condition 2: The following vehicle is obstructed and remains stationary for more than 120 seconds;

[0341] Condition 3: The following vehicle passes around from behind, and passes slowly;

[0342] Condition 4: The doors of both vehicles involved in the accident are open, and people are getting out of the vehicles;

[0343] Condition 5: People stand around the vehicle for more than 60 seconds.

[0344] The confidence level for each of the above conditions is 0.2. The accident detection sub-model determines the number of conditions satisfied by the target image based on the motion trajectory of each target, and determines the target confidence level that there is a traffic accident in the target image based on the number of conditions and the confidence level corresponding to each condition.

[0345] In order to detect traffic accidents and improve the accuracy of traffic accident detection, based on the above embodiments, if the target confidence level exceeds the maximum value of the confidence level range, it is determined that there is a traffic accident in the target image and an accident alarm is triggered; if the target confidence level is less than the minimum value of the confidence level range, it is determined that there is no traffic accident in the target image and no accident alarm is triggered.

[0346] In this embodiment of the disclosure, the edge device has a confidence range stored in advance. If the confidence level output by the accident detection mini-model belongs to the confidence range, the edge device determines that the mini-model cannot accurately identify whether there is a traffic accident in the target image. The edge device then inputs the target image into the multimodal large model for further prediction.

[0347] Based on this, if the target confidence score output by the accident detection mini-model exceeds the maximum value of the confidence score range, the edge device determines that there is a traffic accident in the target image and issues an accident alarm; if the target confidence score output by the accident detection mini-model is less than the minimum value of the confidence score range, the edge device determines that there is no traffic accident in the target image and does not issue an accident alarm.

[0348] The confidence level can be in the range of 0.3-0.7. If the confidence level is greater than 0.7, it can be directly judged that a traffic accident has occurred. If the confidence level is greater than 0.3 and less than 0.7, the target image is input into the multimodal large model for decision-making. If the confidence level is less than 0.3, it is considered that no traffic accident has occurred.

[0349] To improve the accuracy of traffic accident detection, based on the above embodiments, this disclosure also provides a dataset construction method for training the multimodal large model described in the above embodiments. Figure 21 is a schematic diagram of the dataset construction process provided in this disclosure, which includes:

[0350] S2101: Input the sample image into the image language pre-training model and obtain the overall description corresponding to the sample image output by the image language pre-training model.

[0351] In related technologies, dataset construction mainly employs two methods: manual annotation and large-model annotation. Manual annotation involves human input, but since traffic accidents are diverse and cannot be exhaustively listed, and when multiple workers annotate, differing interpretations of the accidents in the images lead to inconsistent descriptions, manual annotation suffers from drawbacks such as being time-consuming, labor-intensive, and inconsistent. Large-model-based data annotation relies directly on large models like GPT4 for answers. However, when fine-tuning training data produced in this way, the model being fine-tuned cannot understand why traffic accidents occurred.

[0352] Based on this, in order to better annotate images and obtain a more accurate dataset, and to fine-tune and train a multimodal large model for accident detection based on this dataset, in this embodiment of the disclosure, the multimodal large model's ability to understand images is combined with the detection of the small model to annotate multiple images from multiple levels, generate descriptions with consistent expression styles, and introduce thought chains into the data annotation. Through step-by-step prompts and answers, the large model understands human intentions, and finally, manual verification or sampling detection is performed.

[0353] Specifically, the edge device inputs the sample image into the image-language pre-trained model, enabling the model to recognize the entire sample image and output a holistic description. The edge device then obtains the holistic description of the sample image output by the image-language pre-trained model.

[0354] Figure 22 is a schematic diagram of a sample image provided in an embodiment of this disclosure. As shown in Figure 22, the end device inputs the sample image into the image language pre-training model, and the overall description output by the image language pre-training model is "many vehicles are driving on the city roads".

[0355] S2102: Input the sample image into the region description pre-training model to obtain the sample image with the detection box corresponding to each region and the region description corresponding to each region output by the region description pre-training model.

[0356] In this embodiment of the disclosure, the edge device inputs a sample image into a region description pre-training model, which detects each region in the sample image and generates a corresponding region description for each region.

[0357] In this embodiment of the disclosure, the regions in the sample image can be automatically identified by the region description pre-training model. Alternatively, each region in the sample image can be identified and labeled by a small model. Then, the labeled sample image is input into the region description pre-training model, which outputs the region description corresponding to each region.

[0358] Figure 23 is a schematic diagram of the sample image provided in the embodiments of this disclosure and the output of the region description pre-training model. As shown in Figure 23, after the sample image is input into the region description pre-training model, the region description pre-training model determines each region contained in the sample image and the corresponding region description and marks it in the sample image. The region description includes: "The car is white", "A woman is standing in front of the car", "A silver car is parked on the road", and "A person is standing behind the car". The positions of the region boxes are: [806,599,961,796], [759,644,828,848], [872,762,1242,1032] and [1089,867,1182,1121].

[0359] S2103: Determine the target description of the sample image based on the overall description and the region description corresponding to each region; obtain the input accident Q&A information determined based on the target description, and save the sample image and the accident Q&A information accordingly.

[0360] In some embodiments, the edge device can determine the target description of the sample image based on the overall description of the sample image, the sample image of the detection box corresponding to each region marked in the sample image, and the region description corresponding to each region.

[0361] In some embodiments, the edge device can determine the target description corresponding to the sample image through a language big model. Specifically, the edge device inputs the overall description, the region description corresponding to each region, and prompt words used by the big model to generate a detailed description based on the overall description and the region description into the language big model. The language big model comprehensively considers the overall description and the region description corresponding to each region to generate and output the target description for the sample image.

[0362] In some embodiments, after the edge device determines the target description of the sample image, the edge device displays the target description, and the technician determines the accident Q&A information based on the target description in the form of a thought chain, and saves the sample image and the accident Q&A information accordingly.

[0363] Based on Figure 23 above, the accident Q&A information obtained by the end-side device can be represented as follows:

[0364] 1. Is there any damage to the vehicle?

[0365] A: The silver-white sedan and the gray van in the picture are quite close together, but no obvious damage is observed.

[0366] 2. Has anyone fallen to the ground?

[0367] A: No one was seen lying on the ground.

[0368] 3. Did the driver or passengers get off the vehicle?

[0369] A: The picture shows two people looking around the vehicle, seemingly inspecting the extent of the damage.

[0370] 4. Analyze whether there is a traffic accident in the image based on the description.

[0371] A: Because two vehicles were close together on a snow-covered road, and people were checking around the vehicles, while other vehicles on the road were trying to avoid them, it is believed that a traffic accident has occurred.

[0372] In addition, for datasets containing labeled sample images, some sample images and their corresponding target descriptions are checked by manual verification or random sampling to determine the accuracy of the verification. If the accuracy meets the preset requirements, the dataset is used for fine-tuning training of a multimodal large model. If the accuracy does not meet the preset requirements, the sample images are re-labeled.

[0373] Based on this, the edge device can also check some sample images and their corresponding descriptions through manual verification or random inspection. If the target description of the random inspection is inconsistent with the content in the corresponding sample image, the sample image will be re-labeled.

[0374] In some embodiments, multiple images can be annotated from multiple levels based on multi-scale feature fusion, thought chain, and instruction annotation to generate descriptions with consistent expression styles. The thought chain is introduced into the data annotation, and through step-by-step prompts and answers, the multimodal large model can understand human intentions.

[0375] To improve the accuracy of dataset annotation, before determining the target description of the sample image based on the overall description and the region description corresponding to each region, the sample image annotated with the detection box corresponding to each region and the region description corresponding to each region can be input into the image language pre-training model to obtain the similarity score between each region and the corresponding region description output by the image language pre-training model; regions with similarity scores lower than a preset threshold and their corresponding region descriptions are deleted.

[0376] In some embodiments, the edge device can also use an image language pre-trained model to filter the region descriptions corresponding to each determined region and remove region descriptions with low matching degree.

[0377] Specifically, the edge device inputs the sample image labeled with the detection box corresponding to each region and the region description corresponding to each region into the image language pre-training model, obtains the similarity score between each region and the corresponding region description output by the image language pre-training model, and deletes regions and their corresponding region descriptions with similarity scores lower than a preset threshold.

[0378] Based on Figure 23, the image language pre-trained model determines the similarity score of the region description "the car is white" to be 0.95, the similarity score of the region description "a man is walking on the road" to be 0.7, the similarity score of the region description "a silver car is parked on the road" to be 0.8, and the similarity score of the region description "a person is standing on the road" to be 0.4. The preset threshold stored in the edge device is 0.6, so the edge device deletes the region description "a person is standing on the road".

[0379] To improve the accuracy of dataset annotation, when determining the target description of the sample image based on the overall description and the region description corresponding to each region, the overall description and the region description corresponding to each region can be input into the language big model to obtain the target description of the sample image output by the language big model.

[0380] In some embodiments, the edge device can determine the target description corresponding to the sample image through a language big model. Specifically, the edge device inputs the overall description, the region description corresponding to each region, and prompt words used by the big model to generate a detailed description based on the overall description and the region description into the language big model. The language big model comprehensively considers the overall description and the region description corresponding to each region to generate and output the target description for the sample image.

[0381] To improve the accuracy of dataset annotation, the image language pre-training model in this embodiment of the disclosure can be the BLIP2 model, the region description pre-training model can be the GRIP model, and the language large model can be the GPT-3.5 model.

[0382] Figure 24 is a schematic diagram of the dataset construction process provided in the embodiments of this disclosure. As shown in Figure 24, it includes the following steps:

[0383] S2401, acquire image data;

[0384] S2402, perform data cleaning on the image data;

[0385] S2403 uses the image language pre-trained model BLIP2 to perform a global image description of the entire image, thus obtaining an overall description;

[0386] For a traffic accident image, the BLIP2 image language pre-trained model is used to perform a global image description of the entire image, resulting in an overall description. For example: Many vehicles are driving on city roads.

[0387] S2404, the region description pre-trained model GRIP detects each region in the graph and describes each region;

[0388] The GRIP pre-trained model detects each region in the graph and describes each region to obtain a region description. For example, the region description includes: "The car is white", "A man is walking on the road", "A silver car is parked on the road", and "A person is standing on the road", with the region box positions as: [806,599,961,796], [759,644,828,848], [872,762,1242,1032] and [1089,867,1182,1121].

[0389] S2405, calculate the matching degree between the global image description of BLIP2 and the region description of GRIP to obtain the similarity score;

[0390] S2406, Determine whether the similarity score is higher than the score threshold;

[0391] S2407, If the score is higher than the threshold, the region description of the image is retained;

[0392] S2408, if the score is not higher than the threshold, remove the detection box and region description of the image;

[0393] The matching degree of the global image description of BLIP2 and the region description of GRIP is calculated to obtain the similarity score. By manually setting the score threshold, the detection boxes and region descriptions with low confidence are removed, and high-quality image descriptions are retained.

[0394] For example, the preserved areas are described as "the car is white" and "a man is walking on the road".

[0395] S2409, GPT3.5 instruction data construction;

[0396] S2410 utilizes GPT-3.5 to comprehensively consider the overall description, the filtered detection box image, and the corresponding description, generating a detailed image description (target description) and a thought chain image question-and-answer (accident question-and-answer information) for the entire image.

[0397] For example, the target description is "a two-way urban road covered in snow, in which two vehicles have been involved in a rear-end collision on one of the lanes, and someone is observing around the vehicles involved in the accident."

[0398] S2411, Construct the instruction dataset;

[0399] After generating a detailed description of the entire image and a thought-chain question-and-answer format, it indicates that the GPT-3.5 model has understood the meaning of the scene in the image. At this point, an instruction dataset can be constructed in the form of a thought chain. The instruction dataset includes a detailed description of the image and accident question-and-answer information, such as the surrounding environment, vehicle positions, the condition between vehicles, vehicle damage, road congestion, driver conditions, and everything you see. Based on this information, answer the following questions:

[0400] 1. Is there any damage to the vehicle?

[0401] A: The silver-white sedan and the gray van in the picture are quite close together, but no obvious damage is observed.

[0402] 2. Has anyone fallen to the ground?

[0403] A: No one was seen lying on the ground.

[0404] 3. Did the driver or passengers get off the vehicle?

[0405] A: The picture shows two people looking around the vehicle, seemingly inspecting the extent of the damage.

[0406] 4: Based on the description, analyze whether there is a traffic accident in the image?

[0407] A: Because two vehicles were close together on a snow-covered road, and people were checking around the vehicles, while other vehicles on the road were trying to avoid them, it is believed that a traffic accident has occurred.

[0408] S2412, determine whether the verification is qualified;

[0409] S2413, if qualified, generate instruction dataset;

[0410] S2414 If it is not qualified, the instruction dataset will be reconstructed.

[0411] For the labeled dataset, some sample images and their corresponding target descriptions in the dataset are checked by manual verification or sampling. If the labeled target description matches the content in the sample image, the sample image is added to the generated instruction dataset for subsequent multimodal large model fine-tuning training. If the requirements are not met, the process of building the instruction dataset in the form of a thought chain is repeated, that is, the instruction dataset is reconstructed.

[0412] Based on the above embodiments, Figure 25 is a schematic flowchart of traffic accident detection using a size model decision-making method provided in this disclosure embodiment. As shown in Figure 25, it includes the following steps:

[0413] S2501 preprocesses the input video stream to obtain icon images and other images;

[0414] S2502, input the target image and other images into the small model for traffic accident detection (i.e., the Yolov5s target detection model shown in the figure);

[0415] The process involves acquiring a video stream, decoding it, and performing frame extraction according to a predefined algorithm to identify the target image and other images. These images are then input into a small traffic accident detection model, where Yolov5s is used for target detection.

[0416] S2503 performs DeepSort target tracking and determines the confidence level of the target in the target image that has been involved in a traffic accident based on the criteria for judging traffic accidents.

[0417] S2504, If the target confidence level is higher than the maximum value of the confidence level range, then it is determined that there is a traffic accident in the target image, and a traffic accident alarm is triggered.

[0418] S2505: If the target confidence level is lower than the minimum value of the confidence level range, it is determined that there is no accident in the target image and no accident alarm is triggered.

[0419] S2506, If the target confidence level falls within the confidence level range, then input the target image into the fine-tuned multimodal large model;

[0420] At this point, if it's uncertain whether the target image contains traffic data, the target image can be input into a finely tuned multimodal large model via an edge device. Pre-defined prompts, such as "Is there a traffic accident in the image?", are used to answer with "yes" or "no." The large model infers based on the image and prompts, ultimately providing an answer, such as: "Yes, there is a traffic accident in the image," or "No, there is no traffic accident in the image."

[0421] S2507, determine whether an accident has occurred based on the detection results of the multimodal large model;

[0422] If an accident occurs, proceed to step S2504 to issue a traffic accident alarm; otherwise, proceed to step S2505 to determine that there is no accident in the target image and continue traffic accident monitoring. The model structure is optimized using a multi-instruction dataset to obtain a fine-tuned multimodal large model.

[0423] The model deployment method provided in this disclosure can also use a small model to identify key regions of the target image to be identified, and determine the coordinate information corresponding to the key regions; based on the coordinate information, determine the target sub-image corresponding to the key regions in the target image; and based on the multimodal large model and the target sub-image, determine the recognition result corresponding to the target sub-image. In the above process, the "small model" can be the first model deployed on the edge device mentioned above, and the "multimodal large model" can be the second model deployed on the edge device. Through the above process, key regions can be detected first using a small model to determine the target sub-image corresponding to the key regions, and then recognition can be performed based on the multimodal large model to obtain the recognition result corresponding to the target sub-image. This reduces the resolution of the input image, removes other irrelevant regions, and only inputs the relevant parts to the multimodal large model, thus improving the recognition effect of the multimodal large model.

[0424] For example, in certain business scenarios, it may be necessary to identify a scene within a region or simply a key target. However, the region or target in the image is not fixed and changes over time, making it impossible to pre-define the location information of the region or scene. Current technologies can only assess the entire image. In such complex scenarios, small models perform poorly, and relying solely on them cannot achieve satisfactory results. Current multimodal large models offer good overall image understanding and accurate descriptions of the entire image, but their accuracy in describing specific regions or targets is lower. For instance, in the transportation field, in snow accumulation recognition scenarios, it is only necessary to determine whether there is snow on the road surface, without considering snow accumulation on the roadside.

[0425] Figure 26 is a schematic diagram of a snow cover recognition scene provided in an embodiment of this disclosure. Inputting Figure 26 into a multimodal large model yields the following output:

[0426] Prompt: Please describe the scene shown in the picture.

[0427] Response: The image shown is taken by a surveillance camera on a highway. In this image, a black highway can be seen, with only a white car driving on it. There are green plants on both sides of the highway, covered with snow. The upper left corner of the image shows the time the image was taken and the camera's number.

[0428] Prompt: Is there snow in the picture?

[0429] Response: Yes, as you can see in the picture, there is snow on the road.

[0430] Prompt: Is there snow in the middle of the road?

[0431] Response: Yes, there is snow on the road. As you can see from the picture, the road surface is covered with a layer of snow, especially on the sides of the road where the snow is thicker.

[0432] As shown in Figure 26, there is snow on both sides of the road in the image, but no snow on the road itself. At this time, it does not affect traffic conditions, and drivers can drive normally. However, the understanding obtained by the multimodal large model is that there is snow in the image, resulting in an incorrect output, which will affect business applications.

[0433] In the railway sector, when dealing with foreign objects, it is only necessary to determine whether there are foreign objects (cat or dog carcasses, large plastic sheets, sheet metal, etc.) in and around the tracks; the presence of foreign objects in other areas is not considered.

[0434] Figure 27 is a schematic diagram of a railway foreign object identification scenario provided by an embodiment of this disclosure. As shown in Figure 27, if the answer is based solely on the description of the entire figure, it may lead to incorrect conclusions and affect business applications.

[0435] Figure 28 is a schematic diagram of a sleeping-on-duty scenario provided by an embodiment of this disclosure. In this scenario, only the staff member needs to be assessed; other areas are not considered. As shown in Figure 28, answers based solely on the description of the entire figure may lead to incorrect conclusions and affect the evaluation of the staff member.

[0436] Another current problem is that image resolution affects the recognition performance of multimodal large model. When the input image resolution is large, the recognition performance of multimodal large model is poor.

[0437] Based on this, in order to improve the image recognition effect, the model deployment method provided in this disclosure can also use a small model to identify key regions of the target image to be recognized, and determine the coordinate information corresponding to the key regions; based on the coordinate information, determine the target sub-image corresponding to the key regions in the target image; and based on the multimodal large model and the target sub-image, determine the recognition result corresponding to the target sub-image, so as to improve the image recognition effect. Specifically, as shown in Figure 29, it includes the following steps:

[0438] S2901: A small model is used to identify key regions of the target image to be identified and determine the coordinate information corresponding to the key regions.

[0439] Because large multimodal models in related technologies cannot effectively identify regions in images, a smaller model is used to extract key regions from the target image. Only the target sub-image corresponding to the key regions is input into the large multimodal model, reducing the resolution of the input image and improving the recognition performance of the large multimodal model. For example, in a sleeping worker scenario, only the region containing the worker is input, which significantly reduces the image resolution compared to the original target image, further improving the recognition performance of the large multimodal model.

[0440] In some embodiments, the edge device inputs the target image to be identified into the small model, the small model identifies the key regions of the target image, determines and outputs the coordinate information corresponding to the key regions in the target image.

[0441] In some embodiments, the small model is a smaller-scale machine learning or deep learning model, which has advantages such as being lightweight, efficient, and easy to deploy. The small model in this disclosure can be an object detection model, an image segmentation model, or other model capable of identifying key regions in an image.

[0442] S2902: Based on the coordinate information, determine the target sub-image corresponding to the key region in the target image.

[0443] In some embodiments, after the edge device obtains the coordinate information corresponding to the key region output by the small model, the edge device determines the target sub-image corresponding to the key region in the target image based on the coordinate information.

[0444] In some embodiments, the coordinate information output by the small model is {(x i ,y i The coordinate information contains multiple coordinates, each corresponding to a vertex of the key region. The edge device can determine multiple points in the target image based on the received coordinate information, and determine the area enclosed by these multiple points as the target sub-image corresponding to the key region.

[0445] If the small model is an object detection model, the key region determined by this small model is a rectangle, and the coordinate information output by the small model contains two coordinates. The edge device identifies these two coordinates as the two vertices of any diagonal of the rectangle, and determines the coordinates of the other two vertices of the rectangle based on these two coordinates. Based on the four coordinates, the edge device identifies four points in the target image, and connects these four points sequentially to define the area enclosed as the target sub-image corresponding to the key region.

[0446] If the small model is an image segmentation model, then the key region determined by the small model is a polygon, and the coordinate information output by the small model contains three or more coordinates. Based on these three or more coordinates, the edge device determines three or more points in the target image, and connects these three or more points sequentially to form the area that is defined as the target sub-image corresponding to the key region.

[0447] S2903: Determine the recognition result corresponding to the target sub-image based on the multimodal large model and the target sub-image.

[0448] In some embodiments, after the edge device determines the target sub-image corresponding to the key region in the target image, the edge device determines the recognition result corresponding to the target sub-image based on the multimodal large model and the target sub-image.

[0449] Specifically, the target sub-image and prompt words are input into the multimodal large model at the edge device. The multimodal large model then identifies the target sub-image based on the prompt words. These prompt words can be used to provide an overall description of the image output by the multimodal large model, or they can be questions that the multimodal large model needs to answer.

[0450] Example 1: Figure 26 is processed using the model deployment method provided in this embodiment of the disclosure. The output of the multimodal large model is:

[0451] Prompt: Please describe the scene shown in the picture;

[0452] The response is an aerial photograph taken on a winding road, with green vegetation on both sides and a car driving in the middle. The entire image is framed by a clear blue sky, and a white dotted line runs down the center of the road.

[0453] Prompt: Is there snow in the picture?

[0454] Response: There is no snow in the picture, and the road is very clean.

[0455] Prompt: Is there snow in the middle of the road?

[0456] Response: According to the description, there is no snow in the middle of the road in the picture, and the road surface is very clean.

[0457] Example 2: Figure 27 is processed using the model deployment method provided in this embodiment of the disclosure. The output of the multimodal large model is:

[0458] Prompt: Are there any foreign objects on the railway tracks in the picture? Foreign objects include, but are not limited to, kittens, puppies, plastic sheets, and sheet metal.

[0459] Response: Yes, there is a kitten on the railway tracks in the picture.

[0460] Example 3: Figure 28 is processed using the model deployment method provided in this embodiment of the disclosure. The output of the multimodal large model is:

[0461] Prompt: Is the staff member in the picture sleeping?

[0462] Response: Yes, the staff member in the picture is sleeping.

[0463] In Examples 1-3 above, the content after "prompt" is the prompt word, and the content after "response" is the output of the multimodal large model.

[0464] In this embodiment, a combination of large and small models is used. First, the key region is detected by the small model to determine the target sub-image corresponding to the key region. Then, the recognition is performed based on the large multimodal model to obtain the recognition result corresponding to the target sub-image. This reduces the resolution of the input image, removes other irrelevant regions, and only inputs the part of interest to the large multimodal model, thereby improving the recognition effect of the large multimodal model.

[0465] To improve image recognition performance, when determining the target sub-image corresponding to the key region in the target image based on coordinate information, the target region corresponding to the coordinate information in the target image can be determined based on the coordinate information; the target region can be expanded according to the pre-configured expansion rules; and the target image can be segmented based on the expanded target region to determine the target sub-image.

[0466] In this embodiment of the disclosure, after the edge device obtains the coordinate information output by the small model, the edge device determines the target region corresponding to the coordinate information in the target image based on the coordinate information. The process by which the edge device determines the target region corresponding to the coordinate information in the target image based on the coordinate information is the same as the process in the above embodiment, and will not be described again here.

[0467] In order to better identify the target area, in this embodiment of the disclosure, the end device is also pre-configured with an expansion rule, which can expand the target area and avoid poor identification effect of the target area that is too small.

[0468] Specifically, the edge device acquires pre-configured expansion rules and expands the target region in the target image according to these rules to obtain the expanded target region. The edge device then segments the target region and determines the segmented image as the target sub-image corresponding to the key region.

[0469] To improve image recognition performance, based on the above embodiments, after determining the target region corresponding to the coordinate information in the target image according to the coordinate information, masking can be performed on other regions in the target image except for the target region before expanding the target region according to the pre-configured expansion rules.

[0470] Since the expanded target region will encompass some other regions, in order to reduce the input image resolution, remove irrelevant regions, and only input the region of interest into the large model to improve the recognition performance of the large model, the edge device, after determining the target region corresponding to the coordinate information in the target image, will also perform masking processing on the other regions in the target image besides the target region. Specifically, the edge device sets the pixel values ​​of the pixels in the other regions of the target image to preset values.

[0471] In some embodiments, by masking regions other than the target region in the target image, the multimodal large model will not be interfered with by targets in other regions. Identifying the presence or absence of targets only in key regions will further improve the recognition performance of the multimodal large model.

[0472] Based on the disclosure in Figure 26, Figure 30 is a schematic diagram of the snow recognition scene after masking provided in the embodiment of this disclosure. As shown in Figure 30, the end device performs masking processing on the areas in the image other than the road.

[0473] Based on the disclosure in Figure 2, Figure 31 is a schematic diagram of railway foreign object identification after masking provided in the embodiment of this disclosure. As shown in Figure 31, the end-side device performs masking processing on other areas in the image except for the railway track.

[0474] To improve image recognition performance, based on the above embodiments, when the small model is an image segmentation model, the target region needs to be expanded according to pre-configured expansion rules. Specifically, the minimum bounding rectangle corresponding to the target region can be determined in the target image; the region corresponding to the minimum bounding rectangle is then determined as the expanded target region.

[0475] In this embodiment, there are many types of small models. If the small model used by the edge device is an image segmentation model employing an image segmentation algorithm, then the target region determined by the image segmentation model is a polygon, and the coordinate information output by the image segmentation model is the coordinates of each vertex of the polygon. After receiving the coordinate information and determining the target region in the target image based on the coordinate information, the edge device determines the minimum bounding rectangle corresponding to the target region and defines the region corresponding to the minimum bounding rectangle as the expanded target region.

[0476] Specifically, in this embodiment of the disclosure, if the small model is an image segmentation model using an image segmentation algorithm, the output is coordinate information as {(x... i ,y i For each coordinate region (i = 1, ..., N), the end-side device determines the target region corresponding to this coordinate information and determines the minimum bounding rectangle of the target region, i.e., the coordinates of the upper left corner of the minimum bounding rectangle are (min{x... i},min{y i}), i=1,…,N, the coordinates of the lower right corner are (max{x i},max{y i}), i=1,…,N. It should be noted that sometimes the target area is large, and the region corresponding to the minimum bounding rectangle of the target area is the target image.

[0477] To improve image recognition performance, based on the above embodiments, when the small model is a target detection model, the target region needs to be expanded according to the pre-configured expansion rules. Specifically, the length and width of the target region in the target image can be expanded by a preset multiple; the expanded region is then determined as the expanded target region.

[0478] As mentioned earlier, there are many types of small models used in practical applications. If the small model used by the edge device is a target detection model employing an object detection algorithm, then the target region determined by this model is a rectangle. The coordinate information output by the image segmentation model consists of the coordinates of the two vertices corresponding to any diagonal, typically the top-left and bottom-right vertices. After receiving the coordinate information and determining the target region in the target image based on it, the edge device enlarges the length and width of the target region by a preset multiple, and defines the enlarged region as the expanded target region. This preset multiple can be an integer greater than 1.

[0479] Specifically, if the small model is a target detection model using an object detection algorithm, the output coordinate information is the coordinates of the two vertices corresponding to any diagonal line: {(x1,y1),(x2,y2)}. The target region is determined based on this coordinate information. Taking the center point of this target region as the origin, the length and width are increased by a factor of α (α≥1), resulting in the expanded region {(x′1,y′1),(x′2,y′2)}. Users can configure the preset factor α to determine whether to expand the target region according to the actual scenario requirements.

[0480] The end-side device can determine the coordinate information of the enlarged area using the following formula:

[0481] Where {(x1,y1),(x2,y2)} are the coordinates of the top left and bottom right vertices of the original target region, {(x′1,y′1),(x′2,y′2)} are the coordinates of the top left and bottom right vertices of the expanded region, and α is a preset multiple.

[0482] To improve image recognition performance, based on the above embodiments, before determining the recognition result corresponding to the target sub-image according to the multimodal large model and the target sub-image, a preset number of sample images can be obtained; for each sample image, the image language pre-trained model BLIP2 and the chat generative pre-trained transformer application programming interface (ChatGPT API) are used to process the sample image to determine the sample description text corresponding to the sample image; the multimodal large model is fine-tuned according to each sample image and the corresponding sample description text.

[0483] Because small models are prone to misidentification, the key regions identified by them may not be the most ideal regions and may include other regions. Therefore, this embodiment further fine-tunes the large multimodal model to improve its region recognition capability and reduce the misidentification rate.

[0484] To better fine-tune the multimodal large model, this embodiment of the disclosure constructs a sample dataset for fine-tuning the multimodal large model through a multi-level data construction method. Based on the constructed sample dataset, the multimodal large model is fine-tuned to further optimize its ability to recognize regional scenes. The sample data construction method includes data diversity and multi-level fusion description.

[0485] Specifically, the edge device acquires a preset number of sample images; for each sample image, an image-language pre-trained model and a chat generation pre-trained transformer are used to process the sample image to determine the corresponding sample description text. The edge device then fine-tunes the multimodal large model based on each sample image and its corresponding sample description text.

[0486] In this embodiment of the disclosure, when the end-side device performs fine-tuning on the multimodal large model, it may use fine-tuning methods such as Adapter-Tuning, Prefix-Tuning, Prompt-Tuning (P-Tuning), P-Tuning v2, LoRA, etc., without any limitation.

[0487] To improve image recognition performance, when fine-tuning the multimodal large model based on each sample image and its corresponding sample description text, the multimodal large model can be trained using each sample image and its corresponding sample description text, and the loss value can be calculated. The parameter values ​​of the pre-configured model parameters used for fine-tuning can be adjusted according to the loss value to obtain the target model parameters. The target model parameters are then merged with the original model parameters of the multimodal large model to obtain the fine-tuned multimodal large model.

[0488] In some embodiments, the edge device can employ the common LoRA fine-tuning method for fine-tuning the multimodal large model. Specifically, the edge device trains the multimodal large model using each sample image and its corresponding sample description text, and calculates the loss value; it then adjusts the pre-configured model parameters for fine-tuning based on the loss value to obtain the target model parameters; finally, it merges the target model parameters with the original model parameters of the multimodal large model to obtain the fine-tuned multimodal large model.

[0489] To improve image recognition performance, based on the above embodiments, an image language pre-trained model and the ChatGPT API are used to process the sample image. When determining the sample description text corresponding to the sample image, the image language pre-trained model can be used to recognize the sample image and determine the initial global description text. An object detection model is used to perform object detection on the sample image, and based on the first coordinate information output by the object detection model, a first sample sub-image is determined. The image language pre-trained model is then used to recognize the first sample sub-image and determine the first sub-description text corresponding to the first sample sub-image. An image segmentation model is used to segment the sample image, and based on the second coordinate information output by the image segmentation model, a second sample sub-image is determined. The image language pre-trained model is then used to recognize the second sample sub-image and determine the second sub-description text corresponding to the second sample sub-image. Finally, the sample image, the initial global description text, the first sample sub-image, the first sub-description text, the second sample sub-image, and the second sub-description text are input into the ChatGPT API to determine the sample description text corresponding to the sample image.

[0490] In some embodiments, when constructing a sample dataset, the edge device first generates an initial global description text for the sample image, then uses a small model to identify key regions and generates sub-description text for the sample sub-images corresponding to the key regions, and then determines the sample description text corresponding to the sample image based on the initial global description text and the sub-description text.

[0491] Specifically, the edge device inputs the sample image into an image-language pre-trained model for recognition, and obtains the initial global description text corresponding to the sample image output by the image-language pre-trained model. The edge device then inputs the sample image into an object detection model, obtains the first coordinate information output by the object detection model, and determines a first sample sub-image based on the first coordinate information. Finally, the edge device inputs the sample image into an image segmentation model, obtains the second coordinate information output by the image segmentation model, and determines a second sample sub-image based on the second coordinate information.

[0492] The edge device inputs the first sample sub-image and the second sample sub-image into the image language pre-training model to determine the first sub-description text corresponding to the first sample sub-image and the second sub-description text corresponding to the second sample sub-image.

[0493] The edge device inputs the sample image, initial global description text, first sample sub-image, first sub-description text, second sample sub-image, and second sub-description text into the ChatGPT API to obtain the sample description text corresponding to the sample image output by the ChatGPT API.

[0494] Furthermore, in this embodiment, after the end device obtains the sample description text output by the ChatGPT API, it can further modify and supplement the sample description based on the sample image, the initial global description text, the first sample sub-image, the first sub-description text, the second sample sub-image, and the second sub-description text. This modification and supplementation can be performed by technical personnel or through preset rules, such as pre-setting necessary information; no limitations are imposed here.

[0495] One possible implementation is illustrated using a snow-covered road scenario as an example:

[0496] When constructing the sample dataset corresponding to the road snow scene, the image types of the selected sample images include the following: images with a large amount of snow in the entire image area, images with snow only on both sides of the road, and images with a small amount of snow on the road.

[0497] Figure 32 is a schematic diagram of the construction process of the sample dataset corresponding to the road snow scene provided in the embodiments of this disclosure. As shown in Figure 32, when determining the sample description text corresponding to the sample image, the following steps are performed:

[0498] S3201, Obtain sample image;

[0499] S3202, Generates initial global description text for the entire sample image based on the image language pre-trained model; the initial global description text is "The earth is covered with white snow, and a snow-white car is driving on a black road";

[0500] S3203, use the trained target detection model to generate the key road region (first subsample image), and describe the scene in the road region in detail based on the image language pre-trained model. The detailed description of the key road region is "a black road with only one white car driving on it, and the sides of the road are covered with snow".

[0501] S3204 uses a trained image segmentation model to generate a road segmentation region (second subsample image), and then describes the road segmentation region in detail based on an image language pre-trained model. The road segmentation region is described as "a black road, which is very clean, and a white car is driving on it".

[0502] S3205, in the final stage, the initial global description text, key road regions and corresponding descriptions, and road segmentation regions and corresponding descriptions are input into the ChatGPT API for summarization, and the overall description (sample description text) is output. The overall description is "In this picture, you can see a black road. The road is very clean. There is only a white car driving on it. There are large areas of green plants on both sides of the road, and the ground is covered with snow."

[0503] To improve image recognition performance, based on the above embodiments, the process for determining the small model in this disclosure may include: receiving an image recognition request carrying an identifier of a target image and a small model; and determining the small model based on the identifier. It should be understood that there are many types of small models in practical applications, and for the input image, the appropriate type of small model can be selected according to specific business requirements; this disclosure does not limit this selection.

[0504] If the key region of interest is an irregular area, then a traditional image segmentation model is used for region segmentation, and the segmented key region is output. For example, in a snow-covered road scene, the image segmentation model is used to segment the road region in the image and output the polygon coordinates corresponding to the road region; in a railway object scene, the image segmentation model is used to segment the railway track region in the image and output the polygon coordinates corresponding to the railway track region.

[0505] If the key region of interest is a regular region, then an object detection model is used to detect objects, and the detected rectangular regions are output. For example, in detecting sleeping on duty, the object detection model is first used to detect workers, and the coordinates of the detected worker's rectangular region are output.

[0506] In some embodiments, a technician may select a small model and send the identifier of the small model to the edge device via an image recognition request. The edge device determines the small model to be used for recognition based on the identifier of the small model carried in the received image recognition request.

[0507] Specifically, the edge device receives an image recognition request carrying an identifier for the target image and a small model; the edge device determines the small model to be used for recognition based on the identifier of the small model.

[0508] It should be noted that the model deployment method provided in this disclosure is applicable to all specific area scene recognition, and the overall concept is universal. Users can flexibly select each part according to the specific scenario requirements to ultimately achieve the goal better, and this disclosure does not limit this.

[0509] Figure 33 is a schematic diagram of a scene recognition process provided in some embodiments of this disclosure. As shown in Figure 33, the process may include the following steps:

[0510] S3301, acquire the image to be recognized and input the image into the small model;

[0511] S3302 processes the input image (detection or segmentation) using a small model (selecting either an object detection model or an image segmentation model based on the actual business scenario) to obtain the coordinates of the key regions of the image;

[0512] S3303, performs image processing on the key region based on the key region coordinates, that is, segments the key region from the image to obtain the processed sub-image;

[0513] S3304, the processed sub-image is sent into the multimodal large model;

[0514] S3305, a multimodal large model, recognizes sub-images of the input model and outputs the recognition results.

[0515] S3306, Construct a dataset and fine-tune the model. That is, in addition to the image recognition process in steps S3301 to S3304, a multimodal large model can be fine-tuned by constructing a dataset.

[0516] Based on the above embodiments, Figure 34 is a schematic diagram of an application of multimodal large model recognition provided by the present disclosure. As shown in Figure 34, the following steps may be included:

[0517] S3401, Obtain the target image containing the scene with foreign objects on the railway tracks;

[0518] S3402, a small model is used to identify the target image and mark the target region in the target image;

[0519] S3403, Mask the regions in the target image other than the target region to obtain the masked target image;

[0520] S3404, expand the target region according to the pre-configured expansion rules, and cut the target image according to the expanded target region to obtain the target sub-image;

[0521] S3405, Input the target sub-image into the multimodal large model, and obtain the recognition result output by the multimodal large model as "In this picture, you can see a railway track, and there is a white kitten on the railway track".

[0522] Based on the same inventive concept, this disclosure provides an end-side device, as specifically shown in FIG35, including:

[0523] Image acquisition device 111 is used to acquire video streams;

[0524] Display 112 is used to display images and / or alarms;

[0525] Memory 113 is used to store computer programs or instructions;

[0526] Processor 114 is configured to execute computer programs or instructions in memory 113 such that any of the methods described in the above embodiments is performed.

[0527] The processor 114 may include one or more central processing units (CPUs) or digital processing units, etc., for executing computer programs or instructions in the memory 113, such that any of the methods described in the above embodiments are executed.

[0528] It should be noted that the specific connection medium between the memory 113 and the processor 114 in this embodiment is not limited. In Figure 35, the memory 113 and the processor 114 are connected via a bus 115. The connection methods between other components are only illustrative and are not intended to be limiting. The bus 115 can be divided into an address bus, a data bus, a control bus, etc. For ease of illustration, only one thick line is used in Figure 35, but this does not mean that there is only one bus or one type of bus.

[0529] Based on the same inventive concept, this disclosure provides a computer-readable storage medium that, when executed by a processor, enables the processor to perform any of the methods described in the above embodiments. Since the principle by which the computer-readable storage medium solves the problem is similar to that of a model deployment method, the implementation of the computer-readable storage medium can be found in the implementation of the method, and repeated details will not be elaborated further.

[0530] Based on the same inventive concept, this disclosure provides a computer program product, which includes computer program code that, when run on a computer, causes the computer to execute any of the methods described in the above embodiments.

[0531] Those skilled in the art will understand that embodiments of this disclosure can be provided as methods, systems, or computer program products. Therefore, this disclosure can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, this disclosure can take the form of a computer program product embodied on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0532] This disclosure is described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to this disclosure. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, create means for implementing the functions specified in one or more blocks of the flowchart illustrations and / or one or more blocks of the block diagrams.

[0533] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means that implement the functions specified in one or more processes in a flowchart and / or one or more blocks in a block diagram.

[0534] These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process, such that the instructions, which execute on the computer or other programmable apparatus, provide steps for implementing the functions specified in one or more processes in the flowchart and / or one or more blocks in the block diagram.

[0535] Obviously, those skilled in the art can make various modifications and variations to this disclosure without departing from its spirit and scope. Therefore, if such modifications and variations fall within the scope of the claims of this disclosure and their equivalents, this disclosure is also intended to include such modifications and variations.

Claims

1. A model deployment method, the method comprising: For the video frame images input to the first model, perform the following operations to construct the training set: Based on the first model, target detection is performed on the video frame image to obtain a first target detection result of the video frame image and a first confidence level of the first target detection result; the first confidence level is used to characterize the accuracy of the first target detection result. The first model is used for target detection under the first business requirement; If the first confidence level is greater than or equal to the first confidence threshold, then the video frame image and the first target detection result are recorded as samples in the training set; If the first confidence level is less than the first confidence level threshold, then target detection is performed on the video frame image based on the second model to obtain the second target detection result of the video frame image and the second confidence level of the second target detection result, and the video frame image and the second target detection result are recorded as samples in the training set; the second confidence level is used to characterize the accuracy of the second target detection result; The second model is used for target detection under various business requirements, including the first business requirement; The first model is trained based on the training set, and the trained first model is used to replace the current first model for subsequent object detection.

2. The method according to claim 1, after obtaining the first target detection result and the first confidence level of the first target detection result of the video frame image, the method further includes: If the first confidence level is greater than or equal to the first confidence threshold, a first alarm is issued to the user, wherein the first alarm is used to prompt the user to confirm whether an event related to the first target detection result has occurred; After receiving the user's confirmation instruction for the first alarm trigger, manually labeled the first target detection results in the training set; Upon receiving a false alarm instruction from the user in response to the first alarm, the system updates the first target detection result in the training set based on the false alarm instruction and sets a manual label for the updated first target detection result.

3. The method according to claim 1, after obtaining the second target detection result and the second confidence level of the second target detection result of the video frame image, the method further includes: If the second confidence level is greater than or equal to the second confidence level threshold, a second alarm is issued to the user, wherein the second alarm is used to prompt the user to confirm whether an event related to the second target detection result has occurred; After receiving the user's confirmation instruction for the second alarm trigger, manually labeled the second target detection results in the training set; Upon receiving a false alarm instruction from the user for the second alarm, the system updates the second target detection result in the training set based on the false alarm instruction and sets a manual label for the updated second target detection result.

4. The method according to claim 2 or 3, wherein training the first model based on the training set comprises: Based on the preset sample balancing rules, each sample is obtained from the training set. The first model is trained based on the samples mentioned above; The sample balancing rules include some or all of the following rules: The ratio of the number of samples with manual labels to the total number of samples in the model training is greater than or equal to the first ratio. The proportion of samples from different data sources is the largest. The proportion of samples from different target categories is the second largest proportion; The proportion of samples with different attribute information is the third proportion. The attribute information is obtained by the second model to identify the attributes of each sample in the training set. The attribute information includes part or all of the image illumination information and the weather information of the image acquisition environment.

5. The method according to claim 4, wherein after training the first model based on the training set, the method further comprises: Multiple samples are obtained from the training set, and the trained first model and the current first model are evaluated and compared based on the multiple samples to obtain a first parameter value. The first parameter value represents the degree of difference between the evaluation index values ​​of the trained first model and the current first model. The evaluation index values ​​include some or all of the following: accuracy, recall, mean squared error, and log loss. Before replacing the current first model with the trained first model for subsequent object detection, the method further includes: The first parameter value is determined to meet the first replacement condition; wherein the first replacement condition includes the first parameter value being greater than the first parameter threshold, or the first parameter value being greater than the first parameter threshold and the current detection time of the first model reaching the first preset time.

6. The method according to claim 2 or 3, further comprising: Select multiple target samples from the samples in the training set that have been manually labeled; The second model is fine-tuned based on the multiple target samples; Multiple samples are obtained from the training set, and the fine-tuned second model and the current second model are evaluated and compared based on the multiple samples to obtain the second parameter value. The second parameter value represents the degree of difference between the evaluation index values ​​of the fine-tuned second model and the current second model. The evaluation index values ​​include some or all of the following: accuracy, recall, mean squared error, and log loss. After the second parameter value meets the second replacement condition, the current second model is replaced with the fine-tuned second model for subsequent target detection. The second replacement condition includes the second parameter value being greater than the second parameter threshold, or the second parameter value being greater than the second parameter threshold and the detection time of the current second model reaching the second preset time.

7. The method according to claim 1, further comprising: If the first target detection result further includes multiple sub-detection information, then based on the confidence level of the multiple sub-detection information, target sub-detection information with a confidence level less than a sub-confidence threshold is selected from the multiple sub-detection information, and / or, based on the relationship between any two sub-detection information, target sub-detection information is selected from the multiple sub-detection information. Each sub-detection information is used to describe the target category of the corresponding target and the image region within the video frame image. The first confidence level also includes the confidence level of the multiple sub-detection information; the relationship includes the intersection-union ratio (IOU) and / or coordinate position relationship between two associated sub-detection information. Based on the selected target sub-detection information, a first prompt instruction is generated, wherein the first prompt instruction is used to guide the second model to perform target detection on the image region where the target described by the target sub-detection information is located within the video frame image; The video frame image is input into the second model, triggering the second model to perform target detection according to the first prompt instruction, and obtaining a third target detection result and a third confidence level of the third target detection result, wherein the third confidence level is used to characterize the accuracy of the third target detection result; When the third confidence level is greater than or equal to the second confidence level threshold, a third alarm is sent to the user, wherein the third alarm is used to prompt the user to confirm and / or process the event associated with the third target detection result.

8. The method according to claim 1, wherein the multiple business requirements include a second business requirement, and the method further includes: Receive a second prompt instruction, wherein the second prompt instruction is used to instruct the second model to perform target detection under the second business requirement based on each sample included in the validation set; Execute the second prompt instruction on the second model to obtain the accuracy of the second model under the second business requirement; If the accuracy of the second model is less than the model accuracy threshold, then the second model is trained until the accuracy of the second model is greater than or equal to the model accuracy threshold. Replace the current second model with the trained second model to perform target detection for the second business requirement.

9. A model deployment method, the method comprising: Acquire a multimodal large model and a target video; wherein the multimodal large model includes a fusion structure, a language large model, and at least two video encoders; The target video is input into the at least two video encoders respectively, and each initial video feature matrix output by the at least two video encoders is obtained; Each initial video feature matrix is ​​input into the fusion structure, which concatenates each initial video feature matrix to obtain a candidate video feature matrix. Based on a preset weight matrix, the candidate video feature matrix is ​​mapped to a target video feature matrix with the same number of rows and columns as the weight matrix. The target video feature matrix and instruction prompts are input into a language model to obtain the target text output by the language model.

10. The method according to claim 9, wherein mapping the candidate video feature matrix to a target video feature matrix with the same number of rows and columns as the weight matrix according to a preset weight matrix comprises: Determine the first product matrix of the transpose of the weight matrix and the candidate video feature matrix, and determine the ratio of the first product matrix to a preset value as the attention score matrix; The attention score matrix is ​​normalized to determine the probability distribution matrix; Determine the second product matrix of the probability distribution matrix and the candidate video feature matrix, and determine the second product matrix as the target video feature matrix.

11. The method according to claim 9, wherein before mapping the candidate video feature matrix to a target video feature matrix with the same number of rows and columns as the weight matrix according to a preset weight matrix, the method further comprises: Each initial video feature matrix is ​​split into a predetermined number of sub-video feature matrices; For each sub-video feature matrix, determine each product of each row vector of the sub-video feature matrix with each pre-configured weight column vector; determine each product vector of each row vector and its corresponding product, and update the sub-video feature matrix using the sum of the product vectors. Based on the updated feature matrix of each sub-video, the updated candidate video feature matrix is ​​determined.

12. The method according to claim 11, further comprising: After determining the product of each row vector of the sub-video feature matrix with each pre-configured weight column vector, a normalization operation is performed on each product.

13. The method according to claim 9, wherein the multimodal large model is trained in the following manner: Freeze the parameters in the first parameter group of the at least two encoders and the parameters in the second parameter group of the language big model, and set the parameters in the third parameter group of the fusion structure to their initial values; Obtain the first sample video and its corresponding first sample text from the dataset; The first sample video is processed using the multimodal large model to obtain the first predicted sample text output by the multimodal large model; Based on the first predicted sample text and the first sample text, the parameters in the third parameter group of the fusion result of the multimodal large model are adjusted.

14. The method according to claim 13, further comprising: Unfreeze the parameters in the second parameter group of the large language model; Obtain the second sample video and the corresponding second sample text carried in the dataset; The multimodal large model is used to process the second sample video to obtain the second predicted sample text output by the multimodal large model; Based on the second predicted sample text and the second sample text, the parameters in the third parameter group of the fusion result of the multimodal large model and the parameters in the second parameter group of the language large model are adjusted.

15. A model deployment method, the method comprising: The target image to be detected and a preset number of other images adjacent to the acquisition time of the target image are input into the accident detection mini-model, so that the accident detection mini-model determines the motion trajectory corresponding to each target, and determines and outputs the target confidence that there is a traffic accident in the target image based on the motion trajectory; wherein, the accident detection mini-model includes at least a target detection sub-model, a target tracking sub-model and an accident detection sub-model; If the target confidence level falls within a preset confidence level range, the target image is input into a multimodal large model for detecting traffic accidents, and the detection result of whether a traffic accident exists in the target image is obtained from the output of the multimodal large model.

16. The method according to claim 15, further comprising: The multimodal large model is fine-tuned based on a pre-configured dataset; The dataset is constructed in the following manner: The sample image is input into the image language pre-training model to obtain the overall description corresponding to the sample image output by the image language pre-training model; The sample image is input into the region description pre-training model to obtain the sample image with the detection box corresponding to each region and the region description corresponding to each region output by the region description pre-training model. The sample image labeled with the detection box corresponding to each region, the overall description, and the region description corresponding to each region are input into the language big model to obtain the target description of the sample image output by the language big model. The input accident question and answer information determined based on the target description is obtained, and the sample image and the accident question and answer information are saved accordingly to obtain the dataset.

17. The method according to claim 15, wherein the multimodal large model comprises a visual encoder, a position-aware visual-language adapter, and a language large model; the step of obtaining the detection result of whether a traffic accident exists in the target image output by the multimodal large model comprises: The visual encoder encodes the target image to determine the feature code corresponding to the target image; The location-aware visual language adapter compresses the feature code and determines the compressed feature code; The language big data model determines and outputs the detection result of whether a traffic accident exists in the target image based on the compressed feature encoding.

18. The method according to claim 16, wherein the visual encoder comprises a convolutional layer, a linear projection layer, and a Transformer layer; the step of encoding the target image and determining the feature encoding corresponding to the target image comprises: The convolutional layer uses dilated convolution to extract features from the target image, determines the feature map corresponding to the target image, and segments the feature map to obtain each sub-feature map; wherein, the parameters of the dilated convolution are the parameters learned during model training; The linear projection layer sorts each sub-feature map according to the segmentation order of the feature map, and performs linear encoding on each sorted sub-feature map to determine the image features corresponding to the feature map; The Transformer layer extracts features from the image and determines the feature encoding.

19. The method according to claim 16, wherein the accident detection sub-model comprises a target detection sub-model, a target tracking sub-model, and an accident detection sub-model; and the step of obtaining the target confidence score of the target image output by the accident detection sub-model indicating the existence of a traffic accident comprises: The target detection sub-model performs target detection on the target image and the other images respectively, identifies each target contained in the target image and the other images, and labels each target in the target image and the other images; wherein, each target is a vehicle or a pedestrian; The target tracking sub-model determines the motion trajectory of each target based on the labeled target image and other images; The accident detection sub-model performs accident detection based on the motion trajectory of each target, and determines and outputs the target confidence score of the target image where a traffic accident exists.

20. The method of claim 15, further comprising: If the target confidence level exceeds the maximum value of the confidence level range, it is determined that there is a traffic accident in the target image, and an accident alarm is triggered. If the target confidence level is less than the minimum value of the confidence level range, it is determined that there is no traffic accident in the target image, and no accident alarm is triggered.

21. The method according to claim 16, wherein before determining the target description of the sample image based on the overall description and the region description corresponding to each region, the method further comprises: The sample image labeled with the detection box corresponding to each region and the region description corresponding to each region are input into the image language pre-training model to obtain the similarity score between each region and the corresponding region description output by the image language pre-training model. Delete regions with similarity scores below a preset threshold and their corresponding descriptions.

22. The method according to claim 16, wherein the image language pre-trained model is a BLIP2 model, the region description pre-trained model is a GRIP model, and the language large model is a GPT-3.5 model.

23. A model deployment method, the method comprising: A small model is used to identify key regions of the target image to be identified and determine the coordinate information corresponding to the key regions. Based on the coordinate information, determine the target sub-image corresponding to the key region in the target image; Based on the multimodal large model and the target sub-image, the recognition result corresponding to the target sub-image is determined.

24. The method according to claim 23, wherein determining the target sub-image corresponding to the key region in the target image based on the coordinate information comprises: Based on the coordinate information, determine the target region corresponding to the coordinate information in the target image; The target region is expanded according to a pre-configured expansion rule, and the target image is segmented based on the expanded target region to determine the target sub-image.

25. The method according to claim 24, wherein after determining the target region corresponding to the coordinate information in the target image based on the coordinate information, the other regions in the image besides the target region are masked according to a pre-configured expansion rule.

26. The method according to claim 24 or 25, wherein expanding the target region according to a pre-configured expansion rule comprises: If the small model is an image segmentation model, then in the target image, determine the minimum bounding rectangle corresponding to the target region; The region corresponding to the smallest bounding rectangle is determined as the expanded target region.

27. The method according to claim 24 or 25, wherein expanding the target region according to a pre-configured expansion rule comprises: If the small model is a target detection model, then in the target image, the length and width of the target region are increased by a preset multiple; The expanded area is defined as the expanded target area.

28. The method according to claim 23, wherein before determining the recognition result corresponding to the target sub-image, the method further comprises: Obtain a preset number of sample images; For each sample image, an image language pre-trained model and a chat generation pre-trained transformer are used to process the sample image to determine the sample description text corresponding to the sample image. The multimodal large model is fine-tuned based on each sample image and its corresponding sample description text.

29. The method according to claim 28, wherein fine-tuning the multimodal large model based on each sample image and the corresponding sample description text comprises: The multimodal large model is trained using each sample image and its corresponding sample description text, and the loss value is calculated. The target model parameters are obtained by adjusting the pre-configured model parameters for fine-tuning based on the loss value. The target model parameters are merged with the original model parameters of the multimodal large model to obtain the fine-tuned multimodal large model.

30. The method according to claim 27, wherein processing the sample image using an image-language pre-trained model and a chat generation pre-trained transformer to determine the sample description text corresponding to the sample image includes: The image language pre-trained model is used to identify the sample image and determine the initial global description text corresponding to the sample image. The target detection model is used to detect targets in the sample image, and the first sample sub-image is determined based on the first coordinate information output by the target detection model. The image language pre-trained model is used to recognize the first sample sub-image to determine the first sub-description text corresponding to the first sample sub-image. The sample image is segmented using an image segmentation model, and a second sample sub-image is determined based on the second coordinate information output by the image segmentation model. The second sample sub-image is then recognized using the image language pre-training model to determine the second sub-description text corresponding to the second sample sub-image. The sample image, the initial global description text, the first sample sub-image, the first sub-description text, the second sample sub-image, and the second sub-description text are input into the chat generation pre-training transformer to determine the sample description text corresponding to the sample image.

31. The method according to claim 23, wherein the process of determining the small model includes: Receive an image recognition request carrying the identifiers of the target image and the small model; The small model is determined based on its identifier.

32. An end-side device, comprising: Image acquisition device, used to acquire video streams; A display, used to show images and / or alarms; Memory is used to store computer programs or instructions; A processor for executing a computer program or instructions in the memory such that the method described in any one of claims 1-31 is performed.

33. A computer-readable storage medium, wherein instructions in the storage medium, when executed by a processor, enable the processor to perform the method as described in any one of claims 1-31.

Images