Data backflow analysis method, device, equipment and storage medium

Abstract

The present application provides a method, device, equipment and storage medium for data backflow analysis, the method comprising: obtaining the backflow data to be analyzed; determining the algorithm identifier corresponding to the alarm event according to the alarm box in the alarm image, and determining the historical data set used for algorithm training according to the algorithm identifier; determining the first text prompt word of the image-text multimodal detection large model according to the label text of the historical data set; and analyzing the backflow data to be analyzed according to the first text prompt word using the image-text multimodal detection large model. This method can realize the automation of data backflow analysis.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to a data backflow analysis method, device, equipment and storage medium. Background Art

[0002] With the development of artificial intelligence technology and the popularization of monitoring equipment, using artificial intelligence technology for event detection has gradually become a popular application direction.

[0003] In the process of using artificial intelligence technology to detect events, it is necessary to implement it based on a trained detection algorithm model. However, due to the limited training data during the training of the detection algorithm model and the influence of the training data quality, the accuracy of event detection using the trained detection algorithm model cannot be guaranteed. Summary of the invention

[0004] In view of this, the present application provides a data backflow analysis method, device, equipment and storage medium.

[0005] According to a first aspect of an embodiment of the present application, a data backflow analysis method is provided, including:

[0006] Acquire the reflow data to be determined; wherein the reflow data to be determined includes an alarm image, and an alarm frame exists in the alarm image;

[0007] Determine an algorithm identifier corresponding to the alarm event based on the alarm frame in the alarm image, and determine a historical data set used for algorithm training based on the algorithm identifier;

[0008] Determining a first text prompt word of a large image-text multimodal detection model according to the label text of the historical data set;

[0009] According to the first text prompt word, the image-text multimodal detection large model is used to analyze the return data to be analyzed;

[0010] The analyzing and judging of the return flow data to be analyzed includes one or more of the following:

[0011] The coordinates of the warning frame in the warning image are adjusted, the erroneous warning frame in the warning image is deleted, and the missed targets in the return flow data to be judged are marked.

[0012] According to a second aspect of an embodiment of the present application, a data backflow analysis device is provided, including:

[0013] An acquisition unit is configured to acquire the reflow data to be determined; wherein the reflow data to be determined includes an alarm image, and an alarm frame exists in the alarm image;

[0014] A first determining unit is configured to determine an algorithm identifier corresponding to the alarm event according to the alarm frame in the alarm image, and determine a historical data set used for algorithm training according to the algorithm identifier;

[0015] A second determination unit is configured to determine a first text prompt word of the image-text multimodal detection large model according to the label text of the historical data set;

[0016] A judging unit is configured to judge the reflux data to be judged based on the first text prompt word and using the image-text multimodal detection large model;

[0017] The analyzing and judging of the return flow data to be analyzed includes one or more of the following:

[0018] The coordinates of the warning frame in the warning image are adjusted, the erroneous warning frame in the warning image is deleted, and the missed targets in the return flow data to be judged are marked.

[0019] According to a third aspect of an embodiment of the present application, there is provided an electronic device, including a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0020] Memory, used to store computer programs;

[0021] The processor is used to implement the method provided in the first aspect when executing the program stored in the memory.

[0022] According to a fourth aspect of an embodiment of the present application, a non-temporary computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the method provided in the first aspect is implemented.

[0023] The data backflow analysis method of the embodiment of the present application obtains the backflow data to be analyzed, determines the algorithm identifier corresponding to the alarm event according to the alarm box in the alarm image included in the backflow data to be analyzed, determines the historical data set used for algorithm training according to the algorithm identifier, and determines the first text prompt word of the image-text multimodal detection large model according to the label text of the historical data set, and then, according to the first text prompt word, uses the image-text multimodal detection large model to analyze the backflow data to be analyzed, determines the prompt word by the label text of the training data, and uses the image-text multimodal detection large model to automatically analyze the backflow data to be analyzed, thereby realizing the automation of data backflow analysis, reducing labor costs and time consumption, and improving the speed of data processing. In addition, since human intervention is reduced, subjective deviations caused by differences in personal experience and understanding are reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 It is a flow chart of a data backflow analysis method provided in an embodiment of the present application;

[0025] Figure 2 It is a flow chart of a data backflow upgrade method based on a large multi-modal model of graphics and text provided in an embodiment of the present application;

[0026] Figure 3 It is a structural schematic diagram of a data backflow analysis device provided in an embodiment of the present application;

[0027] Figure 4 It is a structural schematic diagram of another data backflow analysis device provided in an embodiment of the present application;

[0028] Figure 5 It is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0029] In order to enable those skilled in the art to better understand the technical solutions provided by the embodiments of the present application and to make the above-mentioned purposes, features and advantages of the embodiments of the present application more obvious and understandable, the technical solutions in the embodiments of the present application are further described in detail below in conjunction with the accompanying drawings.

[0030] It should be noted that the serial numbers of the steps in the embodiments of the present application do not mean the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0031] See also Figure 1 , is a flow chart of a data backflow analysis method provided in an embodiment of the present application, such as Figure 1 As shown, the data backflow analysis method may include the following steps:

[0032] Step S100, obtaining the reflow data to be analyzed; wherein the reflow data to be analyzed includes an alarm image, and an alarm frame exists in the alarm image.

[0033] In the embodiments of the present application, in actual scenarios, a batch of starting data sets (hereinafter referred to as historical data sets) can be constructed to train the deployment algorithm model used in the initial stage of the project. After this batch of algorithm models are put online, the real point data will be analyzed, and a series of reflux data to be analyzed will be generated.

[0034] Among them, the deployed algorithm model refers to the algorithm model that is obtained by supervised training using labeled data for deployment to the field, which is generally a model that tends to be highly efficient.

[0035] Exemplarily, the reflux data to be analyzed may include an alarm image reported by the algorithm model detection.

[0036] Step S110: Determine the algorithm identifier corresponding to the alarm event according to the alarm frame in the alarm image, and determine the historical data set used for algorithm training according to the algorithm identifier.

[0037] In an embodiment of the present application, for any alarm frame in any alarm image, the algorithm id (identifier) ​​of the corresponding alarm event can be obtained by backtracking the analysis results, and the training data set (i.e. the above-mentioned historical data set) used to train the algorithm can be obtained based on the algorithm id.

[0038] Exemplarily, for any alarm image, the alarm event message that reports the alarm image can be determined (the alarm event message carries the alarm image), and the alarm event message can be parsed to obtain the coordinate information of each alarm frame in the alarm image carried in the alarm event message, as well as field information such as the algorithm ID corresponding to the alarm frame, thereby determining the algorithm ID corresponding to the alarm frame in the alarm image.

[0039] Exemplarily, for an alarm image, there may be multiple alarm frames corresponding to different alarm events in the alarm image.

[0040] For example, for a monitoring front end installed in a public place, the alarm image reported by it may include an alarm frame corresponding to smoking behavior and an alarm frame corresponding to littering behavior.

[0041] Exemplarily, for an alarm image including multiple alarm boxes corresponding to different alarm events, the algorithm identifiers corresponding to different alarm events can be determined respectively, and based on each algorithm identifier, the corresponding historical data set can be determined respectively, and the corresponding first text prompt word can be determined based on the label text of the historical data set.

[0042] Step S120: Determine the first text prompt word of the image-text multimodal detection model based on the label text of the training data set.

[0043] Exemplarily, for the training data set obtained in step S110, the label text of the training data set can be captured and combined with the prompt word template to generate text prompt words (which can be called first text prompt words) for the image multimodal large model.

[0044] For example, taking the helmet detection algorithm as an example, the label text may include "a person wearing a helmet" and "a person without a helmet", and the first text prompt words generated may be "a person wearing a helmet" and "a person without a helmet". The image-text multimodal detection model refers to a multimodal model that accepts input of one or more modes such as images and texts and outputs detection results.

[0045] Step S130: Based on the first text prompt word, the image-text multimodal detection model is used to analyze the return data to be analyzed.

[0046] In an embodiment of the present application, when the first text prompt word of the image multimodal large model is obtained in the above manner, the image-text multimodal detection large model can be used to analyze the return data to be analyzed based on the obtained first text prompt word.

[0047] Exemplarily, analyzing the return flow data to be analyzed includes one or more of the following:

[0048] The coordinates of the warning frame in the warning image are adjusted, the erroneous warning frame in the warning image is deleted, and the missed targets in the return data to be analyzed are marked.

[0049] Exemplarily, the reflux data to be analyzed can be analyzed based on the analysis results of the image-text multimodal detection large model on the reflux data to be analyzed.

[0050] Exemplarily, the analysis result of the image-text multimodal detection large model on the reflow data to be analyzed includes the coordinates of the target box analyzed by the image-text multimodal detection large model.

[0051] The return data to be analyzed can be analyzed by comparing the coordinates of the warning frame included in the return data to be analyzed and the coordinates of the target frame analyzed by the large image and text multimodal detection model.

[0052] For example, the reflow data to be analyzed can be analyzed based on the degree of overlap between the warning frame included in the reflow data to be analyzed and the target frame analyzed by the large image and text multimodal detection model.

[0053] It can be seen that in Figure 1 In the method flow shown, by obtaining the return flow data to be judged, according to the alarm box in the alarm image included in the return flow data to be judged, the algorithm identifier corresponding to the alarm event is determined, and according to the algorithm identifier, the historical data set used for algorithm training is determined, and according to the label text of the historical data set, the first text prompt word of the image-text multimodal detection large model is determined, and then, according to the first text prompt word, the image-text multimodal detection large model is used to judge the return flow data to be judged, and the prompt word is determined by the label text of the training data, and the image-text multimodal detection large model is used to automatically judge the return flow data to be judged, so as to realize the automation of data return judgment, reduce labor costs and time consumption, and improve the speed of data processing. In addition, due to the reduction of human intervention, the subjective deviation caused by personal experience and understanding differences is reduced.

[0054] In some embodiments, the above-mentioned judging the reflow data to be judged based on the first text prompt word by using the image-text multimodal detection large model may include:

[0055] For any warning image in the reflow data to be judged, the warning image is analyzed according to the first text prompt word using the image-text multimodal detection large model to obtain a first analysis result; wherein the first analysis result includes the coordinates of the first target frame and the confidence corresponding to the first target frame;

[0056] For any warning frame in the warning image, if there is a first target frame whose overlap degree with the warning frame exceeds a first overlap degree threshold and whose confidence is higher than a first confidence threshold, the first target frame is determined as a real warning frame corresponding to the warning frame; or,

[0057] If there is no first target frame whose overlap degree with the warning frame is greater than the second overlap degree threshold, or if there is a first target frame whose overlap degree with the warning frame is greater than the second overlap degree threshold, but the confidence corresponding to the first target frame is less than the second confidence threshold, delete the warning frame; or,

[0058] For any corresponding first target frame whose confidence is greater than the third confidence threshold, if there is no alarm frame in the alarm image data whose overlap degree with the first target frame is greater than the third overlap degree threshold, the third target frame is determined to be a newly added alarm frame.

[0059] For any warning image in the reflow data to be analyzed, the warning image can be analyzed based on the above-mentioned first text prompt word using the image-text multimodal detection large model to obtain an analysis result (which can be called the first analysis result), that is, the image-text multimodal detection large model is used to determine whether there is a target associated with the first text prompt word in the warning image.

[0060] For any target determined to be associated with the first text prompt word, the image-text multimodal detection model can output the confidence level corresponding to the target.

[0061] Exemplarily, the first analysis result may include the coordinates (image coordinates) of a target box (which may be referred to as a first target box) of a target associated with a text prompt word existing in the warning image determined by the large image-text multimodal detection model, and the confidence level corresponding to the first target box.

[0062] When the first analysis result of the image-text multimodal detection large model on the warning image is obtained, the degree of overlap between the first target frame and the warning frame can be determined based on the coordinates of the first target frame and the coordinates of the warning frame in the warning image.

[0063] In an example, the degree of overlap between the first target frame and the warning frame may be represented by an IOU (Intersection over Union) between the first target frame and the warning frame.

[0064] Among them, for a first target frame and an alarm frame, the ratio between the area of ​​the intersection between the first target frame and the alarm frame (that is, the area of ​​the overlapping area) and the area of ​​the union of the first target frame and the alarm frame (that is, the area of ​​the total area covered by the first target frame and the alarm frame, that is, the sum of the areas of the first target frame and the alarm frame, minus the area of ​​the overlapping area between the first target frame and the alarm frame) can be determined as the IOU between the first target frame and the alarm frame.

[0065] In one example, for any warning frame in the warning image, it can be determined whether there is a first target frame whose degree of overlap with the warning frame exceeds a preset overlap degree threshold (which can be called a first overlap degree threshold, for example, a first IOU threshold) and whose confidence level is higher than a preset confidence threshold (which can be called a first confidence threshold).

[0066] When there is a first target frame whose degree of overlap with the alarm frame exceeds a first overlap degree threshold and whose confidence level is higher than a first confidence level threshold, the first target frame can be determined as the real alarm frame corresponding to the alarm frame. In this case, the alarm frame can be deleted and the first target frame can be retained.

[0067] It should be noted that, for any alarm frame in the alarm image, if there is no first target frame whose overlap degree with the alarm frame exceeds the first overlap degree threshold and whose confidence level is higher than the first confidence level threshold, there is no need to adjust the alarm frame and the alarm frame can be regarded as a real alarm frame.

[0068] In one example, for any warning frame in the warning image, it can be determined whether there is a first target frame whose degree of overlap with the warning frame is greater than a preset overlap threshold (which can be called the second overlap threshold) and whose corresponding confidence is greater than or equal to a preset confidence threshold (which can be called the second confidence threshold).

[0069] The warning frame can be deleted if there is no first target frame whose degree of overlap with the warning frame is greater than the second overlap degree threshold, or if there is a first target frame whose degree of overlap with the warning frame is greater than the second overlap degree threshold but the confidence corresponding to the first target frame is less than the second confidence threshold.

[0070] It should be noted that, for any warning frame in the warning image, if there is a first target frame whose overlap degree with the warning frame is greater than the second overlap degree threshold and whose corresponding confidence is greater than or equal to the second confidence threshold, the warning frame can be retained.

[0071] In one example, for any corresponding first target frame whose confidence is greater than a preset confidence threshold (which may be referred to as a third confidence threshold), it can be determined whether there is an alarm frame in the alarm image whose degree of overlap with the first target frame is greater than a preset overlap degree threshold (which may be referred to as a third overlap degree threshold).

[0072] In the case that there is no warning frame in the warning image whose overlapping degree with the first target frame is greater than a third overlapping degree threshold, it can be determined that the first target frame is a newly added warning frame.

[0073] It should be noted that, for any corresponding first target frame whose confidence is greater than the third confidence threshold, if there is an alarm frame in the alarm image whose degree of overlap with the first target frame is greater than the third overlap threshold, the first target frame may not be added as a new alarm frame.

[0074] In some embodiments, the above-mentioned judging the reflow data to be judged based on the first text prompt word by using the image-text multimodal detection large model may include:

[0075] According to the confidence corresponding to each warning frame in the warning image, K1 warning frames whose confidences meet the conditions are selected; wherein the confidence corresponding to the warning frame whose confidences meet the conditions is higher than the confidence corresponding to the warning frame whose confidences do not meet the conditions;

[0076] According to the K1 warning frames, the warning images to which the K1 warning frames belong are cropped respectively to obtain K1 target small images corresponding to the K1 warning frames respectively;

[0077] The K1 target small images are used as visual prompt words of the large image-text multimodal detection model, and according to the visual prompt words and the first text prompt words, the large image-text multimodal detection model is used to analyze the return data to be analyzed and judged to obtain a second analysis result; wherein the second analysis result includes the coordinates of the second target box and the confidence of the second target box;

[0078] For any alarm box in the return flow data to be judged:

[0079] If there is a second target frame whose overlap degree with the warning frame exceeds a fourth overlap degree threshold and whose confidence level is higher than a fourth confidence level threshold, the second target frame is determined as a real warning frame corresponding to the warning frame; or,

[0080] If there is no second target frame whose overlap degree with the warning frame is greater than the fifth overlap degree threshold, or if there is a second target frame whose overlap degree with the warning frame is greater than the fifth overlap degree threshold, but the confidence corresponding to the second target frame is less than the fifth confidence threshold, delete the warning frame; or,

[0081] For any corresponding second target frame whose confidence is greater than the sixth confidence threshold, if there is no alarm frame in the alarm image data whose overlap degree with the second target frame is greater than the sixth overlap degree threshold, the second target frame is determined to be a newly added alarm frame.

[0082] Exemplarily, in order to improve the accuracy of the large model in analyzing the reflow data to be analyzed, K1 (K1≥1) alarm frames whose confidences meet the conditions may be selected based on the confidences corresponding to the alarm frames in the alarm images included in the reflow data to be analyzed.

[0083] In one example, K1 warning frames with higher confidence levels can be determined as warning frames with confidence levels that meet the conditions in descending order of confidence levels. The value of K1 can be set according to actual needs.

[0084] In another example, a confidence threshold (which may be referred to as the tenth confidence threshold) may be preset, and the corresponding alarm frame whose confidence is higher than the tenth confidence threshold is determined as the alarm frame whose confidence satisfies the condition. The value of K1 is determined according to the value of the tenth confidence threshold and the confidence corresponding to the alarm frame in the alarm image.

[0085] According to the selected K1 warning frames, the warning images to which the K1 warning frames belong may be cropped respectively to obtain K1 target small images corresponding to the K1 warning frames respectively.

[0086] The K1 target small images can be used as visual prompt words of the large image-text multimodal detection model. Based on the visual prompt words and the above-mentioned first text prompt words, the large image-text multimodal detection model is used to analyze the return data to be analyzed and obtain corresponding analysis results (which can be called second analysis results).

[0087] Exemplarily, the second analysis result may include the coordinates of the second target frame and the confidence level corresponding to the second target frame.

[0088] The degree of overlap between the second target frame and the warning frame in the warning image included in the reflow data to be analyzed may be determined.

[0089] In one example, for any alarm frame in the reflow data to be analyzed, if there is a second target frame whose degree of overlap with the alarm frame exceeds a preset overlap degree threshold (which can be called the fourth overlap degree threshold) and whose confidence level is higher than a preset confidence threshold (which can be called the fourth confidence threshold), the second target frame can be determined as the real alarm frame corresponding to the alarm frame.

[0090] In one example, for any alarm frame in the reflow data to be analyzed, if there is no second target frame whose degree of overlap with the alarm frame is greater than a preset overlap threshold (which can be called the fifth overlap threshold), or if there is a second target frame whose degree of overlap with the alarm frame is greater than the fifth overlap threshold, but the confidence corresponding to the second target frame is less than the preset confidence threshold (which can be called the fifth overlap threshold), the alarm frame can be considered to be an erroneous alarm frame. In this case, the alarm frame can be deleted.

[0091] In one example, for any corresponding second target frame whose confidence is greater than a preset confidence threshold (which may be referred to as the sixth confidence threshold), if there is no alarm frame in the alarm image data whose degree of overlap with the second target frame is greater than a preset overlap threshold (which may be referred to as the sixth overlap threshold), the second target frame is determined to be a newly added alarm frame.

[0092] In some embodiments, based on the text prompt words, the image-text multimodal detection model is used to analyze the return flow data to be analyzed, which may include:

[0093] According to the text prompt words, the coordinates of the warning frame in the warning image are adjusted by using the image-text multimodal detection large model, and / or the wrong warning frame in the warning image is deleted to obtain a processed warning frame;

[0094] According to the confidence of the processed alarm frame, K2 alarm frames whose confidence meets the condition are selected; wherein the confidence corresponding to the alarm frame whose confidence meets the condition is higher than the confidence corresponding to the alarm frame whose confidence does not meet the condition;

[0095] According to the K2 warning frames, the warning images to which the K2 warning frames belong are cropped respectively to obtain K2 target small images corresponding to the K2 warning frames respectively;

[0096] The K2 target small images are used as visual prompt words of the image-text multimodal detection large model. According to the visual prompt words and the first text prompt words, the image-text multimodal detection large model is used to mark the missed targets in the return data to be analyzed.

[0097] Exemplarily, for the acquired reflow data to be analyzed, the large image-text multimodal detection model can be used to first adjust the coordinates of the alarm frame in the alarm image included in the reflow data to be analyzed, and / or delete the erroneous alarm frame in the alarm image to obtain the processed alarm frame.

[0098] It should be noted that the coordinates of the alarm frame in the alarm image can be adjusted first, and then the erroneous alarm frame deletion process can be performed based on the alarm frame after the coordinate adjustment; or, the erroneous alarm frame deletion process can be performed first, and then the coordinates of the alarm frame after the erroneous alarm frame is deleted can be adjusted. The specific implementation is not limited here.

[0099] When the processed alarm box is obtained, the missed targets in the return data to be analyzed can be marked based on the processed alarm box and text prompt words using the large image-text multimodal detection model.

[0100] Exemplarily, K2 (K2 ≥ 1) warning frames whose confidences meet the condition may be selected according to the confidences of the processed warning frames.

[0101] In one example, K2 warning frames with higher confidence levels can be determined as warning frames with confidence levels that meet the conditions in descending order of confidence levels. The value of K2 can be set according to actual needs.

[0102] In another example, a confidence threshold (which may be referred to as the eleventh confidence threshold) may be preset, and the corresponding alarm frame whose confidence is higher than the eleventh confidence threshold is determined as the alarm frame whose confidence satisfies the condition. The value of K2 is determined according to the value of the eleventh confidence threshold and the confidence corresponding to the alarm frame in the alarm image.

[0103] According to the selected K2 warning frames, the warning images to which the K2 warning frames belong can be cropped respectively to obtain K2 target small images corresponding to the K2 warning frames respectively.

[0104] The K2 target small images can be used as visual prompt words of the large image-text multimodal detection model. Based on the visual prompt words and the above-mentioned first text prompt words, the large image-text multimodal detection model is used to mark the missed targets in the return data to be analyzed.

[0105] For example, based on the visual prompt word and the first text prompt word, the image-text multimodal detection large model can be used to analyze the return flow data to be analyzed and judged to obtain a corresponding analysis result (which can be called a third analysis result).

[0106] Exemplarily, the third analysis result may include the coordinates of the third target frame and the confidence level corresponding to the third target frame.

[0107] For any corresponding third target frame whose confidence is greater than a preset confidence threshold (which can be called the twelfth confidence threshold), if there is no alarm frame in the alarm image data whose degree of overlap with the third target frame is greater than the twelfth overlap degree threshold, the third target frame is determined to be a newly added alarm frame.

[0108] The specific implementation process of adjusting the coordinates of the warning frame in the warning image and deleting the erroneous warning frame in the warning image can refer to the relevant description in the above embodiment.

[0109] In one example, based on the text prompt words, the image-text multimodal detection model is used to analyze the return flow data to be analyzed, which may include:

[0110] According to the text prompt words, the coordinates of the warning frame in the warning image are adjusted by using the image-text multimodal detection model to obtain an adjusted warning frame;

[0111] According to the adjusted warning frame and text prompt words, the image-text multimodal detection large model is used to delete the wrong warning frame in the warning image to obtain the processed warning frame;

[0112] Based on the processed warning boxes and text prompts, the large-scale image-text multimodal detection model is used to mark the missed targets in the reflux data to be analyzed.

[0113] In some embodiments, the data backflow analysis solution provided in the embodiments of the present application may also include:

[0114] According to the confidence corresponding to each alarm box after the return flow data to be judged is judged, K3 alarm boxes whose confidences meet the conditions are selected; wherein, the confidence corresponding to the alarm box whose confidence meets the conditions is higher than the confidence corresponding to the alarm box whose confidence does not meet the conditions;

[0115] According to the K3 warning frames, the warning images to which the K3 warning frames belong are cropped respectively to obtain K3 target small images corresponding to the K3 warning frames respectively;

[0116] According to the label categories of the K3 target small images, the K3 target small images are grouped to obtain C groups of target small images;

[0117] For any group of target small images, the image understanding large model is used to determine the text description of the group of target small images, and based on the text description of the group of target small images, the second text prompt word of the image-text multimodal detection large model corresponding to the group of target small images is determined;

[0118] Use the group of target thumbnails as the visual prompt words of the image-text multimodal detection model corresponding to the group of target thumbnails. According to the visual prompt words, the second text prompt words and the first text prompt words corresponding to the group of target thumbnails, use the image-text multimodal detection model to analyze and detect the historical data set, and upgrade the historical data set based on the analysis and detection results.

[0119] For example, considering that there may be artificial omissions in the initial annotation process of the historical data set, in addition, the object shape in the scene of the historical data set may be different from the object shape in the point of the scene actually used by the model.

[0120] In order to supplement the annotated targets that may be missed in the historical dataset, we can use the text labels of the historical dataset (i.e., the first prompt word mentioned above) and the visual prompt words corresponding to the warning boxes in the warning images that meet the confidence conditions after reflux analysis in the above manner. We can use richer prompt words combined with a large multimodal model of images and texts to analyze and detect each image in the historical dataset.

[0121] For example, if an object that has not been labeled in the historical data set is detected, the detection result is used as new labeling information to complete the upgrade of the historical data set.

[0122] Accordingly, when the reflux data is analyzed and judged in the above manner, the historical data set can be fed back and updated based on the reflux data after the analysis and judgment.

[0123] Exemplarily, K3 alarm frames whose confidences meet the conditions may be selected based on the confidences corresponding to the alarm frames after the return flow data to be analyzed is analyzed.

[0124] In one example, K3 warning frames with the highest confidence rankings can be determined as warning frames with confidence satisfying the condition in descending order of confidence. The value of K3 can be set according to actual needs.

[0125] In another example, a confidence threshold (which may be referred to as the sixth confidence threshold) may be preset, and the corresponding alarm frame whose confidence is higher than the sixth confidence threshold is determined as the alarm frame whose confidence satisfies the condition. The value of K3 is determined according to the value of the sixth confidence threshold and the confidence corresponding to the alarm frame in the alarm image.

[0126] According to the K3 warning frames, the warning images to which the K3 warning frames belong can be cropped to obtain K3 target thumbnails corresponding to the K3 warning frames, and the K3 target thumbnails can be grouped according to their label categories to obtain C groups of target thumbnails (different groups of target thumbnails have different categories). For any group of target thumbnails (i.e., target thumbnails of any category), the image understanding model is used to determine the text description of the group of target thumbnails, and based on the text description of the group of target thumbnails, the text prompt word (which can be called the second text prompt word) of the image-text multimodal detection model corresponding to the group of target thumbnails is determined.

[0127] Among them, the image understanding large model refers to a multimodal large model that accepts image input and outputs a text description of the image (such as a series of text keywords related to the image).

[0128] Exemplarily, the second text prompt word can play a supplementary role. For example, the category of the historical data set includes the text label "garbage". When the target small image is analyzed by the large image understanding model, prompt words that are strongly related to garbage, such as "plastic bag" and "cardboard box", may be given as the second text prompt word.

[0129] Furthermore, this group of target thumbnails can be used as the visual prompt words of the image-text multimodal detection model corresponding to the group of target thumbnails. Based on the visual prompt words corresponding to the group of target thumbnails, the second text prompt words and the above-mentioned first text prompt words, the image-text multimodal detection model can be used to analyze and detect the historical data set, and the historical data set can be upgraded based on the analysis and detection results.

[0130] In an example, the above analysis and detection result may include the coordinates of the fourth target frame and the confidence level corresponding to the fourth target frame;

[0131] The above-mentioned updating of the historical data set based on the analysis and detection results may include:

[0132] For any labeled target box in the historical dataset:

[0133] In the case where there is a fourth target frame whose overlap degree with the marked target frame exceeds a seventh overlap degree threshold and whose confidence level is higher than a seventh confidence level threshold, the fourth target frame is determined as a true target frame corresponding to the marked target frame; or,

[0134] If there is no fourth target frame whose degree of overlap with the marked target frame is greater than the eighth overlap degree threshold, or if there is a fourth target frame whose degree of overlap with the warning frame is greater than the eighth overlap degree threshold, but the confidence corresponding to the fourth target frame is less than the eighth confidence threshold, delete the warning frame; or,

[0135] For any corresponding fourth target frame whose confidence is greater than the ninth confidence threshold, if there is no warning frame in the annotated target frame in the historical data set whose overlap degree with the fourth target frame is greater than the ninth overlap degree threshold, the fourth target frame is determined to be a newly added target frame.

[0136] Exemplarily, for the detection results of analyzing and detecting the historical data set using the large image-text multimodal detection model, the detection results can be compared with the annotation results of the historical data set.

[0137] As an example, for any annotated target box in the historical data set, if there is a fourth target box whose degree of overlap with the annotated target box exceeds a preset overlap degree threshold (which can be called the seventh overlap degree threshold) and whose confidence is higher than a preset confidence threshold (which can be called the seventh confidence threshold), the fourth target box is determined as the true target box corresponding to the annotated target box.

[0138] As an example, for any annotated target box in the historical data set, if there is no fourth target box whose degree of overlap with the annotated target box is greater than a preset overlap degree threshold (which can be called the eighth overlap degree threshold), or if there is a fourth target box whose degree of overlap with the warning box is greater than the eighth overlap degree threshold, but the confidence corresponding to the fourth target box is less than the preset confidence threshold (which can be called the eighth confidence threshold), the warning box is deleted.

[0139] As an example, for any corresponding fourth target box whose confidence is greater than a preset confidence threshold (which can be called the ninth confidence threshold), if there is no alarm box in the annotated target box in the historical data set whose degree of overlap with the fourth target box is greater than a preset overlap threshold (which can be called the ninth overlap threshold), the fourth target box is determined to be a newly added target box.

[0140] In order to enable those skilled in the art to better understand the technical solutions provided in the embodiments of the present application, the technical solutions provided in the embodiments of the present application are described below with reference to specific examples.

[0141] The present application embodiment provides a data backflow upgrade solution based on a large multi-modal model of graphics and texts, such as Figure 2 As shown, the specific implementation process is as follows:

[0142] 1. Obtain the data to be analyzed (i.e. the reflux data to be analyzed mentioned above).

[0143] For example, a batch of starting data sets can be constructed to train the deployment algorithm models used in the initial launch of the project. After these algorithm models are launched, they will analyze the real point data and generate a series of reflux data to be analyzed.

[0144] 2. The alarm data is automatically analyzed by the multimodal large model and flows back to the corresponding algorithm data set.

[0145] 2.1. Adjust the wrong alarm target (i.e. adjust the coordinates of the alarm box).

[0146] For example, in the process of analyzing a certain alarm frame of a specific alarm image, the algorithm id of the alarm event can be obtained by backtracking the analysis result, and the historical data set for training the algorithm can be obtained according to the algorithm id.

[0147] By capturing all the label texts of the historical data set and combining them with the prompt word template, the prompt words of the image-text multimodal detection large model (i.e., the first text prompt words mentioned above) can be generated. Then, based on the obtained first text prompt words, the image-text multimodal detection large model can be used to analyze the warning image.

[0148] If the IOU between a target box (such as the first target box mentioned above) and the warning box in the analysis result is greater than a preset threshold T A (i.e., the first overlap threshold, also called the first IOU threshold), and the confidence of the target box given by the large model is higher than a preset threshold T B (Such as the first confidence threshold mentioned above), in this case, it can be judged that the coordinates of the warning box are wrong (there is a deviation), and the detection result of the target box by the large image and text multimodal detection model can be used as the real warning box, so as to achieve the purpose of adjusting the coordinates of the warning box.

[0149] Exemplarily, the obtained real warning frame can be stored in a real warning frame queue together with the confidence level output by its large model.

[0150] Exemplarily, the real alarm frame queue contains alarm frames adjusted in the above manner (for the alarm frames with adjusted coordinates, the original alarm frames will be deleted), and it is determined that the alarm frames do not need to have their coordinates adjusted.

[0151] 2.2. Remove incorrect alarm targets.

[0152] Exemplarily, the first K warning frames are taken out from the real warning frame queue in order of confidence from high to low, and based on the first K warning frames, corresponding target thumbnails (a total of K target thumbnails) are respectively cut out from the warning images to which they belong.

[0153] The K target small images are used as visual prompt words of the large image-text multimodal detection model. Based on the visual prompt words and the first text prompt words of the large image-text multimodal detection model (the generation method can be seen in 2.1), the large image-text multimodal detection model is used to analyze the return data to be analyzed and obtain the analysis results (such as the second analysis result mentioned above).

[0154] For any alarm frame in the return flow data to be judged, if there is no IOU with the alarm frame greater than the threshold T C (such as the fifth overlap threshold, also referred to as the fifth IOU threshold) of the second target frame, or, there is an IOU with the warning frame greater than the threshold T C , but the confidence corresponding to the target box is less than the confidence threshold T D If the confidence threshold is greater than 5, the warning box is deleted.

[0155] 2.3. Mark the targets that are missed by the on-site model (the deployment algorithm model mentioned above).

[0156] After being processed in 2.1 and 2.2, the first K warning frames are taken out from the real warning frame queue in order of confidence from high to low, and based on the first K warning frames, the corresponding target thumbnails are cut out from the warning images to which they belong (a total of K target thumbnails).

[0157] The K small images are used as visual prompt words of the large image-text multimodal detection model. According to the visual prompt words of the large image-text multimodal detection model and the first text prompt words (the generation method can be seen in 2.1), the large image-text multimodal detection model is used to analyze the return data to be analyzed and obtain the analysis results (such as the second analysis result mentioned above).

[0158] If the confidence of any target box in the analysis result is greater than the confidence threshold T E , and the target frame does not overlap with the warning frame processed by 2.1 and 2.2 (judged by IOU, if IOU is less than the threshold T F ), the target frame is determined as a new alarm frame (the alarm frame corresponding to the missed target).

[0159] Through 2.1, 2.2 and 2.3, the data analysis work on the false positives, wrong positives and missed positives of the returned data is finally completed, and the entire work does not require human participation.

[0160] After the analysis is completed, the data is returned to the corresponding data set of the training platform (the analyzed data is stored in the corresponding data set of the training platform), and a new version number is created to save it.

[0161] 3. The reflux dataset feeds back to update the historical dataset.

[0162] Since the training and deployment algorithm models use early data sets (i.e., the historical data sets mentioned above), the image features of these data sets may differ from the model usage scenarios.

[0163] For example, taking the garbage pile detection algorithm as an example, suppose that the garbage pile targets marked in the initial data set are mostly construction waste. However, most of the garbage in the scenario where this algorithm is finally used is domestic waste. This is the problem of inconsistent data distribution between the training data and the usage scenario.

[0164] In addition, there may be problems with missing annotations in the training dataset, for example, missing annotations in the manually annotated initial dataset.

[0165] In order to solve the problem of inconsistent data distribution between training data and usage scenarios and the problem of manual omission of annotations, the embodiment of the present application also provides a method for a reflux dataset to feed back a historical dataset.

[0166] After the process in point 2 (such as 2.1 to 2.3 above), the judged reflux data set can be obtained, and the real alarm frame queue is retained in the process. The real alarm queue can be used to cut out the target small images according to the first K real alarm frames in order of confidence from high to low, and the cut K target small images are grouped according to the label category to obtain C groups of target small images (C is the number of label categories of the data set).

[0167] For any group of target small images (assuming the category is C1), they can be sent to the image understanding model in turn to obtain the text description belonging to the group of target small images, and then combined with the prompt word template to generate the text prompt words belonging to the group of target small images (that is, the second text prompt words mentioned above).

[0168] All the small pictures in the group of target small pictures can be used as visual prompt words. According to the visual prompt words, the second text prompt words and the above-mentioned first text prompt words, the image-text multimodal detection large model is used to analyze and detect the historical data set. After the detection result is purified by using IOU to remove the original labels, the newly added target annotations belonging to the C1 category on the historical data set are appended to the historical annotations.

[0169] Exemplarily, for the detection results of analyzing and detecting the historical data set using the large image-text multimodal detection model, the detection results can be compared with the annotation results of the historical data set.

[0170] For any detected target box (such as the fourth target box mentioned above) detected using the large image and text multimodal detection model, if there is no annotated target box in the historical data set whose IOU with the detected target box is greater than the preset IOU threshold (such as the ninth overlap threshold mentioned above), then the detected target box is determined to be a new target box. In this case, the detected target box can be added to the historical data set.

[0171] The above process is executed repeatedly until the labeling data of the target small pictures in group C are updated. In this case, the update of the historical data set is completed.

[0172] For example, the garbage pile detection model has an initial training set of point materials collected from a living area in a southern city, where most of the scenes are domestic garbage. The trained algorithm was eventually deployed in a major industrial city in the north, where most of the scenes were construction waste.

[0173] After the data analysis in the second point above, it is found that most of the labels in the return data set are construction waste. The labels in the historical data are all domestic waste, but there is a high possibility that there is unlabeled construction waste. In order to reflect the target features of the return data in the historical data set, the image understanding model can be used to analyze the targets in the return data to obtain text descriptions of the target features in the return data set (text description examples: garbage, construction waste, plastic bags, cardboard boxes, etc., which are related to garbage and industrial waste). Then, this text description is used as a text prompt word, and the target small picture corresponding to the high-confidence alarm box of the return data is used as a visual prompt word. The analysis shows that there are additional targets in the historical data set that have the same target category as the return data set but different features. For example, domestic garbage and construction waste belong to the same garbage pile, but have different features. At this point, the purpose of feeding back the historical data set is completed.

[0174] Exemplarily, when the historical data set is updated in the above manner, the deployed algorithm model can be optimized and updated using the updated historical data set.

[0175] The method provided by the present application is described above. The device provided by the present application is described below:

[0176] See also Figure 3 , is a structural diagram of a data backflow analysis device provided in an embodiment of the present application, such as Figure 3 As shown, the data backflow analysis device may include:

[0177] The acquisition unit 310 is configured to acquire the reflow data to be analyzed; wherein the reflow data to be analyzed includes an alarm image, and an alarm frame exists in the alarm image;

[0178] The first determination unit 320 is configured to determine the algorithm identifier corresponding to the alarm event according to the alarm frame in the alarm image, and determine the historical data set used for algorithm training according to the algorithm identifier;

[0179] The second determination unit 330 is configured to determine the first text prompt word of the image-text multimodal detection large model according to the label text of the historical data set;

[0180] The judging unit 340 is configured to judge the return flow data to be judged based on the first text prompt word using the image-text multimodal detection model;

[0181] The analysis of the reflux data to be analyzed includes one or more of the following:

[0182] The coordinates of the warning frame in the warning image are adjusted, the erroneous warning frame in the warning image is deleted, and the missed targets in the return data to be analyzed are marked.

[0183] In some embodiments, the second determining unit 330 is specifically configured to determine, for any warning image, the warning event message corresponding to the warning image;

[0184] Parse the alarm event message to obtain coordinate information of the alarm frame in the alarm image carried in the alarm event message, and an algorithm identifier corresponding to the alarm frame;

[0185] The algorithm identifier corresponding to the alarm frame in the alarm image is determined according to the coordinate information of each alarm frame in the alarm image carried in the alarm event message and the algorithm identifier corresponding to the alarm frame.

[0186] In some embodiments, the analysis and judgment unit 340 is specifically configured to:

[0187] For any warning image in the reflow data to be judged, the warning image is analyzed according to the first text prompt word using the image-text multimodal detection large model to obtain a first analysis result; wherein the first analysis result includes the coordinates of the first target frame and the confidence corresponding to the first target frame;

[0188] For any alarm frame in the alarm image:

[0189] If there is a first target frame whose overlap degree with the warning frame exceeds a first overlap degree threshold and whose confidence level is higher than a first confidence level threshold, the first target frame is determined as a real warning frame corresponding to the warning frame; or,

[0190] If there is no first target frame whose overlap degree with the warning frame is greater than the second overlap degree threshold, or if there is a first target frame whose overlap degree with the warning frame is greater than the second overlap degree threshold, but the confidence corresponding to the first target frame is less than the second confidence threshold, delete the warning frame; or,

[0191] For any corresponding first target frame whose confidence is greater than the third confidence threshold, if there is no alarm frame in the alarm image data whose overlap degree with the first target frame is greater than the third overlap degree threshold, the first target frame is determined to be a newly added alarm frame.

[0192] In some embodiments, the analysis and judgment unit 340 is specifically configured to:

[0193] According to the confidence corresponding to each warning frame in the warning image, K1 warning frames whose confidences meet the conditions are selected; wherein the confidence corresponding to the warning frame whose confidences meet the conditions is higher than the confidence corresponding to the warning frame whose confidences do not meet the conditions;

[0194] According to the K1 warning frames, the warning images to which the K1 warning frames belong are cropped respectively to obtain K1 target small images corresponding to the K1 warning frames respectively;

[0195] The K1 target small images are used as visual prompt words of the large image-text multimodal detection model, and the large image-text multimodal detection model is used to analyze the return data to be analyzed based on the visual prompt words and the first text prompt words to obtain a second analysis result; wherein the second analysis result includes the coordinates of the second target box and the confidence corresponding to the second target box;

[0196] For any alarm box in the return flow data to be judged:

[0197] If there is a second target frame whose overlap degree with the warning frame exceeds a fourth overlap degree threshold and whose confidence level is higher than a fourth confidence level threshold, the second target frame is determined as a real warning frame corresponding to the warning frame; or,

[0198] If there is no second target frame whose overlap degree with the warning frame is greater than the fifth overlap degree threshold, or if there is a second target frame whose overlap degree with the warning frame is greater than the fifth overlap degree threshold, but the confidence corresponding to the second target frame is less than the fifth confidence threshold, delete the warning frame; or,

[0199] For any corresponding second target frame whose confidence is greater than the sixth confidence threshold, if there is no alarm frame in the alarm image data whose overlap degree with the second target frame is greater than the sixth overlap degree threshold, the second target frame is determined to be a newly added alarm frame.

[0200] In some embodiments, the analysis and judgment unit 340 is specifically configured to:

[0201] According to the first text prompt word, using the image-text multimodal detection large model, the coordinates of the warning frame in the warning image are adjusted, and / or the wrong warning frame in the warning image is deleted to obtain a processed warning frame;

[0202] According to the confidence of the processed alarm frame, K2 alarm frames whose confidence meets the condition are selected; wherein the confidence corresponding to the alarm frame whose confidence meets the condition is higher than the confidence corresponding to the alarm frame whose confidence does not meet the condition;

[0203] According to the K2 warning frames, the warning images to which the K2 warning frames belong are cropped respectively to obtain K2 target small images corresponding to the K2 warning frames respectively;

[0204] The K2 target small images are used as visual prompt words of the image-text multimodal detection large model. According to the visual prompt words and the first text prompt words, the image-text multimodal detection large model is used to mark the missed targets in the return data to be analyzed.

[0205] In some embodiments, Figure 4 As shown, the data backflow analysis device may also include:

[0206] The upgrading unit 350 is configured to select K3 alarm boxes whose confidences meet the conditions according to the confidences corresponding to the alarm boxes after the reflow data to be judged is judged; wherein the confidences corresponding to the alarm boxes whose confidences meet the conditions are higher than the confidences corresponding to the alarm boxes whose confidences do not meet the conditions;

[0207] According to the K3 warning frames, the warning images to which the K3 warning frames belong are cropped respectively to obtain K3 target small images corresponding to the K3 warning frames respectively;

[0208] According to the label categories of the K3 target small images, the K3 target small images are grouped to obtain C groups of target small images;

[0209] For any group of target small images, the image understanding large model is used to determine the text description of the group of target small images, and based on the text description of the group of target small images, the second text prompt word of the image-text multimodal detection large model corresponding to the group of target small images is determined;

[0210] Use the group of target thumbnails as the visual prompt words of the image-text multimodal detection model corresponding to the group of target thumbnails. According to the visual prompt words, the second text prompt words and the first text prompt words corresponding to the group of target thumbnails, use the image-text multimodal detection model to analyze and detect the historical data set, and upgrade the historical data set based on the analysis and detection results.

[0211] In some embodiments, the analysis and detection result includes the coordinates of the fourth target frame and the confidence level corresponding to the fourth target frame;

[0212] The upgrading unit 350 is specifically configured to, for any labeled target box in the historical data set:

[0213] In the case where there is a fourth target frame whose overlap degree with the marked target frame exceeds a seventh overlap degree threshold and whose confidence level is higher than a seventh confidence level threshold, the fourth target frame is determined as a true target frame corresponding to the marked target frame; or,

[0214] If there is no fourth target frame whose degree of overlap with the marked target frame is greater than the eighth overlap degree threshold, or if there is a fourth target frame whose degree of overlap with the warning frame is greater than the eighth overlap degree threshold, but the confidence corresponding to the fourth target frame is less than the eighth confidence threshold, delete the warning frame; or,

[0215] For any corresponding fourth target frame whose confidence is greater than the ninth confidence threshold, if there is no warning frame in the annotated target frame in the historical data set whose overlap degree with the fourth target frame is greater than the ninth overlap degree threshold, the fourth target frame is determined to be a newly added target frame.

[0216] See also Figure 5 , is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application. The electronic device may include a processor 501, a communication interface 502, a memory 503 and a communication bus 504. The processor 501, the communication interface 502 and the memory 503 communicate with each other through the communication bus 504. Among them, the memory 503 stores a computer program; the processor 501 can execute the data reflux analysis method described above by executing the program stored in the memory 503.

[0217] The memory 503 mentioned herein may be any electronic, magnetic, optical or other physical storage device that may contain or store information, such as executable instructions, data, etc. For example, the memory 503 may be: RAM (Radom Access Memory), volatile memory, non-volatile memory, flash memory, storage drive (such as hard disk drive), solid state drive, any type of storage disk (such as optical disk, DVD, etc.), or similar storage medium, or a combination thereof.

[0218] The present application also provides a non-transitory machine-readable storage medium storing a computer program, such as Figure 5 The computer program may be stored in the memory 503. Figure 5 The processor 501 in the electronic device shown executes to implement the data backflow analysis method described above.

[0219] The present application also provides a computer program stored in a non-transitory machine-readable storage medium, such as Figure 5 The memory 503 in the memory 503, and when the processor executes the computer program, it prompts the processor 501 to execute the data backflow judgment method described above.

Claims

1. A method for analyzing data backflow, characterized in that: include: Acquire the reflow data to be determined; wherein the reflow data to be determined includes an alarm image, and an alarm frame exists in the alarm image; According to the alarm frame in the alarm image, the algorithm identifier corresponding to the alarm event is determined, and according to the algorithm identifier, the historical data set used for algorithm training is determined; wherein the return flow data to be judged is generated by analyzing the real point data using the deployment algorithm model, and the deployment algorithm model is trained using the historical data set; Determining a first text prompt word of a large image-text multimodal detection model according to the label text of the historical data set; According to the first text prompt word, the image-text multimodal detection large model is used to analyze the return data to be analyzed; The judging of the reflux data to be judged includes one or more of the following: The coordinates of the warning frame in the warning image are adjusted, the erroneous warning frame in the warning image is deleted, and the missed targets in the return flow data to be judged are marked.

2. The method according to claim 1, characterized in that The step of determining an algorithm identifier corresponding to an alarm event according to an alarm frame in the alarm image includes: For any alarm image, determine the alarm event message corresponding to the alarm image; Parsing the alarm event message to obtain coordinate information of an alarm frame in an alarm image carried in the alarm event message, and an algorithm identifier corresponding to the alarm frame; The algorithm identifier corresponding to the alarm frame in the alarm image is determined according to the coordinate information of each alarm frame in the alarm image carried in the alarm event message and the algorithm identifier corresponding to the alarm frame.

3. The method according to claim 1, characterized in that The step of analyzing the to-be-analyzed reflux data based on the first text prompt word and using the image-text multimodal detection large model includes: For any warning image in the reflow data to be judged, the warning image is analyzed according to the first text prompt word using the image-text multimodal detection large model to obtain a first analysis result; wherein the first analysis result includes the coordinates of the first target box and the confidence level corresponding to the first target box; For any alarm frame in the alarm image: If there is a first target frame whose overlap degree with the warning frame exceeds a first overlap degree threshold and whose confidence level is higher than a first confidence level threshold, the first target frame is determined as a real warning frame corresponding to the warning frame; or, If there is no first target frame whose overlap degree with the warning frame is greater than the second overlap degree threshold, or if there is a first target frame whose overlap degree with the warning frame is greater than the second overlap degree threshold, but the confidence corresponding to the first target frame is less than the second confidence threshold, delete the warning frame; or, For any corresponding first target frame whose confidence is greater than the third confidence threshold, if there is no alarm frame in the alarm image data whose overlap degree with the first target frame is greater than the third overlap degree threshold, the first target frame is determined to be a newly added alarm frame.

4. The method according to claim 1, characterized in that: The step of analyzing the to-be-analyzed reflux data based on the first text prompt word and using the image-text multimodal detection large model includes: According to the confidence corresponding to each warning frame in the warning image, K1 warning frames whose confidences meet the conditions are selected; wherein the confidence corresponding to the warning frame whose confidences meet the conditions is higher than the confidence corresponding to the warning frame whose confidences do not meet the conditions; According to the K1 warning frames, respectively perform image cropping on the warning images to which the K1 warning frames belong, to obtain K1 target small images respectively corresponding to the K1 warning frames; The K1 target small images are used as visual prompt words of the image-text multimodal detection large model, and according to the visual prompt words and the first text prompt words, the image-text multimodal detection large model is used to analyze the return data to be judged to obtain a second analysis result; wherein the second analysis result includes the coordinates of the second target box and the confidence corresponding to the second target box; For any alarm box in the return flow data to be judged: If there is a second target frame whose overlap degree with the warning frame exceeds a fourth overlap degree threshold and whose confidence level is higher than a fourth confidence level threshold, the second target frame is determined as a real warning frame corresponding to the warning frame; or, If there is no second target frame whose overlap degree with the warning frame is greater than the fifth overlap degree threshold, or if there is a second target frame whose overlap degree with the warning frame is greater than the fifth overlap degree threshold, but the confidence corresponding to the second target frame is less than the fifth confidence threshold, delete the warning frame; or, For any corresponding second target frame whose confidence is greater than the sixth confidence threshold, if there is no alarm frame in the alarm image data whose overlap degree with the second target frame is greater than the sixth overlap degree threshold, the second target frame is determined to be a newly added alarm frame.

5. The method according to any one of claims 1 to 4, characterized in that: The step of analyzing the to-be-analyzed reflux data based on the first text prompt word and using the image-text multimodal detection large model includes: According to the first text prompt word, using the image-text multimodal detection large model, the coordinates of the warning frame in the warning image are adjusted, and / or the wrong warning frame in the warning image is deleted to obtain a processed warning frame; According to the confidence of the processed alarm frame, K2 alarm frames whose confidences meet the conditions are selected; wherein the confidences corresponding to the alarm frames whose confidences meet the conditions are higher than the confidences corresponding to the alarm frames whose confidences do not meet the conditions; According to the K2 warning frames, respectively cropping the warning images to which the K2 warning frames belong, to obtain K2 target small images respectively corresponding to the K2 warning frames; The K2 target small images are used as visual prompt words of the image-text multimodal detection large model. According to the visual prompt words and the first text prompt words, the image-text multimodal detection large model is used to mark the missed targets in the reflux data to be judged.

6. The method according to any one of claims 1 to 4, characterized in that: The method further comprises: According to the confidence corresponding to each alarm frame after the return data to be judged is judged, K3 alarm frames whose confidences meet the conditions are selected; wherein the confidence corresponding to the alarm frame whose confidence meets the conditions is higher than the confidence corresponding to the alarm frame whose confidence does not meet the conditions; According to the K3 warning frames, respectively cropping the warning images to which the K3 warning frames belong, to obtain K3 target small images respectively corresponding to the K3 warning frames; According to the label categories of the K3 target small images, the K3 target small images are grouped to obtain C groups of target small images; For any group of target small images, determine the text description of the group of target small images by using the image understanding large model, and determine the second text prompt word of the image-text multimodal detection large model corresponding to the group of target small images based on the text description of the group of target small images; The group of target thumbnails is used as the visual prompt words of the image-text multimodal detection large model corresponding to the group of target thumbnails. According to the visual prompt words corresponding to the group of target thumbnails, the second text prompt words and the first text prompt words, the image-text multimodal detection large model is used to analyze and detect the historical data set, and the historical data set is upgraded according to the analysis and detection results.

7. The method according to claim 6, characterized in that The analysis and detection result includes the coordinates of the fourth target frame and the confidence level corresponding to the fourth target frame; The updating of the historical data set according to the analysis and detection results includes: For any labeled target box in the historical dataset: In the case where there is a fourth target frame whose overlap degree with the marked target frame exceeds a seventh overlap degree threshold and whose confidence level is higher than a seventh confidence level threshold, the fourth target frame is determined as a true target frame corresponding to the marked target frame; or, If there is no fourth target frame whose degree of overlap with the marked target frame is greater than the eighth overlap degree threshold, or if there is a fourth target frame whose degree of overlap with the warning frame is greater than the eighth overlap degree threshold, but the confidence corresponding to the fourth target frame is less than the eighth confidence threshold, delete the warning frame; or, For any corresponding fourth target frame whose confidence is greater than the ninth confidence threshold, if there is no warning frame in the annotated target frame in the historical data set whose overlap degree with the fourth target frame is greater than the ninth overlap degree threshold, the fourth target frame is determined to be a newly added target frame.

8. A data backflow analysis device, characterized in that: include: An acquisition unit is configured to acquire the reflow data to be determined; wherein the reflow data to be determined includes an alarm image, and an alarm frame exists in the alarm image; A first determination unit is configured to determine an algorithm identifier corresponding to the alarm event according to the alarm frame in the alarm image, and determine a historical data set used for algorithm training according to the algorithm identifier; wherein the return flow data to be determined is generated by analyzing the real point data using a deployment algorithm model, and the deployment algorithm model is trained using the historical data set; A second determination unit is configured to determine a first text prompt word of the image-text multimodal detection large model according to the label text of the historical data set; A judging unit is configured to judge the reflux data to be judged based on the text prompt word and using the image-text multimodal detection large model; The judging of the reflux data to be judged includes one or more of the following: The coordinates of the warning frame in the warning image are adjusted, the erroneous warning frame in the warning image is deleted, and the missed targets in the return flow data to be judged are marked.

9. An electronic device, characterized in that: include: A processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus; Memory, used to store computer programs; A processor, configured to implement the method according to any one of claims 1 to 7 when executing a program stored in a memory.

10. A non-transitory computer-readable storage medium, characterized in that: The non-transitory computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images