Data detection method, related device, equipment and storage medium

Abstract

The invention discloses a data detection method, a related device, equipment and a storage medium, and the method comprises the steps: obtaining to-be-detected multimedia data; constructing a multi-modal prompt instruction based on the sample multimedia data related to the target event and the description text of the sample multimedia data; wherein the multi-modal prompt instruction is used for indicating the multi-modal large model reference sample multimedia data and the description text to detect whether the to-be-detected multimedia data contains a target event; and obtaining a detection result representing whether the to-be-detected multimedia data contains the target event or not based on an output result of the multi-modal large model in response to the multi-modal prompt instruction. According to the scheme, the application range of data detection can be expanded, the data requirement for supervised data is reduced as much as possible, and the response timeliness of data detection is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a data detection method and related apparatus, equipment and storage medium. Background Technology

[0002] Data detection has a wide range of applications in many fields such as urban management and industrial production. It can detect whether a specific event has occurred (such as "motorcycles turning upside down" in urban management, "not wearing a safety helmet" in industrial production, etc.).

[0003] Thanks to the rapid development of artificial intelligence technology, existing data detection techniques mainly rely on deep learning models (e.g., computer vision models) pre-trained on supervised data (e.g., multimedia data labeled with samples indicating the presence or absence of specific events) to predict the existence of specific events in the current multimodal data. However, existing data detection techniques have several limitations: firstly, they are difficult to apply to events other than specific ones, meaning they are quite restrictive; secondly, their accuracy depends entirely on the quantity and quality of supervised data; and thirdly, their training cycles are typically long, making them difficult to meet the need for timely responses in data detection. Therefore, expanding the applicability of data detection, minimizing the need for supervised data, and improving the timeliness of data detection responses have become urgent problems to be solved. Summary of the Invention

[0004] The main technical problem addressed by this application is to provide a data detection method and related apparatus, equipment, and storage medium that can expand the applicability of data detection, minimize the data requirements for supervised data, and improve the timeliness of data detection response.

[0005] To address the aforementioned technical problems, the first aspect of this application provides a data detection method, comprising: acquiring multimedia data to be tested; constructing a multimodal prompt instruction based on sample multimedia data related to a target event and descriptive text of the sample multimedia data; wherein the multimodal prompt instruction is used to instruct a multimodal large model to detect whether the multimedia data to be tested contains a target event based on the sample multimedia data and the descriptive text; and obtaining a detection result characterizing whether the multimedia data to be tested contains a target event based on the output result of the multimodal large model responding to the multimodal prompt instruction.

[0006] Therefore, the process involves acquiring the multimedia data to be tested, and then constructing a multimodal cue instruction based on sample multimedia data and its descriptive text related to the target event. This multimodal cue instruction instructs the multimodal large model to detect whether the multimedia data to be tested contains the target event, referencing the sample multimedia data and its descriptive text. Finally, based on the output of the multimodal large model in response to the multimodal cue instruction, a detection result representing whether the multimedia data to be tested contains the target event is obtained. Thus, on the one hand, the multimodal large model can detect whether the multimedia data to be tested contains the target event under the instruction of the multimodal cue instruction; on the other hand, when there is a need to detect other events besides the target event, the sample multimedia data and its descriptive text are adaptively adjusted according to these other events. The text describes how reconstructing multimodal prompts enables large multimodal models to detect events beyond the target event, thus expanding the applicability of data detection. Furthermore, thanks to the general knowledge representation capabilities of the large multimodal model, it exhibits strong generalization reasoning abilities even in zero-shot or small-shot scenarios. Compared to traditional deep learning models, this helps minimize the need for supervised data. Moreover, unlike traditional deep learning models that require full-scale retraining for task adaptation, the large multimodal model achieves task adaptation through dynamic adjustment of multimodal instructions, improving the responsiveness of data detection. Therefore, it expands the applicability of data detection, minimizes the need for supervised data, and enhances the responsiveness of data detection.

[0007] The process of acquiring sample multimedia data and descriptive text related to the target event includes: displaying a configuration interface; wherein the configuration interface includes a first control for configuring sample multimedia data and a second control for configuring descriptive text; acquiring input multimedia data from the first control as sample multimedia data, and acquiring input text from the second control as descriptive text.

[0008] Therefore, the configuration interface is displayed, and the configuration interface includes a first control for configuring sample multimedia data and a second control for configuring descriptive text. Then, the input multimedia data from the first control is obtained as sample multimedia data, and the input text from the second control is obtained as descriptive text. Thus, only sample multimedia data and its descriptive text are needed during the data detection process, without uploading sensitive user data, which helps to achieve model fine-tuning through local configuration of prompt words.

[0009] The configuration interface is divided into a first area for configuring positive examples and a second area for configuring negative examples. Both the first and second areas contain a first control and a second control. The interface acquires input multimedia data from the first control as example multimedia data, including: acquiring input multimedia data from the first control in the first area as positive example multimedia data, and acquiring input multimedia data from the first control in the second area as negative example multimedia data; wherein the positive example multimedia data contains a target event, and the negative example multimedia data contains the opposite event of the target event. The interface also acquires input text from the second control as description text, including: acquiring input text from the second control in the first area as description text for the positive example multimedia data, and acquiring input text from the second control in the second area as description text for the negative example multimedia data. A multimodal prompt instruction is used to instruct the multimodal large model to refer to the positive example multimedia data and the description text of the positive and negative example multimedia data to detect whether the multimedia data under test contains the target event.

[0010] Therefore, the configuration interface is divided into a first area for configuring positive examples and a second area for configuring negative examples. Both the first and second areas contain a first control and a second control. Input multimedia data from the first control in the first area is obtained as positive example multimedia data, and input multimedia data from the first control in the second area is obtained as negative example multimedia data. The positive example multimedia data contains the target event, and the negative example multimedia data contains the opposite event of the target event. Input text from the second control in the first area is obtained as the description text of the positive example multimedia data, and input text from the second control in the second area is obtained as the description text of the negative example multimedia data. The multimodal prompt instruction is used to instruct the multimodal large model to refer to the positive example multimedia data and its description text, as well as the negative example multimedia data and its description text, to detect whether the multimedia data under test contains the target event. In this way, it is possible to support local configuration of positive example multimedia data and its description text, as well as negative example multimedia data and its description text, so that the multimodal large model can refer to both positive and negative examples at the same time, which helps to improve the accuracy of data detection.

[0011] The configuration interface also includes a third control for configuring event thresholds. The method further includes: obtaining an input threshold from the third control as the event threshold; wherein the output result includes the predicted probability that the multimedia data under test contains the target event, and when the predicted probability meets the filtering condition of containing the event threshold, the detection result includes that the multimedia data under test contains the target event, and when the predicted probability does not meet the filtering condition, the detection result includes that the multimedia data under test does not contain the target event.

[0012] Therefore, the configuration interface also includes a third control for configuring event thresholds. The input threshold from the third control is used as the event threshold, and the output includes the predicted probability that the multimedia data under test contains the target event. When the predicted probability meets the filtering condition of containing the event threshold, the detection result includes that the multimedia data under test contains the target event. When the predicted probability does not meet the filtering condition, the detection result includes that the multimedia data under test does not contain the target event. It can support the custom configuration of event thresholds, so as to combine the event thresholds and the predicted probability output by the multimodal large model to determine the detection result, which helps to meet the data detection needs of different levels.

[0013] The configuration interface is divided into a third area for configuring event thresholds from the reference perspective of sample multimedia data and a fourth area for configuring event thresholds from the reference perspective of descriptive text. Both the third and fourth areas contain third controls. The input thresholds from the third controls are used as event thresholds, including: obtaining the input threshold from the third control in the third area as a first threshold, and obtaining the input threshold from the third control in the fourth area as a second threshold. The event thresholds include the first and second thresholds. The prediction probabilities include: a first probability that the multimodal large model predicts the target multimedia data containing the target event from the reference perspective of the sample multimedia data; and a second probability that the multimodal large model predicts the target multimedia data containing the target event from the reference perspective of the descriptive text. When the first probability satisfies the first condition containing the first threshold and the second probability satisfies the second condition containing the second threshold, the detection result includes the target multimedia data containing the target event. When the first probability does not satisfy the first condition and / or the second probability does not satisfy the second condition, the detection result includes the target multimedia data not containing the target event.

[0014] Therefore, the configuration interface is divided into a third area for configuring event thresholds from the reference perspective of sample multimedia data and a fourth area for configuring event thresholds from the reference perspective of descriptive text. Both the third and fourth areas contain third controls. The input threshold from the third control in the third area is used as the first threshold, and the input threshold from the third control in the fourth area is used as the second threshold. The event thresholds include the first threshold and the second threshold. The prediction probabilities include: a first probability that the multimodal large model predicts the target multimedia data contains the target event from the reference perspective of sample multimedia data, and a second probability that the multimodal large model predicts the target multimedia data contains the target event from the reference perspective of descriptive text. When the first probability satisfies the first condition of containing the first threshold and the second probability satisfies the second condition of containing the second threshold, the detection result includes the target multimedia data containing the target event. When the first probability does not satisfy the first condition and / or the second probability does not satisfy the second condition, the detection result includes the target multimedia data not containing the target event. This allows for custom configuration of the corresponding thresholds from the reference perspective of descriptive text and the reference perspective of sample multimedia data, which helps to further refine the data detection needs at different levels.

[0015] The method further includes, after obtaining the detection result representing whether the multimedia data to be tested contains the target event based on the output result of the multimodal large model responding to the multimodal prompt command, the method also includes: determining whether to display the multimedia data to be tested in the target event retrieval interface based on the detection result of the multimedia data to be tested; determining whether to select the multimedia data to be tested as sample multimedia data for fine-tuning training of the multimodal large model based on the annotation result of the target object on the multimedia data to be tested in the retrieval interface; wherein, the annotation result represents whether the detection result of the multimedia data to be tested is incorrect.

[0016] Therefore, after obtaining the detection results representing whether the multimedia data to be tested contains the target event based on the output results of the multimodal large model responding to the multimodal prompt instructions, it is determined whether to display the multimedia data to be tested in the target event retrieval interface based on the detection results of the multimedia data to be tested. Then, based on the annotation results of the multimedia data to be tested in the retrieval interface of the target object, it is determined whether to select the multimedia data to be tested as sample multimedia data for fine-tuning training of the multimodal large model. Moreover, the annotation results represent whether the detection results of the multimedia data to be tested are erroneous. It can support the custom annotation of the detected multimedia data to be tested, and combine the annotation results to select a portion of the multimedia data to be tested for fine-tuning training of the multimodal large model, forming a dynamic iterative closed loop of "annotation-training-optimization", which helps to improve the detection capability of the multimodal large model.

[0017] Specifically, when the detection results indicate that the multimedia data to be tested contains the target event, the retrieval interface displays the multimedia data to be tested.

[0018] Therefore, when the detection results indicate that the multimedia data to be tested contains the target event, the retrieval interface displays the multimedia data to be tested, thus supporting the display of the retrieved data at the interactive level, which helps to improve the perceptibility of the retrieval results.

[0019] In cases where the annotation results indicate that the detection results of the multimedia data to be tested are incorrect, the multimedia data to be tested is selected as sample multimedia data.

[0020] Therefore, when the annotation results indicate that the detection results of the multimedia data to be tested are incorrect, the multimedia data to be tested is selected as sample multimedia data. The multimodal large model can be further fine-tuned and trained by using the multimedia data to be tested that are labeled as having incorrect detection results, which helps to improve the detection capability of the multimodal large model.

[0021] The task information for fine-tuning training includes at least the name of the large model and the large model hint instructions associated with the name of the large model. The name of the large model is the search text of the target event in the search interface by default, and the large model hint instructions are multimodal hint instructions by default.

[0022] Therefore, the task information for fine-tuning training includes at least the name of the large model and the large model hint instructions associated with the name of the large model. The name of the large model is the search text of the target event in the search interface by default, and the large model hint instructions are multimodal hint instructions by default, which helps to start fine-tuning training in combination with task information.

[0023] The method, after fine-tuning the training, further includes: displaying a task interface; wherein the task interface displays a fourth control, which is used to configure the target event detection task; obtaining input information from the fourth control as configuration information for the detection task; wherein the configuration information includes: the target area to be monitored for the target event, and a multimodal prompt instruction for detecting whether new multimedia data to be tested from the target area contains the target event; and starting the detection task based on the configuration information to detect whether new multimedia data to be tested contains the target event using the multimodal large model after fine-tuning the training.

[0024] Therefore, after fine-tuning the training, the task interface is displayed, which shows the fourth control. The fourth control is used to configure the target event detection task and obtain input information from the fourth control as the configuration information for the detection task. The configuration information includes: the target area to be monitored for the target event, and a multimodal prompt instruction for detecting whether new multimedia data to be tested from the target area contains the target event. The detection task is then started based on the configuration information, so that the multimodal large model after fine-tuning the training can detect whether new multimedia data to be tested contains the target event. It can support the custom configuration of the detection task and start data detection in combination with the custom configuration information, which helps to detect whether the target event has occurred in new multimedia data to be tested from the target area.

[0025] In the process of processing new multimedia data to be tested by the multimodal large model after fine-tuning training, the method further includes: in response to the detection that the multimodal large model falsely reports that the new multimedia data to be tested contains the target event, adjusting at least one of the sample multimedia data and the description text in the multimodal prompt instruction based on the new multimedia data to be tested that is falsely reported to contain the target event, to obtain a new multimodal prompt instruction, and restarting the fine-tuning training of the multimodal large model based on the new multimodal prompt instruction.

[0026] Therefore, in response to the detection of a new test multimedia data containing a target event that is falsely reported by the multimodal large model, at least one of the sample multimedia data and the descriptive text in the multimodal prompt instruction is adjusted based on the new test multimedia data that is falsely reported to contain the target event to obtain a new multimodal prompt instruction. Based on the new multimodal prompt instruction, the fine-tuning training of the multimodal large model is restarted. This allows for the adjustment of the multimodal prompt instruction and the restart of fine-tuning training when a false report scenario is detected, without having to redeploy the entire model.

[0027] The sample multimedia data includes at least one of positive and negative multimedia data of the target event. The positive multimedia data includes the target event, and the negative multimedia data includes the opposite event of the target event. When the sample multimedia data includes positive multimedia data, the description text of the sample multimedia data includes the description text of the positive multimedia data. When the sample multimedia data includes negative multimedia data, the description text of the sample multimedia data includes the description text of the negative multimedia data.

[0028] Therefore, the sample multimedia data includes at least one of the positive and negative multimedia data of the target event, the positive multimedia data includes the target event, and the negative multimedia data includes the opposite event of the target event. Furthermore, when the sample multimedia data includes positive multimedia data, the description text of the sample multimedia data includes the description text of the positive multimedia data, and when the sample multimedia data includes negative multimedia data, the description text of the sample multimedia data includes the description text of the negative multimedia data. This enables the multimodal large model to simultaneously refer to both positive and negative samples during data detection, which helps to improve the accuracy of data detection.

[0029] The sample multimedia data and the multimedia data to be tested belong to the same data type, and the data type includes at least one of image, video, and audio.

[0030] Therefore, the sample multimedia data and the multimedia data to be tested belong to the same data type, and the data type includes at least one of images, videos, and audio. This enables the multimodal large model to perform relevant detection on the multimedia data to be tested that involves the same data type after referring to the sample multimedia data.

[0031] To address the aforementioned technical problems, a second aspect of this application provides a data detection device, comprising: an acquisition module, a construction module, and a processing module. The acquisition module is used to acquire sample multimedia data related to a target event and descriptive text of the sample multimedia data, and to acquire multimedia data to be tested. The construction module is used to construct a multimodal prompt instruction based on the sample multimedia data and the descriptive text. The multimodal prompt instruction is used to instruct a multimodal large model to reference the sample multimedia data and its descriptive text to detect whether the multimedia data to be tested contains the target event. The processing module is used to obtain a detection result characterizing whether the multimedia data to be tested contains the target event based on the output result of the multimodal large model responding to the multimodal prompt instruction.

[0032] To address the aforementioned technical problems, a third aspect of this application provides an electronic device comprising at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor executes the program instructions to implement the data detection method of the first aspect described above.

[0033] To address the aforementioned technical problems, a fourth aspect of this application provides a computer-readable storage medium storing program instructions executable by a processor, the program instructions being used to implement the data detection method of the first aspect described above.

[0034] The above scheme acquires the multimedia data to be tested, and then constructs a multimodal cue instruction based on sample multimedia data and descriptive text related to the target event. This multimodal cue instruction instructs the multimodal large model to detect whether the multimedia data to be tested contains the target event, referencing the sample multimedia data and descriptive text. Based on the output of the multimodal large model in response to the multimodal cue instruction, a detection result representing whether the multimedia data to be tested contains the target event is obtained. Therefore, on the one hand, the multimodal large model can detect whether the multimedia data to be tested contains the target event under the instruction of the multimodal cue instruction; on the other hand, when there is a need to detect other events besides the target event, the sample multimedia data and its descriptive text are adaptively adjusted according to the other events. Describing text to reconstruct multimodal prompts enables large multimodal models to detect events beyond the target event, thus expanding the applicability of data detection. Furthermore, thanks to the general knowledge representation capabilities of the large multimodal model itself, it exhibits strong generalization reasoning abilities even in zero-shot or small-shot scenarios. Compared to traditional deep learning models, this helps minimize the need for supervised data. Moreover, unlike traditional deep learning models that require full-scale retraining for task adaptation, the large multimodal model achieves task adaptation through dynamic adjustment of multimodal instructions, improving the responsiveness of data detection. Therefore, it expands the applicability of data detection, minimizes the need for supervised data, and enhances the responsiveness of data detection. Attached Figure Description

[0035] Figure 1 This is a flowchart illustrating an embodiment of the data detection method of this application; Figure 2a This is a schematic diagram of one embodiment of the configuration interface of this application; Figure 2b This is a schematic diagram of an embodiment of the search interface of this application; Figure 2c This is a schematic diagram of one embodiment of the task interface of this application; Figure 3 This is a schematic diagram of the framework of an embodiment of the data detection device of this application; Figure 4 This is a schematic diagram of the framework of an embodiment of the electronic device of this application; Figure 5 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. Detailed Implementation

[0036] The embodiments of this application will now be described in detail with reference to the accompanying drawings.

[0037] In the following description, specific details such as particular system architectures, interfaces, and technologies are presented for illustrative purposes rather than for limiting purposes, in order to provide a thorough understanding of this application.

[0038] In this paper, the terms "system" and "network" are often used interchangeably. The term "and / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone. Additionally, the slash " / " generally indicates that the preceding and following related objects have an "or" relationship. Furthermore, "many" in this paper indicates two or more objects.

[0039] Please see Figure 1 , Figure 1 This is a flowchart illustrating an embodiment of the data detection method of this application. Specifically, it may include the following steps: Step S11: Obtain the multimedia data to be tested.

[0040] In one implementation scenario, the multimedia data to be tested can be collected by multimedia data acquisition devices such as cameras and mobile phones. For example, when there is a detection requirement at a target location (e.g., a pedestrian street, a station), multimedia data acquisition devices can be set up at the target location to collect multimedia data (e.g., image data, audio data, video data, etc.) as the multimedia data to be tested. It should be noted that when setting up multimedia data acquisition devices at a target location, a prominent sign (or other reminders such as voice prompts) can be displayed to indicate the presence of multimedia data acquisition devices, their purpose for collecting multimedia data, and their main function, such as conducting relevant data detection at the target location. This ensures that people are fully aware of the presence of multimedia data acquisition devices and the purpose of the collected multimedia data before or upon entering the target location.

[0041] In one implementation scenario, the multimedia data to be tested can be real-time online multimedia data, such as multimedia data from the target location that can be collected in real time; or, the multimedia data to be tested can be non-real-time offline multimedia data, such as multimedia data that can be captured at the target location for a period of time and then obtained as the multimedia data to be tested through network transmission, copying to storage media, etc. It should be noted that the above examples are only two possible ways to obtain the multimedia data to be tested, and other possible ways to obtain the multimedia data to be tested are not limited here, nor will they be listed one by one.

[0042] Step S12: Construct a multimodal prompt instruction based on the sample multimedia data related to the target event and the descriptive text of the sample multimedia data.

[0043] In an implementation scenario, the target event can be any relevant event that is expected to be detected. For example, in urban management, the target event could be something like "a motorcycle lifting its front wheel"; or, in industrial production, it could be something like "not wearing a helmet." The specific content of the target event is not limited here. It should be noted that although the target events exemplified above are all undesirable abnormal events, this does not mean that in practical applications, the target event must necessarily be an abnormal event. The target event can also be a normal event; the specific type of target event is not limited here either.

[0044] In an implementation scenario, sample multimedia data related to the target event can include positive example multimedia data of the target event. It should be noted that positive example multimedia data includes the target event. Taking the target event "motorcycle wheelies" in urban management as an example, the positive example multimedia data could be multimedia data showing "motorcycle wheelies"; or, taking the target event "not wearing a helmet" in industrial production as an example, the positive example multimedia data could be multimedia data showing "not wearing a helmet". Of course, the above examples are merely a few possible examples of positive example multimedia data in practical applications, and the specific content of positive example multimedia data will not be listed here.

[0045] In another implementation scenario, sample multimedia data related to the target event can also include counterexample multimedia data of the target event. It should be noted that counterexample multimedia data includes the opposite event of the target event. Taking the target event "motorcycle wheelies" in urban management as an example, the opposite event could be "motorcycle driving normally," in which case the counterexample multimedia data could show "motorcycle driving normally." Or, taking the target event "not wearing a helmet" in industrial production as an example, the opposite event could be "wearing a helmet," in which case the counterexample multimedia data could show "wearing a helmet." Of course, the above examples are merely a few possible examples of counterexample multimedia data in practical applications, and the specific content of counterexample multimedia data will not be listed here.

[0046] In another implementation scenario, the sample multimedia data related to the target event can also include positive and negative example multimedia data of the target event. It should be noted that the specific meanings of positive and negative example multimedia data can be found in the aforementioned descriptions, and will not be repeated here. Furthermore, when the sample multimedia data includes positive example multimedia data, the descriptive text of the sample multimedia data can include the descriptive text of the positive example multimedia data. For example, taking the target event "motorcycle wheelies" in urban management as an example, the descriptive text of the positive example multimedia data can include, but is not limited to, "motorcycle front wheel lifts off the ground." Similarly, when the sample multimedia data includes negative example multimedia data, the descriptive text of the sample multimedia data can include the descriptive text of the negative example multimedia data. For example, taking the target event "motorcycle wheelies" in urban management as an example, the descriptive text of the negative example multimedia data can include, but is not limited to, "motorcycle drives normally and smoothly." Of course, the above examples are merely a few possible examples of descriptive text in practical applications; the specific content of the descriptive text is not limited here, nor will it be listed in detail.

[0047] In one implementation scenario, the sample multimedia data and the multimedia data to be tested can belong to the same data type. It should be noted that the data type includes at least one of image, video, and audio. For example, both the sample multimedia data and the multimedia data to be tested can be image data; or, for another example, both can be video data; or, for yet another example, both can be audio data. The specific types of the sample multimedia data and the multimedia data to be tested are not limited here. In addition, the sample multimedia data and the multimedia data to be tested can also belong to different data types. For example, when sample multimedia data of the same data type as the multimedia data to be tested is lacking or difficult to obtain in a real-world scenario, sample multimedia data of a different data type can be obtained. For example, the sample multimedia data can be image data, and the multimedia data to be tested can be video data.

[0048] In one implementation scenario, to obtain sample multimedia data and its descriptive text, a configuration interface can be displayed. This interface can include a first control for configuring sample multimedia data and a second control for configuring descriptive text. Based on this, input multimedia data from the first control can be obtained as sample multimedia data, and input text from the second control can be obtained as descriptive text. It should be noted that the first control can include, but is not limited to, file upload controls, and the second control can include, but is not limited to, text boxes. The specific types of the first and second controls are not limited here, nor will they be listed individually. This method, by displaying a configuration interface containing a first control for configuring sample multimedia data and a second control for configuring descriptive text, and then obtaining input multimedia data from the first control as sample multimedia data and input text from the second control as descriptive text, only sample multimedia data and its descriptive text are needed during data detection, without uploading sensitive user data. This facilitates model fine-tuning through local configuration of prompts. In other words, by locally configuring sample multimedia data and its descriptive text related to the required detection events according to detection needs, the multimodal large model can be fine-tuned to meet detection requirements.

[0049] In a specific implementation scenario, during local configuration, it is also possible to customize the name of the multimodal large model that is paired with the aforementioned sample multimedia data and its descriptive text in the configuration interface. For example, still using the target event "motorcycle wheelie" as an example, when the multimodal large model is paired with the aforementioned sample multimedia data and its description, the large model name can be customized, such as "Motorcycle Wheelie Detection Model". Of course, when the multimodal large model is paired with other sample multimedia data and their descriptive text, the large model name can be customized to other names; this is not limited here, nor will it be listed in detail.

[0050] In a specific implementation scenario, during local configuration, it is also possible to customize the theme of the multimodal large model by configuring it in the configuration interface, which is paired with the aforementioned sample multimedia data and its descriptive text. For example, still taking the target event "motorcycle wheelies" as an example, when the multimodal large model is paired with the aforementioned sample multimedia data and its description, if it is applied to urban management, the large model theme can be customized as "General," or if it is applied to criminal investigation, the large model theme can be customized as "Criminal." Of course, the above examples are merely illustrative examples to facilitate understanding of the specific meaning of customized themes; other possible scenarios are not limited here, nor will they be listed in detail.

[0051] In another implementation scenario, as mentioned above, to obtain sample multimedia data and its descriptive text, a configuration interface can be displayed. This interface allows the acquisition of sample multimedia data and its descriptive text through relevant controls. Unlike the aforementioned implementation, as another possible implementation, the configuration interface can be divided into a first area for configuring positive samples and a second area for configuring negative samples. Both the first and second areas can contain first and second controls. Based on this, input multimedia data from the first control in the first area can be obtained as positive sample multimedia data, and input multimedia data from the first control in the second area can be obtained as negative sample multimedia data. It should be noted that positive sample multimedia data contains the target event, and negative sample multimedia data contains the opposite event of the target event. Furthermore, input text from the second control in the first area can be obtained as the descriptive text of the positive sample multimedia data, and input text from the second control in the second area can be obtained as the descriptive text of the negative sample multimedia data. It should be noted that the multimodal prompt instruction is used to instruct the multimodal large model to refer to the positive sample multimedia data and its descriptive text, as well as the negative sample multimedia data and its descriptive text, to detect whether the multimedia data under test contains the target event. The above method can support local configuration of positive example multimedia data and its descriptive text, as well as negative example multimedia data and its descriptive text, so that multimodal large models can refer to both positive and negative examples simultaneously, which helps to improve the accuracy of data detection.

[0052] In a specific implementation scenario, please refer to the relevant documents. Figure 2a , Figure 2a This is a schematic diagram of one embodiment of the configuration interface of this application. For example... Figure 2a As shown, Figure 2a The configuration interface is divided into two sections: a first area and a second area. Both sections contain a first control, the "File Upload Control." Triggering this control allows for the upload of custom sample multimedia data. For example, triggering the "File Upload Control" in the first area allows for the upload of a positive example of multimedia data; similarly, triggering the "File Upload Control" in the second area allows for the upload of a negative example of multimedia data.

[0053] In a specific implementation scenario, please refer to the following: Figure 2aBoth the first and second areas contain a second control, a "text box." By entering text into the second control's "text box," users can customize the description text of sample multimedia data. For example, entering text into the second control's "text box" in the first area allows for customizing the description text of positive example multimedia data; similarly, entering text into the second control's "text box" in the second area allows for customizing the description text of negative example multimedia data.

[0054] In this embodiment of the disclosure, the multimodal prompt instruction can be used to instruct the multimodal large model reference sample multimedia data and its descriptive text to detect whether the multimedia data to be tested contains the target event.

[0055] In one implementation scenario, as a possible example, the multimodal cue instruction can specifically be used to instruct the multimodal large model reference sample multimedia data and its descriptive text to detect the predicted probability that the multimedia data to be tested contains the target event.

[0056] In another implementation scenario, as a different example from the aforementioned implementation, the multimodal prompt instruction can specifically be used to indicate the first probability that the multimodal large model reference sample multimedia data detects that the multimedia data under test contains the target event, and to detect the second probability that the multimedia data under test contains the target event based on the descriptive text of the reference sample multimedia data.

[0057] In another implementation scenario, as another possible example, the multimodal prompt instruction can be specifically used to indicate the third probability that the multimodal large model references positive example multimedia data and its descriptive text to detect that the multimedia data under test contains the target event, and to refer to negative example multimedia data and its descriptive text to detect the fourth probability that the multimedia data under test does not contain the target event.

[0058] In another implementation scenario, as another possible example, the multimodal prompt instruction can also specifically instruct the multimodal large model reference sample multimedia data and its descriptive text to detect whether the multimedia data under test contains the target event and directly output "Yes (or True, Yes, etc.)" or "No (or False, No, etc.)".

[0059] In another implementation scenario, as another possible example, different from the aforementioned implementation, when the sample multimedia data contains multiple positive multimedia data and multiple negative multimedia data of the target event, the multimodal prompt instruction can be specifically used to instruct the multimodal large model to output a fifth probability that the multimedia data under test contains the target event by referring to each positive multimedia data and its description text, and to output a sixth probability that the multimedia data under test does not contain the target event by referring to each negative multimedia data and its description text.

[0060] It should be noted that the above examples are only a few possible examples of multimodal prompts. The specific content of multimodal prompts is not limited here, nor will they be listed one by one.

[0061] Step S13: Based on the output of the multimodal prompting command in response to the multimodal large model, obtain the detection result characterizing whether the multimedia data under test contains the target event.

[0062] It should be noted that the multimodal large model in this embodiment may include, but is not limited to, MoE-LLaVA, InternLM-XComposer2, CLIP, BLIP-2, DreamLLM, etc. The specific selection of multimodal large models is not limited here, nor will they be listed one by one.

[0063] In one implementation scenario, the aforementioned configuration interface may also include a third control for configuring event thresholds. Figure 2a (Not illustrated in the text) The input threshold from the third control can then be obtained as the event threshold. It should be noted that the output result can include the predicted probability that the tested multimedia data contains the target event. When the predicted probability meets the filtering condition of including the event threshold, the detection result can include that the tested multimedia data contains the target event; when the predicted probability does not meet the filtering condition, the detection result can include that the tested multimedia data does not contain the target event. This method supports custom configuration of the event threshold, combining the event threshold and the predicted probability output by the multimodal large model to determine the detection result, which helps meet different levels of data detection needs.

[0064] In a specific implementation scenario, the third control may include, but is not limited to, text boxes, drop-down lists, sliders, etc. The specific type of the third control is not limited here, nor will it be listed one by one.

[0065] In a specific implementation scenario, the multimodal cue instruction can be used to instruct the multimodal large model reference sample multimedia data and its descriptive text to detect the predicted probability that the multimedia data under test contains the target event. In this way, the output result of the multimodal large model responding to the multimodal cue instruction can be obtained, and the output result can contain the predicted probability that the multimedia data under test contains the target event.

[0066] In a specific implementation scenario, filtering conditions containing event thresholds may include, but are not limited to, prediction probabilities higher than the event threshold, prediction probabilities not lower than the event threshold, etc. No specific limitations are imposed on the filtering conditions here. In practical applications, the specific settings of the filtering conditions should be such that the prediction probabilities can accurately reflect whether the multimedia data under test contains the target event through the event threshold; further examples will not be provided here.

[0067] In another implementation scenario, unlike the aforementioned implementation, as another possible implementation, the configuration interface may be divided into a third area for configuring event thresholds from a reference perspective of sample multimedia data. Figure 2a (not shown in the image) and a fourth region for configuring event thresholds from the perspective of the descriptive text (not shown in the image) Figure 2a (Not illustrated in the text), and both the third and fourth regions can contain third controls for configuring event thresholds. Based on this, an input threshold from the third control in the third region can be obtained as a first value, and an input threshold from the third control in the fourth region can be obtained as a second threshold. It should be noted that in this scenario, the event threshold can include both the first and second thresholds, and the predicted probability can include: a first probability that the multimodal large model predicts, from the reference perspective of the sample multimedia data, that the tested multimedia data contains the target event, and a second probability that the multimodal large model predicts, from the reference perspective of the descriptive text, that the tested multimedia data contains the target event. That is, in this scenario, the multimodal prompt instruction can specifically be used to instruct the multimodal large model to detect the first probability that the tested multimedia data contains the target event based on the sample multimedia data and to detect the second probability that the tested multimedia data contains the target event based on the descriptive text of the sample multimedia data. Furthermore, when the first probability satisfies the first condition containing the first threshold and the second probability satisfies the second condition containing the second threshold, the detection result can include that the tested multimedia data contains the target event; when the first probability does not satisfy the first condition and / or the second probability does not satisfy the second condition, the detection result can include that the tested multimedia data does not contain the target event. For example, the first condition may include, but is not limited to, a first probability higher than a first threshold, a first probability not lower than a first threshold, etc., and the second condition may include, but is not limited to, a second probability higher than a second threshold, a second probability not lower than a second threshold, etc. Of course, the above examples are merely a few possible examples of the first and second conditions; the specific content of the first and second conditions is not limited here, nor will they be listed one by one. The above method allows for custom configuration of corresponding thresholds from both the perspective of the descriptive text and the perspective of the sample multimedia data, helping to further refine the satisfaction of different levels of data detection needs.

[0068] In another implementation scenario, distinct from the aforementioned implementation methods, as another possible implementation, as described above, the sample multimedia data includes both positive and negative examples of the target event. The multimodal cue instruction can specifically instruct the multimodal large model to detect the third probability that the tested multimedia data contains the target event by referring to the positive multimedia data and its descriptive text, and to detect the fourth probability that the tested multimedia data does not contain the target event by referring to the negative multimedia data and its descriptive text. Based on this, the output of the multimodal large model in response to the multimodal cue instruction can include the third and fourth probabilities. Therefore, based on the comparison of the third and fourth probabilities, a detection result characterizing whether the tested multimedia data contains the target event can be obtained. For example, if the third probability is not lower than the fourth probability, it can be determined that the detection result includes the tested multimedia data containing the target event; conversely, if the third probability is lower than the fourth probability, it can be determined that the detection result includes the tested multimedia data not containing the target event.

[0069] In another implementation scenario, distinct from the aforementioned implementation methods, as another possible approach, the sample multimedia data can include multiple positive and negative examples of the target event. The multimodal cue instruction can specifically instruct the multimodal large model to output a fifth probability that the tested multimedia data contains the target event, referring to each positive example multimedia data and its description text, and to output a sixth probability that the tested multimedia data does not contain the target event, referring to each negative example multimedia data and its description text. In other words, the output of the multimodal large model in response to the multimodal cue instruction can include a fifth probability corresponding to each positive example multimedia data and its description text, and a sixth probability corresponding to each negative example multimedia data and its description text. Based on this, for each positive example multimedia data and its descriptive text, the maximum fifth probability exceeding the event threshold can be selected. Similarly, for each negative example multimedia data and its description, the maximum sixth probability exceeding the event threshold can be selected. If the aforementioned maximum fifth probability exceeding the event threshold is not lower than the aforementioned maximum sixth probability exceeding the event threshold, it can be determined that the detection result includes the multimedia data under test containing the target event. Conversely, if the aforementioned maximum fifth probability exceeding the event threshold is lower than the aforementioned maximum sixth probability exceeding the event threshold, it can be determined that the detection result includes the multimedia data under test not containing the target event.

[0070] It should be noted that the above-described implementation methods are merely examples of determining detection results based on output results in practical applications. Other possible implementation methods are not limited here. For instance, the multimodal prompting command could also instruct the multimodal large model reference sample multimedia data and its descriptive text to detect whether the multimedia data to be tested contains the target event and directly output "Yes (or True, Yes, etc.)" or "No (or False, No, etc.)". In this case, the detection result can be determined directly based on whether the multimodal large model's output result is "Yes (or True, Yes, etc.)" or "No (or False, No, etc.)". Other possible implementation methods will not be listed here.

[0071] In one implementation scenario, after obtaining the detection results, it can be determined whether to display the multimedia data to be tested on the retrieval interface of the target event, based on the detection results of the multimedia data to be tested. Furthermore, based on the annotation results of the target object on the multimedia data to be tested in the retrieval interface, it can be determined whether to select the multimedia data to be tested as sample multimedia data for fine-tuning training of the multimodal large model. It should be noted that the annotation results can characterize whether the detection results of the multimedia data to be tested are erroneous. The above method supports custom annotation of the detected multimedia data to be tested, and combines the annotation results to select a portion of the multimedia data to be tested for fine-tuning training of the multimodal large model, forming a dynamic iterative closed loop of "annotation-training-optimization," which helps to improve the detection capability of the multimodal large model.

[0072] In a specific implementation scenario, real-time multimedia data can be detected to determine whether it contains objects related to the target event. If so, multimedia data segments containing these objects can be selected as the multimedia data to be tested. As a possible example, taking the target event "motorcycle wheelies," the related objects could include "motorcycle." Therefore, real-time multimedia data (such as real-time video streams) can be detected to determine whether it contains "motorcycle." If so, multimedia data segments containing "motorcycle" (e.g., video frames containing the related object "motorcycle") can be selected as the multimedia data to be tested. Of course, the above example is merely one possible application; other possible scenarios are not limited here, nor will they be listed individually. It should be noted that when the real-time multimedia data is visual data such as real-time video streams, CV (computer vision) models can be used to detect multimedia data segments containing related objects as the multimedia data to be tested. CV models can include, but are not limited to, convolutional neural networks, etc. The model structure of CV models is not limited here, nor will it be listed individually.

[0073] In a specific implementation scenario, when the detection results indicate that the multimedia data under test contains the target event, the search interface can display the multimedia data under test. Furthermore, as a possible example, when the detection results indicate that the multimedia data under test does not contain the target event, the search interface may not display the multimedia data under test.

[0074] In a specific implementation scenario, please refer to the relevant documents. Figure 2b , Figure 2b This is a schematic diagram of one embodiment of the search interface of this application. For example... Figure 2b As shown, each piece of multimedia data to be tested displayed on the search interface has "Correct" and "Incorrect" annotation controls. This allows the annotation result of the target object on the multimedia data to be determined based on the target object's trigger command to the annotation control. For example, responding to the target object's trigger command to the "Correct" annotation control, it can be determined that the target object's annotation result on the multimedia data to be tested is correct; conversely, responding to the target object's trigger command to the "Incorrect" annotation control, it can be determined that the target object's annotation result on the multimedia data to be tested is incorrect. Of course, Figure 2b The example shown is merely one possible implementation of annotation; other possible annotation methods are not limited here, nor will they be listed one by one. Furthermore, to improve annotation efficiency, batch annotation can be triggered in response to the target object. If the target object triggers the annotation control to "correct" for the selected multimedia data to be tested, it confirms that the annotation results for the selected multimedia data are all correct; conversely, if the target object triggers the annotation control to "incorrect" for the selected multimedia data to be tested, it confirms that the annotation results for the selected multimedia data are all incorrect.

[0075] In a specific implementation scenario, if the annotation results indicate an error in the detection of the multimedia data under test, the multimedia data under test can be selected as sample multimedia data. Furthermore, as a possible example, if the annotation results indicate an accurate detection of the multimedia data under test, the multimedia data under test may not be selected as sample multimedia data.

[0076] In a specific implementation scenario, after annotation is completed, a fine-tuning training task can be automatically created. Alternatively, a fine-tuning training task can be created only when the target object triggers a control in the search interface that indicates the start of fine-tuning training (e.g., a virtual button labeled "Model Training"). It should be noted that the fine-tuning training task information can at least include the main model name and a related main model hint instruction. The main model name defaults to the search text of the target event in the search interface (e.g., ...). Figure 2bIn the example of "motorcycle lifting its front wheel" (a specific action), the default prompt for the large model is a multimodal prompt. Furthermore, task information may also include the aforementioned event thresholds, which are not limited here. Based on this, fine-tuning training can begin using the task information and sample multimedia data. It should be noted that during fine-tuning training, parameters of the multimodal large model can be fine-tuned. In addition, as a possible example, if the task information also includes event thresholds, the event thresholds can also be adjusted and optimized. As another possible example, the training status of the multimodal large model during fine-tuning training can also be queried through the interactive interface with the target object.

[0077] In one implementation scenario, after fine-tuning the training, a task interface can be displayed, which can show a fourth control. This fourth control can be used to configure the target event detection task. Based on this, input information from the fourth control can be obtained as configuration information for the detection task. The detection task can then be started based on this configuration information to detect whether new multimedia data to be tested contains the target event using the fine-tuned multimodal large model. It should be noted that the configuration information may include, but is not limited to: the target region to be monitored for the target event, and multimodal prompts for detecting whether new multimedia data to be tested from the target region contains the target event. Thus, the multimodal large model can detect whether new multimedia data to be tested from the target region contains the target event under the guidance of the multimodal instructions in the configuration information. This method supports custom configuration of the detection task, and starting data detection in conjunction with the custom configuration information helps in detecting whether a target event has occurred in new multimedia data to be tested from the target region.

[0078] In a specific implementation scenario, please refer to the relevant documents. Figure 2c , Figure 2c This is a schematic diagram of one embodiment of the task interface of this application. For example... Figure 2c As shown, the task interface may include, but is not limited to, different types of fourth controls such as text boxes, checkboxes, and drop-down lists, to obtain configuration information for the detection task. For example, input from a text box can yield information such as the task name and the event name of the target event; input from a checkbox can yield information such as the effective time; and input from a drop-down list can yield information such as the monitoring area (i.e., the target area where the target event needs to be monitored). It should be noted that... Figure 2cThe examples shown are merely illustrative of some possible ways to obtain input information. Other possible scenarios are not limited here, nor will they be listed one by one. For example, from the input information in the drop-down list, frame skipping strategies such as high frame skipping can also be obtained as configuration information; as another example, from the input information in the drop-down list, alarm levels such as blue alarms can also be obtained as configuration information; and as yet another example, from the input information in the checkbox, target types such as human bodies and their appendages, motor vehicles and their appendages, and non-motor vehicles and their appendages can also be obtained as configuration information. In addition, it should be noted that... Figure 2c The "Target Description" can be configured to include the aforementioned positive example multimedia data and its descriptive text. Figure 2c The "filter description" can be configured to include the aforementioned counterexample multimedia data and its descriptive text, combining both to obtain the multimodal prompt instructions in the configuration information. Of course, Figure 2c The layout shown is just one possible way to display the task interface. We will not limit the layout of the task interface here. For example, we can also use tabs to display basic information, event information and other information in the configuration information in the task interface. We will not give examples of these here.

[0079] In a specific implementation scenario, in addition to the configuration information mentioned above, as one possible example, an alarm cooldown time (e.g., 5 minutes) can be further configured to prevent repeated alarms for the same event (e.g., a car turning up) on the same object (e.g., a vehicle) within the alarm cooldown time. Besides the configuration information mentioned above, as another possible example, permission allocation can be further configured to complete the setting of the target event detection task. After completing the configuration, the detection task can be started. The multimodal large model, after fine-tuning and training, can detect new multimedia data from the target area under the instructions of the multimodal prompts in the configuration information to determine whether it contains the target event, and issue an alarm if it does.

[0080] In a specific implementation scenario, after fine-tuning training, when the multimodal large model processes new test multimedia data, in response to detecting a false positive from the multimodal large model indicating that the new test multimedia data contains a target event, at least one of the sample multimedia data and the descriptive text in the multimodal prompt instruction can be adjusted based on the falsely reported new test multimedia data containing the target event to obtain a new multimodal prompt instruction. Based on this new multimodal prompt instruction, the multimodal large model can then restart fine-tuning training. It should be noted that if the new test multimedia data does not actually contain the target event, but the multimodal large model detects that it does, then it can be considered a false positive. As a possible example, when the multimodal large model falsely reports that the new test multimedia data contains a target event, at least one of the sample multimedia data and the descriptive text in the multimodal prompt instruction can be adjusted based on the similarity between the falsely reported new test multimedia data containing the target event and the target event. For example, when adjusting sample multimedia data and its descriptive text, one can adjust positive example multimedia data and its descriptive text, or one can adjust negative example multimedia data and its descriptive text, or one can adjust both positive example multimedia data and its descriptive text and negative example multimedia data and their descriptive text. For ease of understanding, let's take the target event "motorcycle lifting its nose" as an example. In the new test multimedia data, the motorcycle is in an uphill state. At this point, the new test multimedia data is misreported by the multimodal large model as containing the target event "motorcycle lifting its nose". In this case, the negative example multimedia data and its descriptive text can be adjusted. For example, a new negative example multimedia data showing "motorcycle normally going uphill" and its descriptive text "motorcycle going uphill" can be added. This way, when the multimodal large model detects the target event, it can exclude this misreported scenario as much as possible, that is, avoid misreporting it as containing the target event again as much as possible. In other words, when adjusting multimodal prompts based on new test multimedia data that is falsely reported to contain the target event, one possible example is to treat the falsely reported new test multimedia data and its description as new negative example multimedia data and its description, and update the multimodal prompts based on this new negative example multimedia data and its description. Similarly, if the large multimodal model misses reporting new test multimedia data containing the target event, the multimodal prompts can also be adjusted based on the missed new test multimedia data containing the target event. Specifically, the missed new test multimedia data and its description can be treated as new positive example multimedia data and its description, and the multimodal prompts can be updated based on this new positive example multimedia data and its description. This approach allows for adjusting multimodal prompts and restarting fine-tuning training when a false alarm scenario is detected, without redeploying the entire model.

[0081] The above scheme acquires the multimedia data to be tested, and then constructs a multimodal cue instruction based on sample multimedia data and descriptive text related to the target event. This multimodal cue instruction instructs the multimodal large model to detect whether the multimedia data to be tested contains the target event, referencing the sample multimedia data and descriptive text. Based on the output of the multimodal large model in response to the multimodal cue instruction, a detection result representing whether the multimedia data to be tested contains the target event is obtained. Therefore, on the one hand, the multimodal large model can detect whether the multimedia data to be tested contains the target event under the instruction of the multimodal cue instruction; on the other hand, when there is a need to detect other events besides the target event, the sample multimedia data and its descriptive text are adaptively adjusted according to the other events. Describing text to reconstruct multimodal prompts enables large multimodal models to detect events beyond the target event, thus expanding the applicability of data detection. Furthermore, thanks to the general knowledge representation capabilities of the large multimodal model itself, it exhibits strong generalization reasoning abilities even in zero-shot or small-shot scenarios. Compared to traditional deep learning models, this helps minimize the need for supervised data. Moreover, unlike traditional deep learning models that require full-scale retraining for task adaptation, the large multimodal model achieves task adaptation through dynamic adjustment of multimodal instructions, improving the responsiveness of data detection. Therefore, it expands the applicability of data detection, minimizes the need for supervised data, and enhances the responsiveness of data detection.

[0082] Please see Figure 3 , Figure 3 This is a schematic diagram of the framework of an embodiment of the data detection device of this application. The data detection device 30 includes: an acquisition module 31, a construction module 32, and a processing module 33. The acquisition module 31 is used to acquire the multimedia data to be tested; the construction module 32 is used to construct a multimodal prompt instruction based on sample multimedia data and descriptive text of the sample multimedia data related to the target event; wherein, the multimodal prompt instruction is used to instruct the multimodal large model reference sample multimedia data and descriptive text to detect whether the multimedia data to be tested contains the target event; the processing module 33 is used to obtain a detection result characterizing whether the multimedia data to be tested contains the target event based on the output result of the multimodal large model responding to the multimodal prompt instruction.

[0083] In the above scheme, the data detection device 30 acquires the multimedia data to be tested, and then constructs a multimodal prompt instruction based on the sample multimedia data and the descriptive text of the sample multimedia data related to the target event. The multimodal prompt instruction is used to instruct the multimodal large model to detect whether the multimedia data to be tested contains the target event by referring to the sample multimedia data and the descriptive text. Then, based on the output result of the multimodal large model responding to the multimodal prompt instruction, a detection result characterizing whether the multimedia data to be tested contains the target event is obtained. Therefore, on the one hand, the multimodal large model can detect whether the multimedia data to be tested contains the target event under the instruction of the multimodal prompt instruction. On the other hand, when there is a need to detect other events besides the target event, the sample multimedia data can be adaptively adjusted according to the other events. By reconstructing multimodal prompts using data and its descriptive text, the multimodal large model can support the detection of events beyond the target event, thus expanding the applicability of data detection. Furthermore, thanks to the multimodal large model's inherent general knowledge representation capabilities, it exhibits strong generalization reasoning abilities even in zero-shot or small-shot scenarios. Compared to traditional deep learning models, this helps minimize the need for supervised data. Moreover, unlike traditional deep learning models that require retraining with full data sets to achieve task adaptation, the multimodal large model achieves task adaptation through dynamic adjustment of multimodal instructions, improving the responsiveness of data detection. Therefore, it expands the applicability of data detection, minimizes the need for supervised data, and enhances the responsiveness of data detection.

[0084] In some disclosed embodiments, the acquisition module 31 includes a configuration display submodule for displaying a configuration interface; wherein the configuration interface includes a first control for configuring sample multimedia data and a second control for configuring descriptive text; the acquisition module 31 includes a data acquisition submodule for acquiring input multimedia data from the first control as sample multimedia data, and the acquisition module 31 includes a description acquisition submodule for acquiring input text from the second control as descriptive text.

[0085] In some disclosed embodiments, the configuration interface is divided into a first area for configuring positive examples and a second area for configuring negative examples. Both the first and second areas contain a first control and a second control. The data acquisition submodule is specifically used to acquire input multimedia data from the first control in the first area as positive example multimedia data, and to acquire input multimedia data from the first control in the second area as negative example multimedia data. The positive example multimedia data contains a target event, and the negative example multimedia data contains the opposite event of the target event. The description acquisition submodule is specifically used to acquire input text from the second control in the first area as descriptive text of the positive example multimedia data, and to acquire input text from the second control in the second area as descriptive text of the negative example multimedia data. The multimodal prompt instruction is used to instruct the multimodal large model reference positive example multimedia data and its descriptive text and negative example multimedia data and its descriptive text to detect whether the multimedia data under test contains the target event.

[0086] In some disclosed embodiments, the configuration interface further includes a third control for configuring event thresholds, and the acquisition module 31 includes a threshold acquisition submodule for acquiring the input threshold from the third control as the event threshold; wherein, the output result includes the predicted probability that the multimedia data to be tested contains the target event, and when the predicted probability meets the filtering condition of containing the event threshold, the detection result includes that the multimedia data to be tested contains the target event, and when the predicted probability does not meet the filtering condition, the detection result includes that the multimedia data to be tested does not contain the target event.

[0087] In some disclosed embodiments, the configuration interface is divided into a third region for configuring event thresholds from the reference perspective of sample multimedia data and a fourth region for configuring event thresholds from the reference perspective of descriptive text. Both the third and fourth regions contain third controls. The threshold acquisition submodule is specifically used to acquire the input threshold from the third control in the third region as the first threshold and to acquire the input threshold from the third control in the fourth region as the second threshold. The event threshold includes the first threshold and the second threshold. The prediction probability includes: a first probability that the multimodal large model predicts from the reference perspective of sample multimedia data that the multimedia data to be tested contains the target event, and a second probability that the multimodal large model predicts from the reference perspective of descriptive text that the multimedia data to be tested contains the target event. When the first probability satisfies the first condition containing the first threshold and the second probability satisfies the second condition containing the second threshold, the detection result includes that the multimedia data to be tested contains the target event. When the first probability does not satisfy the first condition and / or the second probability does not satisfy the second condition, the detection result includes that the multimedia data to be tested does not contain the target event.

[0088] In some disclosed embodiments, the data detection device 30 includes a display module for determining whether to display the multimedia data to be tested in the retrieval interface of the target event based on the detection results of the multimedia data to be tested; the data detection device 30 includes a filtering module for determining whether to filter the multimedia data to be tested as sample multimedia data for fine-tuning training of a multimodal large model based on the annotation results of the target object on the multimedia data to be tested in the retrieval interface; wherein, the annotation results characterize whether the detection results of the multimedia data to be tested are erroneous.

[0089] In some disclosed embodiments, when the detection result indicates that the multimedia data to be tested contains the target event, the retrieval interface displays the multimedia data to be tested; and / or, when the annotation result indicates that the detection result of the multimedia data to be tested is incorrect, the multimedia data to be tested is selected as sample multimedia data; and / or, the task information for fine-tuning training includes at least the name of the large model and a large model prompt instruction associated with the name of the large model, wherein the name of the large model is the retrieval text of the target event in the retrieval interface by default, and the large model prompt instruction is the multimodal prompt instruction by default.

[0090] In some disclosed embodiments, the data detection device 30 includes a display module for displaying a task interface; wherein the task interface displays a fourth control, which is used to configure the detection task of the target event; the data detection device 30 includes a configuration module for acquiring input information from the fourth control as configuration information for the detection task; wherein the configuration information includes: the target area to be monitored for the target event, and a multimodal prompt instruction for detecting whether new multimedia data to be tested from the target area contains the target event; the data detection device 30 includes a startup module for starting the detection task based on the configuration information, so as to detect whether new multimedia data to be tested contains the target event through a multimodal large model after fine-tuning training.

[0091] In some disclosed embodiments, the startup module includes an instruction adjustment submodule, which, in response to detecting a new test multimedia data containing a target event that is falsely reported by the multimodal large model, adjusts at least one of the sample multimedia data and the descriptive text in the multimodal prompt instruction based on the new test multimedia data that is falsely reported to contain the target event, to obtain a new multimodal prompt instruction, and restarts fine-tuning training of the multimodal large model based on the new multimodal prompt instruction.

[0092] In some disclosed embodiments, the sample multimedia data includes at least one of positive and negative multimedia data of the target event, the positive multimedia data includes the target event, and the negative multimedia data includes the opposite event of the target event. When the sample multimedia data includes positive multimedia data, the description text of the sample multimedia data includes the description text of the positive multimedia data. When the sample multimedia data includes negative multimedia data, the description text of the sample multimedia data includes the description text of the negative multimedia data. And / or, the sample multimedia data and the multimedia data to be tested belong to the same data type, and the data type includes at least one of image, video, and audio.

[0093] Please see Figure 4 , Figure 4 This is a schematic diagram of a framework of an embodiment of the electronic device of this application. The electronic device 40 includes at least a memory 41 and a processor 42 coupled to each other. The memory 41 stores at least program instructions, and the processor 42 is used to execute the program instructions to implement the steps in any of the above-described data detection method embodiments. For details, please refer to the foregoing disclosed embodiments, which will not be repeated here. As a possible example, the electronic device 40 may include, but is not limited to, devices such as tablet computers, mobile phones, industrial control computers, and servers. The specific type of the electronic device 40 is not limited here.

[0094] Specifically, processor 42 controls itself and memory 41 to implement the steps in any of the above-described data detection method embodiments. Processor 42 can also be referred to as a CPU (Central Processing Unit). Processor 42 may be an integrated circuit chip with signal processing capabilities. Processor 42 can also be a general-purpose processor, digital signal processor (DSP), application-specific integrated circuit (ASIC), field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. A general-purpose processor can be a microprocessor or any conventional processor. Furthermore, processor 42 can be implemented using integrated circuit chips.

[0095] In the above scheme, the electronic device 40 acquires the multimedia data to be tested, and then constructs a multimodal prompt instruction based on the sample multimedia data and the descriptive text of the sample multimedia data related to the target event. The multimodal prompt instruction is used to instruct the multimodal large model to detect whether the multimedia data to be tested contains the target event by referring to the sample multimedia data and the descriptive text. Then, based on the output result of the multimodal large model responding to the multimodal prompt instruction, a detection result characterizing whether the multimedia data to be tested contains the target event is obtained. Therefore, on the one hand, the multimodal large model can detect whether the multimedia data to be tested contains the target event under the instruction of the multimodal prompt instruction. On the other hand, when there is a need to detect other events besides the target event, the sample multimedia data is adaptively adjusted according to the other events. Reconstructing multimodal prompts using the data and its descriptive text enables large multimodal models to detect events beyond the target event, thus expanding the applicability of data detection. Furthermore, thanks to the general knowledge representation capabilities of the large multimodal model itself, it exhibits strong generalization reasoning abilities even in zero-shot or small-shot scenarios. Compared to traditional deep learning models, this helps minimize the need for supervised data. Moreover, unlike traditional deep learning models that require retraining with full data sets to achieve task adaptation, the large multimodal model achieves task adaptation through dynamic adjustment of multimodal instructions, improving the responsiveness of data detection. Therefore, it expands the applicability of data detection, minimizes the need for supervised data, and enhances the responsiveness of data detection.

[0096] Please see Figure 5 , Figure 5 This is a schematic diagram of a framework of an embodiment of the computer-readable storage medium of this application. The computer-readable storage medium 50 stores program instructions 51 that can be executed by a processor. The program instructions 51 are used to implement the steps in any of the above-described data detection method embodiments.

[0097] In the above scheme, the computer-readable storage medium 50 acquires the multimedia data to be tested, and then constructs a multimodal cue instruction based on the sample multimedia data and the descriptive text of the sample multimedia data related to the target event. The multimodal cue instruction is used to instruct the multimodal large model to detect whether the multimedia data to be tested contains the target event by referring to the sample multimedia data and the descriptive text. Then, based on the output result of the multimodal large model responding to the multimodal cue instruction, a detection result characterizing whether the multimedia data to be tested contains the target event is obtained. Therefore, on the one hand, the multimodal large model can detect whether the multimedia data to be tested contains the target event under the instruction of the multimodal cue instruction. On the other hand, when there is a need to detect other events besides the target event, the sample multimedia data is adaptively adjusted according to the other events. By reconstructing multimodal prompts using body data and its descriptive text, the multimodal large model can support the detection of events other than the target event, thus expanding the applicability of data detection. Furthermore, thanks to the multimodal large model's inherent general knowledge representation capabilities, it exhibits strong generalization reasoning abilities even in zero-shot or small-shot scenarios. Compared to traditional deep learning models, this helps minimize the need for supervised data. Moreover, compared to traditional deep learning models that require retraining with full data sets to achieve task adaptation, the multimodal large model achieves task adaptation through dynamic adjustment of multimodal instructions, improving the responsiveness of data detection. Therefore, it expands the applicability of data detection, minimizes the need for supervised data, and enhances the responsiveness of data detection.

[0098] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0099] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0100] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus implementations described above are merely illustrative. For instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.

[0101] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0102] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0103] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) or processor to execute all or part of the steps of the methods of various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0104] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

Claims

1. A data detection method, characterized in that, include: Acquire the multimedia data to be tested; Based on sample multimedia data related to the target event and the descriptive text of the sample multimedia data, a multimodal prompt instruction is constructed; The multimodal prompt instruction is used to instruct the multimodal large model to refer to the sample multimedia data and the descriptive text to detect whether the multimedia data to be tested contains the target event; Based on the output of the multimodal large model in response to the multimodal prompting command, a detection result characterizing whether the multimedia data under test contains the target event is obtained.

2. The method according to claim 1, characterized in that, Before constructing the multimodal prompt instruction based on sample multimedia data related to the target event and the descriptive text of the sample multimedia data, the method further includes: Display configuration interface; wherein, the configuration interface includes a first control for configuring the sample multimedia data and a second control for configuring the description text; The input multimedia data from the first control is obtained as the sample multimedia data, and the input text from the second control is obtained as the description text.

3. The method according to claim 2, characterized in that, The configuration interface is divided into a first area for configuring positive examples and a second area for configuring negative examples. Both the first and second areas contain the first control and the second control, respectively. The step of obtaining input multimedia data from the first control as the example multimedia data includes: The system acquires input multimedia data from the first control in the first area as positive multimedia data, and acquires input multimedia data from the first control in the second area as negative multimedia data; wherein the positive multimedia data includes the target event, and the negative multimedia data includes the opposite event of the target event; The step of obtaining the input text from the second control as the description text includes: The input text from the second control in the first area is obtained as the description text of the positive example multimedia data, and the input text from the second control in the second area is obtained as the description text of the negative example multimedia data; wherein, the multimodal prompt instruction is used to instruct the multimodal large model to refer to the positive example multimedia data and the description text of the positive example multimedia data and the description text of the negative example multimedia data to detect whether the multimedia data to be tested contains the target event.

4. The method according to claim 2, characterized in that, The configuration interface also includes a third control for configuring event thresholds, and the method further includes: Obtain the input threshold from the third control and use it as the event threshold; The output result includes the predicted probability that the multimedia data under test contains the target event. When the predicted probability meets the filtering condition of containing the event threshold, the detection result includes that the multimedia data under test contains the target event. When the predicted probability does not meet the filtering condition, the detection result includes that the multimedia data under test does not contain the target event.

5. The method according to claim 4, characterized in that, The configuration interface is divided into a third area for configuring the event threshold from the reference perspective of the sample multimedia data and a fourth area for configuring the event threshold from the reference perspective of the description text. Both the third and fourth areas contain the third control. The step of obtaining the input threshold from the third control as the event threshold includes: The input threshold from the third control in the third region is obtained as the first threshold, and the input threshold from the third control in the fourth region is obtained as the second threshold; Wherein, the event threshold includes a first threshold and a second threshold, and the prediction probability includes: the multimodal large model predicts a first probability that the multimedia data to be tested contains the target event from the reference perspective of the sample multimedia data; the multimodal large model predicts a second probability that the multimedia data to be tested contains the target event from the reference perspective of the description text; when the first probability satisfies a first condition containing the first threshold and the second probability satisfies a second condition containing the second threshold, the detection result includes that the multimedia data to be tested contains the target event; when the first probability does not satisfy the first condition and / or the second probability does not satisfy the second condition, the detection result includes that the multimedia data to be tested does not contain the target event.

6. The method according to any one of claims 1 to 5, characterized in that, After obtaining a detection result characterizing whether the multimedia data under test contains the target event based on the output result of the multimodal large model responding to the multimodal cue instruction, the method further includes: Based on the detection results of the multimedia data to be tested, determine whether to display the multimedia data to be tested on the retrieval interface of the target event; Based on the annotation results of the target object on the multimedia data to be tested in the search interface, it is determined whether to screen the multimedia data to be tested as sample multimedia data for fine-tuning training of the multimodal large model; wherein, the annotation results characterize whether the detection results of the multimedia data to be tested are erroneous.

7. The method according to claim 6, characterized in that, When the detection result indicates that the multimedia data to be tested contains the target event, the search interface displays the multimedia data to be tested; And / or, when the annotation result indicates that the detection result of the multimedia data to be tested is incorrect, the multimedia data to be tested is selected as the sample multimedia data; And / or, the task information for fine-tuning training includes at least the name of the large model and a large model hint instruction associated with the name of the large model, wherein the name of the large model is by default the search text of the target event in the search interface, and the large model hint instruction is by default the multimodal hint instruction.

8. The method according to claim 6, characterized in that, Following the fine-tuning training, the method further includes: The task interface is displayed; wherein, the task interface displays a fourth control, which is used to configure the detection task of the target event; Input information from the fourth control is obtained as configuration information for the detection task; wherein, the configuration information includes: the target area to be monitored for the target event, and a multimodal prompt instruction for detecting whether new multimedia data to be tested from the target area contains the target event; The detection task is initiated based on the configuration information, so as to detect whether the new multimedia data to be tested contains the target event through the multimodal large model after fine-tuning training.

9. The method according to claim 6, characterized in that, Following the fine-tuning training, during the process of the multimodal large model processing new multimedia data to be tested, the method further includes: In response to the detection that the new test multimedia data contains the target event, the multimodal large model is falsely reported to contain the target event. Based on the new test multimedia data that is falsely reported to contain the target event, at least one of the sample multimedia data and the description text in the multimodal prompt instruction is adjusted to obtain a new multimodal prompt instruction. Based on the new multimodal prompt instruction, the fine-tuning training of the multimodal large model is restarted.

10. The method according to any one of claims 1 to 9, characterized in that, The sample multimedia data includes at least one of positive and negative multimedia data of the target event, wherein the positive multimedia data includes the target event, the negative multimedia data includes the opposite event of the target event, and when the sample multimedia data includes the positive multimedia data, the description text of the sample multimedia data includes the description text of the positive multimedia data; when the sample multimedia data includes the negative multimedia data, the description text of the sample multimedia data includes the description text of the negative multimedia data. And / or, the sample multimedia data and the multimedia data to be tested belong to the same data type, and the data type includes at least one of image, video, and audio.

11. A data detection device, characterized in that, include: The acquisition module is used to acquire the multimedia data to be tested. A construction module is used to construct multimodal prompt instructions based on sample multimedia data related to the target event and the descriptive text of the sample multimedia data; The multimodal prompt instruction is used to instruct the multimodal large model to refer to the sample multimedia data and the descriptive text to detect whether the multimedia data to be tested contains the target event; The processing module is used to obtain a detection result characterizing whether the multimedia data under test contains the target event based on the output result of the multimodal prompting instruction in response to the multimodal large model.

12. An electronic device, characterized in that, It includes at least a memory and a processor coupled to each other, wherein the memory stores at least program instructions, and the processor is used to execute the program instructions to implement the data detection method according to any one of claims 1 to 10.

13. A computer-readable storage medium, characterized in that, The device stores program instructions that can be executed by a processor, the program instructions being used to implement the data detection method according to any one of claims 1 to 10.

Images