Multi-modal data processing method and device, equipment and storage medium

Abstract

The invention provides a multi-modal data processing method and device, equipment and a storage medium, and relates to the technical field of data processing, in particular to the technical field of artificial intelligence, large models and the like. According to the specific implementation scheme, first multi-modal data sent by a target client side are obtained, wherein the first multi-modal data are multi-modal data which are sent by the target client side and aim at a target scene; based on the target historical feedback data, adjusting the initial scene rule to obtain a target scene rule for the target client; wherein the target historical feedback data is data associated with a scene rule; the multi-modal features of the first multi-modal data are matched with the rule features of the target scene rule to obtain a matching result, and the matching result represents whether a picture which does not meet the target scene rule exists in the target scene or not.

Description

Technical Field

[0001] The present disclosure relates to the field of data processing technology, and in particular to the technical fields of artificial intelligence and large models. Background Art

[0002] Surveillance cameras are widely used in homes, businesses, education, medical care, and communities to monitor and store images in real time. With the development of the intelligent era, people have higher requirements for the accuracy of monitoring alarms. For example, in terms of home safety, monitoring whether babies or the elderly at home have dangerous behaviors and other scene requirements, the accuracy of traditional monitoring and personalized monitoring requirements cannot meet current user needs. Summary of the invention

[0003] The present disclosure provides a multimodal data processing method, apparatus, device and storage medium.

[0004] According to one aspect of the present disclosure, a multimodal data processing method is provided, comprising:

[0005] Acquire first multimodal data sent by a target client, where the first multimodal data is multimodal data sent by the target client for a target scenario;

[0006] Based on the target historical feedback data, the initial scenario rules are adjusted to obtain the target scenario rules for the target client; wherein the target historical feedback data is data associated with the scenario rules;

[0007] The multimodal features of the first multimodal data are matched with the rule features of the target scenario rule to obtain a matching result, wherein the matching result indicates whether there is a picture in the target scene that does not satisfy the target scenario rule.

[0008] According to another aspect of the present disclosure, there is provided a multimodal data processing device, comprising:

[0009] A data acquisition unit, configured to acquire first multimodal data sent by a target client, wherein the first multimodal data is multimodal data for a target scenario sent by the target client;

[0010] A rule processing unit, configured to adjust the initial scenario rule based on the target history feedback data to obtain a target scenario rule for the target client; wherein the target history feedback data is data associated with the scenario rule;

[0011] A data matching unit is used to match the multimodal features of the first multimodal data with the rule features of the target scenario rule to obtain a matching result, wherein the matching result represents whether there is a picture in the target scene that does not satisfy the target scenario rule.

[0012] According to another aspect of the present disclosure, there is provided an electronic device, comprising:

[0013] at least one processor; and

[0014] a memory communicatively connected to the at least one processor; wherein,

[0015] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute any method in the embodiments of the present disclosure.

[0016] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable the computer to execute any method according to the embodiments of the present disclosure.

[0017] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements any method according to the embodiments of the present disclosure when executed by a processor.

[0018] In this way, the disclosed solution can use the target historical feedback data to adjust the initial scenario rules to achieve dynamic updating of the scenario rules, and then match the multimodal features of the first multimodal data with the rule features of the updated initial scenario rules (i.e., the target scenario rules) to determine whether there are pictures in the target scene that do not meet the target scenario rules. Here, since the target scenario rules used for matching are adjusted based on the target historical feedback data, the accuracy of the matching results can be significantly improved, and the matching results can be made closer to user needs, which effectively improves the user experience while enriching the user experience.

[0019] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0021] Figure 1 This is a schematic flow chart of a multimodal data processing method according to an embodiment of the present application. Figure 1 ;

[0022] Figure 2 is a schematic diagram of a scenario in an example of a multimodal data processing method according to an embodiment of the present application;

[0023] Figure 3 This is a schematic flow chart of a multimodal data processing method according to an embodiment of the present application. Figure 2 ;

[0024] Figure 4 This is a schematic flow chart of a multimodal data processing method according to an embodiment of the present application. Figure 3 ;

[0025] Figure 5 is a structural diagram of a multimodal data processing device according to an embodiment of the present application;

[0026] Figure 6 It is a block diagram of an electronic device used to implement the multimodal data processing method of the embodiment of the present disclosure. DETAILED DESCRIPTION

[0027] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.

[0028] The term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there may be three relationships. For example, A and / or B can mean: A exists alone, A and B exist at the same time, and B exists alone. The term "at least one" in this article means any combination of at least two of any one or more of a plurality of. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C. The terms "first" and "second" in this article refer to multiple similar technical terms and distinguish them. They do not mean to limit the order or to limit them to only two. For example, the first feature and the second feature refer to two types / two features. The first feature can be one or more, and the second feature can also be one or more.

[0029] In addition, in order to better illustrate the present disclosure, numerous specific details are given in the specific embodiments below. It should be understood by those skilled in the art that the present disclosure can also be implemented without certain specific details. In some examples, methods, means, components and circuits well known to those skilled in the art are not described in detail in order to highlight the main purpose of the present disclosure.

[0030] The disclosed solution proposes a multimodal data processing method to improve the accuracy of monitoring identification.

[0031] Specifically, Figure 1 This is a schematic flow chart of a multimodal data processing method according to an embodiment of the present application. Figure 1 The method may be optionally applied to electronic devices, such as personal computers, servers, server clusters, and other electronic devices.

[0032] Further, the method includes at least part of the following contents. Figure 1 As shown, including:

[0033] Step S101: Acquire first multimodal data sent by a target client.

[0034] Here, the first multimodal data is multimodal data for a target scenario sent by the target client.

[0035] For example, in one example, the first multimodal data includes at least one of the following: images, audio data, video data, and text data.

[0036] Furthermore, in one example, the target client obtains video data collected by a collection device (such as a camera) connected to the target client and installed in the target scene.

[0037] Furthermore, in one example, the acquisition device may also perform video and audio splitting processing on the acquired video data to split out audio data and video frames, and then send the audio data and video frames together to the target client.

[0038] Step S102: Based on the target historical feedback data, the initial scenario rules are adjusted to obtain target scenario rules for the target client.

[0039] Here, the target history feedback data is data associated with the scenario rules.

[0040] Furthermore, in one example, the initial scenario rule may specifically be a pre-set scenario rule, or may specifically be a scenario rule adjusted last time based on historical feedback data.

[0041] Step S103: Match the multimodal features of the first multimodal data with the rule features of the target scenario rule to obtain a matching result.

[0042] Here, the matching result indicates whether there are pictures in the target scene that do not meet the target scenario rule.

[0043] In this way, the disclosed solution can use the target historical feedback data to adjust the initial scenario rules to achieve dynamic updating of the scenario rules, and then match the multimodal features of the first multimodal data with the rule features of the updated initial scenario rules (i.e., the target scenario rules) to determine whether there are pictures in the target scene that do not meet the target scenario rules. Here, since the target scenario rules used for matching are adjusted based on the target historical feedback data, the accuracy of the matching results can be significantly improved, and the matching results can be made closer to user needs, which effectively improves the user experience while enriching the user experience.

[0044] In addition, since the target scenario rules used for matching can be adjusted based on the target historical feedback data, compared with the existing solutions with fixed rules, the disclosed solution is more flexible and intelligent, meeting the different customized needs of users, thereby further improving the user experience.

[0045] In a specific example, the target scenario is but not limited to one of the following: home security, educational venues, industrial monitoring, community security, etc. It should be noted that the disclosed solution does not impose specific restrictions on the target scenario.

[0046] Furthermore, in a specific example, after obtaining the matching result, the method further includes:

[0047] Step S104: Based on the matching results, determine whether there is a screen that does not meet the target scenario rule; if so, proceed to step S105; otherwise, end the process.

[0048] Step S105: if it is determined that there is a picture that does not meet the target scenario rule, generate warning information.

[0049] Step S106: Send the warning information to the target client.

[0050] That is to say, after obtaining the matching result, it can be determined based on the matching result whether there is a screen that does not meet the target scenario rule. If so, an alarm message is generated based on the matching result, and the generated alarm message is sent to the target client for prompting.

[0051] In this way, the disclosed solution can use the obtained matching results to quickly determine the images that do not meet the target scenario rules, and then generate alarm information. In this way, when information prompts are needed, alarms can be issued in a timely manner. For example, it can ensure that the target object can be aware of potential threats or abnormal events in real time, thereby further improving the user experience while meeting the user's customized needs.

[0052] In a specific example, if Figure 2 As shown, in actual applications, the disclosed solution includes a collection device end and a client end (such as a target client), which are bound to each other; wherein the collection device end may be specifically a camera, or an intelligent speaker installed in the target environment, or an intelligent entity with interactive functions and collection functions, for collecting video data, audio data, etc. The client end may be specifically installed on a mobile phone or a computer end, and the main functions of the client end may include: viewing surveillance videos, customizing surveillance alarm behaviors (i.e., scenario rules), analyzing video content, triggering alarm notifications, etc. Further, in one example, the collection device end may transmit the collected video or audio content within a preset time period to the client end at regular intervals for the client end to view or match, so as to quickly determine whether there are any pictures that do not meet the target scenario rules.

[0053] Figure 3 This is a schematic flow chart of a multimodal data processing method according to an embodiment of the present application. Figure 2 The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters and other electronic devices. It can be understood that the above Figure 1 and Figure 2 The relevant contents of the method shown can also be applied to this example, and the relevant contents will not be described in detail in this example.

[0054] Further, the method includes at least part of the following contents. Figure 3 As shown, including:

[0055] Step S301: Acquire first multimodal data sent by a target client.

[0056] Here, the first multimodal data is multimodal data for a target scenario sent by the target client.

[0057] Here, the relevant contents about the target client and the first multimodal data can be referred to the above example, which will not be repeated here.

[0058] Step S302: Acquire multiple historical feedback data.

[0059] Here, the plurality of historical feedback data includes at least one of the following: historical feedback data acquired from a plurality of clients (eg, a plurality of clients corresponding to a target scenario), and historical feedback data acquired from the target client.

[0060] Further, the historical feedback data represents feedback data generated by the client after at least evaluating the alarm information, such as feedback on false positives, false negatives, accuracy, etc. of the alarm information. Further, the historical feedback data represents feedback data generated by the client after at least evaluating the alarm information for the target scenario.

[0061] Step S303: based on the plurality of historical feedback data, obtaining the target historical feedback data associated with the target client.

[0062] For example, in one example, for the same target scene, multiple clients can be connected to the acquisition device in the target scene. At this time, not only the target client can receive the alarm information for the target scene, but other clients connected to the acquisition device in the target scene can also receive the alarm information about the target scene. In this scenario, multiple clients can provide feedback on the received alarm information. In this way, multiple historical feedback data of the same or different alarm information for the same target scene can be obtained. At this time, the target historical feedback data can be determined based on the multiple historical feedback data. For example, screening rules (such as deduplication rules, priority rules, etc.) can be set in advance to filter out the target historical feedback data from the multiple historical feedback data. Alternatively, the multiple historical feedback data can be scored and sorted to filter out the target historical feedback data based on the sorting results. In this way, strong data support is provided for the subsequent real-time update of the initial scenario rules.

[0063] Step S304: Based on the target historical feedback data, the initial scenario rules are adjusted to obtain target scenario rules for the target client.

[0064] Here, the target history feedback data is data associated with the scenario rules.

[0065] Step S305: Match the multimodal features of the first multimodal data with the rule features of the target scenario rule to obtain a matching result.

[0066] Here, the matching result indicates whether there are pictures in the target scene that do not meet the target scenario rule.

[0067] Step S306: Based on the matching result, determine whether there is a screen that does not meet the target scenario rule; if so, proceed to step S307; otherwise, end the process.

[0068] Step S307: if it is determined that there is a screen that does not meet the target scenario rule, generate warning information.

[0069] Step S308: Send the warning information to the target client.

[0070] In this way, the disclosed solution can use the collected target historical feedback data to adjust the initial scenario rules in real time, and then use the adjusted initial scenario rules to determine whether there is any non-standard behavior in the first multimodal data. In this way, the accuracy of the matching results is effectively improved, and the matching results are closer to user needs. At the same time, the disclosed solution is made more intelligent, flexible, and reliable, thereby further improving the user experience.

[0071] Figure 4 This is a schematic flow chart of a multimodal data processing method according to an embodiment of the present application. Figure 3 The method can be optionally applied to electronic devices, such as personal computers, servers, server clusters and other electronic devices. It can be understood that the above Figures 1 to 3 The relevant contents of the method shown can also be applied to this example, and the relevant contents will not be described in detail in this example.

[0072] Further, the method includes at least part of the following contents. Figure 4 As shown, including:

[0073] Step S401: Acquire second multimodal data from the target client.

[0074] Here, the second multimodal data represents the rule requirements that the target scene specified by the target client needs to meet, including at least two of the following: images, texts, and video clips.

[0075] Step S402: input the second multimodal data into a first model (eg, a multimodal large model) to obtain rule parsing data for the second multimodal data.

[0076] Step S403: Parsing data based on the rules for the second multimodal data to obtain the initial scenario rules.

[0077] That is to say, in this example, the disclosed solution can be based on the second multimodal data uploaded by the target object and use the multimodal large model to perform automatic rule customization. In this way, compared with the existing single rule method, the rule customization method of the disclosed solution is more novel, flexible and diverse, effectively avoiding the tedious operations of the target object, thereby further improving the user experience.

[0078] Step S404: Acquire the first multimodal data sent by the target client.

[0079] Here, the first multimodal data is multimodal data for a target scenario sent by the target client.

[0080] Here, the relevant contents about the target client and the first multimodal data can be referred to the above example, which will not be repeated here.

[0081] Step S405: Based on the target historical feedback data, the initial scenario rules are adjusted to obtain target scenario rules for the target client.

[0082] Here, the target history feedback data is data associated with the scenario rules.

[0083] Step S406: Match the multimodal features of the first multimodal data with the rule features of the target scenario rule to obtain a matching result.

[0084] Here, the matching result indicates whether there are pictures in the target scene that do not meet the target scenario rule.

[0085] Further, in a specific example, the target scenario rule may be obtained in the following manner; specifically, the above-mentioned adjustment of the initial scenario rule based on the target historical feedback data to obtain the target scenario rule for the target client may specifically include:

[0086] Inputting the target historical feedback data into the first model to obtain rule parsing data for the target historical feedback data;

[0087] Based on the rule parsing data for the target historical feedback data, the initial scenario rule is adjusted to obtain the target scenario rule.

[0088] Here, the first model can be specifically a pre-trained model that can parse feedback data. For example, in one scenario, the first model can also be a large language model. In this case, the first target prompt word for the target historical feedback data can be generated based on the target historical feedback data and the first preset prompt word template (for example, including the required generated parsed data case, etc.), and the first target prompt word is input into the large language model to obtain rule parsing data.

[0089] Furthermore, in yet another example, the large language model can be directly used to obtain the target scenario rules. For example, based on the target historical feedback data and a second preset prompt word template (for example, including the required generated scenario rule cases, rule adjustment cases, etc.), a second target prompt word can be generated for the target historical feedback data and the initial scenario rules, and the second target prompt word can be synchronously input into the large language model to obtain the target scenario rules.

[0090] In this way, the disclosed solution provides a refined solution for adjusting the initial scenario rules using a large model. The solution is simple and efficient, and can effectively improve processing efficiency. At the same time, it also effectively uses the performance of the large model to further make the target scenario rules closer to user needs, thereby laying the foundation for further improving the accuracy of matching results and avoiding false alarms. At the same time, it also further improves the intelligence level of the disclosed solution, thereby providing strong support for further improving user experience.

[0091] Further, in a specific example, the above-mentioned matching of the multimodal features of the first multimodal data with the rule features of the target scenario rule to obtain a matching result may specifically include:

[0092] The first multimodal data and the target scenario rules are input into a second model to obtain the multimodal features of the first multimodal data and the rule features of the target scenario rules using the second model, and the multimodal features of the first multimodal data are similarly matched with the rule features of the target scenario rules to obtain the matching result.

[0093] That is to say, in one example, the first multimodal data and the target scenario rules can be directly input into the second model. At this time, the second model will first perform feature fusion on the different modal data contained in the first multimodal data to obtain the multimodal features of the first multimodal data, and perform feature extraction on the target scenario rules to obtain the rule features of the target scenario rules, and then perform similarity matching on the obtained multimodal features of the first multimodal data and the rule features of the target scenario rules to obtain the matching results.

[0094] Here, the second model can be specifically a pre-trained model that can obtain the required matching result. For example, in one scenario, the first model can also be a large language model. At this time, a third target prompt word can be generated based on the first multimodal data and the target scenario rule, and the third target prompt word is input into the large language model to obtain the matching result.

[0095] In this way, the disclosed solution can use the second model to quickly calculate the feature-level similarity between the first multimodal data and the target scenario rules to obtain a matching result. This effectively improves the accuracy of the matching result, thereby laying a foundation for further improving the accuracy of the matching result and avoiding false alarms. At the same time, it also further improves the intelligence level of the disclosed solution, thereby providing strong support for further improving the user experience.

[0096] The following is a detailed description of the disclosed solution with reference to specific examples; the disclosed solution proposes a multimodal data processing method to improve the accuracy of monitoring identification.

[0097] Specifically, the specific implementation steps of the disclosed solution include:

[0098] Part I: Technical Implementation of Multimodal Large Models

[0099] 1.1 Data preprocessing and feature extraction

[0100] Step 1: Data Preprocessing

[0101] The multimodal data uploaded by the target object (corresponding to the first multimodal data above) is processed into structured feature data, for example, into text features, image features and video features.

[0102] For example, in one example, text preprocessing can be performed in the following manner:

[0103] The text data is segmented to remove stop words and retain keywords to obtain processed text data; then word embedding technology (such as word vector conversion model (Word to Vector Model, Word2Vec), global vector word embedding (Global Vectors for Word Representation, GloVe), or bidirectional encoder representation converter (Bidirectional Encoder Representations from Transformers, BERT)) is used to convert the processed text data (such as keywords) into vector form to obtain a text feature vector.

[0104] Furthermore, in another example, the image preprocessing may be performed in the following manner:

[0105] The image is normalized, resized, and data enhancement techniques (such as rotation, flipping, and cropping) are applied to improve the robustness of the model. Then, a pre-trained convolutional neural network (such as Residual Network (ResNet) and Visual Geometry Group Network (VGG)) is used to extract features from the data-enhanced image data to obtain an image feature vector.

[0106] Furthermore, in yet another example, the video preprocessing may be performed in the following manner:

[0107] The video is divided into frames, and key frame detection is performed in combination with algorithms (such as Scale-Invariant Feature Transform (SIFT) and Speeded Up Robust Features (SURF)) to select key frames; then, based on the selected key frames, feature extraction is performed using a spatiotemporal convolutional neural network (such as 3D Convolutional Neural Network (3D-CNN) and Convolutional 3D Network (C3D)) to obtain a video feature vector.

[0108] Step 2: Feature Fusion

[0109] The text feature vector, image feature vector and video feature vector processed in step 1 are fused to obtain multimodal features.

[0110] For example, in one example, the following techniques can be used to implement feature fusion:

[0111] (1) Cross-Modal Attention: By calculating the similarity between text feature vectors, image feature vectors, and video feature vectors, a weight matrix is ​​generated to fuse feature vectors of different modalities.

[0112] (2) Transformer architecture: Use Transformer-based multimodal models (such as Contrastive Language-Image Pre-training (CLIP) and Universal Image-Text Representation (UNITER)) for feature fusion.

[0113] Here, the Transformer-based multimodal model helps to capture the correlation between different modalities and integrate this information into a unified multimodal feature representation.

[0114] Here, it should be noted that the processing logic of the above steps 1 and 2 can be implemented in the multimodal big model. In other words, the multimodal data uploaded by the target object can be directly input into the multimodal big model to obtain the multimodal features of the multimodal data.

[0115] 1.2 Matching and Reasoning of Multimodal Scenario Rules

[0116] Step 3: Establishment of scenario rule base

[0117] The multimodal data defined by the target object (corresponding to the second multimodal data above) is processed into a structured scenario rule base.

[0118] For example, in one example, this can be achieved as follows:

[0119] (1) Directly use natural language processing technology to convert the scenario description of the target object into rule features to build a scenario rule library;

[0120] (2) Clustering the feature vectors of the example images and videos with the rule features of other scenario rules to form a scenario rule library containing multiple scenario rules.

[0121] Step 4: Scenario rule matching

[0122] The multimodal features obtained in step 2 above and the rule features of the scenario rules in the scenario rule library (corresponding to the initial scenario rules above) are matched by similarity calculation and inference engine to obtain matching results, for example, similarity scores.

[0123] For example, in one example, the following technical implementation can be used:

[0124] (1) Similarity calculation: Use cosine similarity or Euclidean distance to calculate the similarity between multimodal features and rule features in the scenario rule base.

[0125] (2) Reasoning engine: Introduce a reasoning engine based on a large model (such as GPT-4 or other multimodal models) to perform deep reasoning based on feature similarity, context associations, and complex situational conditions to output the final matching results.

[0126] Here, it should be noted that in actual applications, feedback data can also be collected. For example, a feedback module can be installed on the client to collect feedback from the target object on the alarm results (such as false alarms, missed alarms, accuracy, etc.) to obtain a feedback data set (corresponding to the above target historical feedback data) and perform structured storage.

[0127] Furthermore, in one example, the feedback data set includes a revised scenario description of the target object and newly added multimodal data (eg, newly added data related to scenario rule formulation).

[0128] Furthermore, the feedback data set can be used to adjust the scenario rule base:

[0129] The feedback data set is used to optimize the scenario rules in the scenario rule base to obtain the optimized scenario rules, so as to obtain the optimized scenario rule base, and then the optimized scenario rule base can be used for matching, so as to further improve the accuracy of the matching results.

[0130] Furthermore, the alarm threshold of the scenario rule can be dynamically adjusted. For example, the alarm threshold corresponding to the scenario rule can be adjusted using feedback data, so as to adapt to different scenario requirements.

[0131] In summary, the disclosed solution has the following advantages:

[0132] First, higher accuracy. The disclosed solution can achieve deep fusion of features such as text, images, and videos, and perform matching after fusion, thus effectively improving the accuracy of scene recognition.

[0133] Second, it is more adaptable. The disclosed solution can respond to user feedback immediately and quickly adjust scenario rules to ensure that the accuracy of scenario rule matching is greatly improved, thereby achieving real-time and intelligent alarm management; in addition, it also effectively enhances the intelligence and flexibility of the system.

[0134] Third, better user experience. On the one hand, the disclosed solution can use user feedback to adjust the scenario rules in real time, thus improving the accuracy of the matching results. On the other hand, the disclosed solution ensures that key information is conveyed to the user while effectively avoiding false alarms, thus improving the user experience and the level of intelligence.

[0135] The disclosed solution also provides a multi-modal data processing device, such as Figure 5 As shown, including:

[0136] The data acquisition unit 501 is used to acquire first multimodal data sent by a target client, where the first multimodal data is multimodal data for a target scenario sent by the target client;

[0137] A rule processing unit 502 is used to adjust the initial scenario rule based on the target history feedback data to obtain the target scenario rule for the target client; wherein the target history feedback data is data associated with the scenario rule;

[0138] The data matching unit 503 is used to match the multimodal features of the first multimodal data with the rule features of the target scenario rule to obtain a matching result, wherein the matching result indicates whether there is a picture in the target scene that does not satisfy the target scenario rule.

[0139] In a specific example of the disclosed solution, the data matching unit is further used to:

[0140] If it is determined based on the matching result that there is a picture that does not meet the target scenario rule, generating warning information;

[0141] The warning information is sent to the target client.

[0142] In a specific example of the disclosed solution, the rule processing unit is further used to:

[0143] A plurality of historical feedback data are obtained, wherein the plurality of historical feedback data include at least one of the following: historical feedback data obtained from a plurality of clients, and historical feedback data obtained from the target client; the historical feedback data represents feedback data generated after the client at least evaluates the alarm information;

[0144] Based on the multiple historical feedback data, the target historical feedback data associated with the target client is obtained.

[0145] In a specific example of the disclosed solution, the rule processing unit is further used to:

[0146] Acquire second multimodal data from the target client, where the second multimodal data represents rule requirements that the target scene specified by the target client needs to meet, including at least two of the following: images, texts, and video clips;

[0147] Inputting the second multimodal data into the first model to obtain rule parsing data for the second multimodal data;

[0148] The initial scenario rule is obtained based on rule parsing data for the second multimodal data.

[0149] In a specific example of the disclosed solution, the rule processing unit is specifically used to:

[0150] Inputting the target historical feedback data into the first model to obtain rule parsing data for the target historical feedback data;

[0151] Based on the rule parsing data for the target historical feedback data, the initial scenario rule is adjusted to obtain the target scenario rule.

[0152] In a specific example of the disclosed solution, the data matching unit is specifically used to:

[0153] The first multimodal data and the target scenario rules are input into a second model to obtain the multimodal features of the first multimodal data and the rule features of the target scenario rules using the second model, and the multimodal features of the first multimodal data are similarly matched with the rule features of the target scenario rules to obtain the matching result.

[0154] For the description of the specific functions and examples of each unit of the device in the embodiment of the present disclosure, reference can be made to the relevant description of the corresponding steps in the above method embodiment, which will not be repeated here.

[0155] In the technical solution disclosed herein, the acquisition, storage and application of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0156] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0157] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0158] like Figure 6 As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0159] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0160] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as multimodal data processing methods. For example, in some embodiments, the multimodal data processing method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via ROM 602 and / or communication unit 609. When the computer program is loaded into RAM 603 and executed by the computing unit 601, one or more steps of the multimodal data processing method described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to perform the multimodal data processing method in any other appropriate manner (e.g., by means of firmware).

[0161] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), load programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0162] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0163] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0164] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0165] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0166] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.

[0167] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0168] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the principles of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A multimodal data processing method, comprising: Acquire first multimodal data sent by a target client, where the first multimodal data is multimodal data sent by the target client for a target scenario; Based on the target historical feedback data, the initial scenario rules are adjusted to obtain the target scenario rules for the target client; wherein the target historical feedback data is data associated with the scenario rules; The multimodal features of the first multimodal data are matched with the rule features of the target scenario rule to obtain a matching result, wherein the matching result indicates whether there is a picture in the target scene that does not satisfy the target scenario rule.

2. The method according to claim 1, further comprising: If it is determined based on the matching result that there is a picture that does not meet the target scenario rule, generating warning information; The warning information is sent to the target client.

3. The method according to claim 2, further comprising: A plurality of historical feedback data are obtained, wherein the plurality of historical feedback data include at least one of the following: historical feedback data obtained from a plurality of clients, and historical feedback data obtained from the target client; the historical feedback data represents feedback data generated after the client at least evaluates the alarm information; Based on the multiple historical feedback data, the target historical feedback data associated with the target client is obtained.

4. The method according to any one of claims 1 to 3, further comprising: Acquire second multimodal data from the target client, where the second multimodal data represents rule requirements that the target scene specified by the target client needs to meet, including at least two of the following: images, texts, and video clips; Inputting the second multimodal data into the first model to obtain rule parsing data for the second multimodal data; The initial scenario rule is obtained based on rule parsing data for the second multimodal data.

5. The method according to claim 4, wherein: The adjusting the initial scenario rules based on the target historical feedback data to obtain the target scenario rules for the target client includes: Inputting the target historical feedback data into the first model to obtain rule parsing data for the target historical feedback data; Based on the rule parsing data for the target historical feedback data, the initial scenario rule is adjusted to obtain the target scenario rule.

6. The method according to any one of claims 1 to 5, wherein: The matching the multimodal feature of the first multimodal data with the rule feature of the target scenario rule to obtain a matching result includes: The first multimodal data and the target scenario rules are input into a second model to obtain the multimodal features of the first multimodal data and the rule features of the target scenario rules using the second model, and the multimodal features of the first multimodal data are similarly matched with the rule features of the target scenario rules to obtain the matching result.

7. A multimodal data processing device, comprising: A data acquisition unit, configured to acquire first multimodal data sent by a target client, wherein the first multimodal data is multimodal data for a target scenario sent by the target client; A rule processing unit, configured to adjust the initial scenario rule based on the target history feedback data to obtain a target scenario rule for the target client; wherein the target history feedback data is data associated with the scenario rule; A data matching unit is used to match the multimodal features of the first multimodal data with the rule features of the target scenario rule to obtain a matching result, wherein the matching result represents whether there is a picture in the target scene that does not satisfy the target scenario rule.

8. The device according to claim 7, wherein: The data matching unit is further used for: If it is determined based on the matching result that there is a picture that does not meet the target scenario rule, generating warning information; The warning information is sent to the target client.

9. The device according to claim 8, wherein: The rule processing unit is further used for: A plurality of historical feedback data are obtained, wherein the plurality of historical feedback data include at least one of the following: historical feedback data obtained from a plurality of clients, and historical feedback data obtained from the target client; the historical feedback data represents feedback data generated after the client at least evaluates the alarm information; Based on the multiple historical feedback data, the target historical feedback data associated with the target client is obtained.

10. The device according to any one of claims 7 to 9, wherein: The rule processing unit is further used for: Acquire second multimodal data from the target client, where the second multimodal data represents rule requirements that the target scene specified by the target client needs to meet, including at least two of the following: images, texts, and video clips; Inputting the second multimodal data into the first model to obtain rule parsing data for the second multimodal data; The initial scenario rule is obtained based on rule parsing data for the second multimodal data.

11. The device according to claim 10, wherein: The rule processing unit is specifically used for: Inputting the target historical feedback data into the first model to obtain rule parsing data for the target historical feedback data; Based on the rule parsing data for the target historical feedback data, the initial scenario rule is adjusted to obtain the target scenario rule.

12. The device according to any one of claims 7 to 11, wherein: The data matching unit is specifically used for: The first multimodal data and the target scenario rules are input into a second model to obtain the multimodal features of the first multimodal data and the rule features of the target scenario rules using the second model, and the multimodal features of the first multimodal data are similarly matched with the rule features of the target scenario rules to obtain the matching result.

13. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-6.

15. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 6.

Images