Event recognition method, device, storage medium, and program product

Abstract

Embodiments of the present application provide an event recognition method, device, storage medium and program product. In the event recognition method, after obtaining scene image data in a work operation process, a target detection model can be used to perform target detection on the scene image data to obtain attribute information of a target object contained in the scene image data. According to the attribute information of the target object, a target work scene corresponding to the scene image data can be determined, and a target multi-modal model can be used to recognize a target event type corresponding to the scene image data in the target work scene according to the attribute information of the target object and knowledge information corresponding to different event types in the target work scene. In this implementation, the target detection model based on computer vision is combined with the target multi-modal model, the advantages of the target detection model in fine-grained target detection and the powerful ability of the multi-modal model in advanced semantic understanding and reasoning are fully utilized, and more accurate event recognition is achieved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to an event identification method and device, a storage medium and a program product. BACKGROUND

[0002] In the process of work operations in the fields of electric power, transportation, and industrial production, the operation environment of operators and equipment is relatively complex, and therefore, the requirements for ensuring operation safety and avoiding risks are relatively high. It is very important to identify safety events and risk events in the operation scene and to timely warn or intervene in risk events. For example, in the field of industrial production, it can be identified whether a welding operator wears goggles when performing welding operations. In the field of electric power, it can be identified whether a maintenance worker uses a compliant operation tool when performing electric power maintenance operations.

[0003] The definition of safety events and risk events in the operation scene is relatively complex in different fields, and traditional event identification methods cannot accurately identify different types of events in the operation scene. Therefore, a new solution is needed. SUMMARY

[0004] The embodiments of the present application provide an event identification method, device, storage medium and program product to accurately identify the event type in the work operation scene.

[0005] The embodiments of the present application provide an event identification method, which includes: acquiring scene image data in a work operation process; using a target detection model to perform target detection on the scene image data to obtain attribute information of a target object contained in the scene image data; determining a target operation scene corresponding to the scene image data according to the attribute information of the target object; and using a target multi-modal model to identify a target event type corresponding to the scene image data in the target operation scene according to the attribute information of the target object and knowledge information corresponding to different event types in the target operation scene.

[0006] Optionally, determining the target operation scene corresponding to the scene image data according to the attribute information of the target object includes: acquiring class information of the target object according to the attribute information of the target object; comparing the class information of the target object with class information of necessary objects contained in different operation scenes, respectively; and if the class information of the target object matches the class information of necessary objects contained in any operation scene in the different operation scenes, determining the any operation scene as the target operation scene corresponding to the scene image data.

[0007] Optionally, the target event type corresponding to the scene image data in the target work scene is identified according to the attribute information of the target object and the knowledge information corresponding to different event types in the target work scene by using the target multi-modal model, including: determining whether the target event type corresponding to the scene image data in the target work scene can be determined according to the attribute information of the target object according to the event identification rule of the target work scene; if not, the target event type corresponding to the scene image data in the target work scene is identified according to the attribute information of the target object and the knowledge information corresponding to different event types in the target work scene by using the target multi-modal model.

[0008] Optionally, it further includes: if the target event type corresponding to the scene image data in the target work scene can be determined according to the attribute information of the target object, the category of the target object is determined according to the attribute information of the target object; the category of the target object is matched with the necessary object categories dependent on different event types in the target work scene respectively; if the category of the target object is consistent with the necessary object category dependent on any event type in the target work scene, the any event type is determined as the target event type, and the target event type is output.

[0009] Optionally, determining whether the target event type corresponding to the scene image data in the target work scene can be determined according to the attribute information of the target object according to the event identification rule of the target work scene includes: if the event identification rule of the target work scene depends on the category of the object, it is determined that the target event type corresponding to the scene image data in the target work scene can be determined according to the attribute information of the target object; if the event identification rule of the target work scene depends on the category of the object and the relationship between objects, it is determined that the target event type corresponding to the scene image data in the target work scene cannot be determined according to the attribute information of the target object.

[0010] Optionally, the attribute information of the target object includes coordinate information and category information of the target object; and the target event type corresponding to the scene image data in the target work scene is identified by using the target multi-modal model according to the attribute information of the target object and the knowledge information corresponding to different event types in the target work scene, including: the scene sub-image corresponding to the target object is obtained by cropping the scene image data according to the coordinate information of the target object; the target prompt word is constructed according to the scene sub-image, the coordinate information and the category information of the target object, and the knowledge information corresponding to different events in the target work scene; and the target event type corresponding to the scene image data in the target work scene is identified by using the target multi-modal model according to the target prompt word.

[0011] Optionally, the target event type corresponding to the scene image data in the target work scene is identified by using the target multi-modal model according to the target prompt word, including: the target prompt word is input into the target multi-modal model; the coordinate information and the category information of each of the plurality of objects in the target object are obtained from the target prompt word in the target multi-modal model, and the semantic features of the scene sub-image of each of the plurality of objects are extracted; the logical relationship between the plurality of objects is identified according to the coordinate information, the category information and the semantic features of each of the plurality of objects; the target knowledge information adapted to the logical relationship between the plurality of objects is obtained by matching the knowledge information corresponding to different event types according to the logical relationship between the plurality of objects; and the event type corresponding to the target knowledge information is determined as the event type corresponding to the scene image data in the target work scene.

[0012] Optionally, the target multi-modal model is obtained by lightweight compression of a pre-trained multi-modal large model, and the target multi-modal model is deployed on a terminal device or an edge device.

[0013] Embodiments of the present application also provide an electronic device, comprising: a memory and a processor; the memory is used to store one or more computer instructions; and the processor is used to execute the one or more computer instructions to execute the steps in the method provided by the embodiments of the present application.

[0014] Embodiments of the present application also provide a computer readable storage medium storing a computer program, which can implement the steps in the method provided by the embodiments of the present application when executed by a processor.

[0015] The embodiment of the present application further provides a computer program product, comprising: computer programs / instructions, which can realize the steps in the method provided by the embodiment of the present application when executed by a processor.

[0016] In the embodiment of the present application, after the scene image data in the engineering operation process is acquired, the target detection model can be used to perform target detection on the scene image data to obtain attribute information of the target object contained in the scene image data. According to the attribute information of the target object, the target operation scene corresponding to the scene image data can be determined, and the target multi-modal model can be used to identify the target event type corresponding to the scene image data in the target operation scene according to the attribute information of the target object and the knowledge information corresponding to different event types in the target operation scene. In this implementation, the target detection model based on computer vision is combined with the target multi-modal model, the advantages of the target detection model in fine-grained target detection and the powerful ability of the multi-modal model in advanced semantic understanding and reasoning are fully utilized, and more accurate event recognition is achieved.

[0017] Secondly, the target detection model is used as the pre-model of the target multi-modal model, which can make the target multi-modal model focus on advanced semantic understanding and reasoning, is conducive to reducing the parameter quantity of the target multi-modal model, can significantly reduce the engineering landing complexity while ensuring the semantic understanding ability, and thus facilitates the engineering deployment of the target multi-modal model. BRIEF DESCRIPTION OF DRAWINGS

[0018] The accompanying drawings used to provide further understanding of the present application and form a part of the present application, and the illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute improper limitations on the present application. In the drawings:

[0019] Figure 1 A flowchart of an event recognition method provided by an exemplary embodiment of the present application is shown in FIG. 2;

[0020] Figure 2 A flowchart of an event recognition method provided by another exemplary embodiment of the present application is shown in FIG. 3;

[0021] Figure 3 A structural diagram of an electronic device provided by an exemplary embodiment of the present application is shown in FIG. 4. DETAILED DESCRIPTION

[0022] In order to make the objects, technical solutions and advantages of the present application clearer, the following will clearly and completely describe the technical solutions of the present application with reference to the specific embodiments and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, but not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0023] The terms used in the embodiments of the present application are only for the purpose of describing particular embodiments and are not intended to limit the present application. The singular forms "a", "said" and "the" used in the embodiments of the present application and the appended claims are also intended to include the plural forms, unless the context clearly indicates otherwise. "Plural" generally includes at least two, but does not exclude the case of including at least one.

[0024] It should be understood that the term "and / or" used herein is only to describe the association relationship of the associated objects, which means that there can be three relationships, for example, A and / or B, which can represent the three cases of A alone, A and B together, and B alone. In addition, the character " / " herein generally represents an "or" relationship between the front and rear associated objects.

[0025] It should also be noted that the terms "comprising", "including", or any other variant thereof are intended to cover non-exclusive inclusion, so that a product or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such product or system. Without more limitations, the element defined by the statement "including a" does not exclude the presence of additional identical elements in the product or system including the element.

[0026] In the process of work operations in the fields of electric power, transportation, industrial production, etc., the operation environment of the operators and the equipment is relatively complex, and therefore the requirements for ensuring operation safety and avoiding risks are relatively high. It is very important to identify safety events and risk events in the operation scene and to timely warn or intervene in the risk events. For example, in the field of industrial production, it can be identified whether a welding operator wears goggles when performing welding operations. In the field of electric power, it can be identified whether a maintenance worker uses a compliant operation tool when performing electric power maintenance operations. In different fields, the definition of safety events and risk events in the operation scene is relatively complex, and traditional event identification means cannot accurately identify different types of events in the operation scene.

[0027] In one solution, the event type in a work scene can be identified based on a computer vision (CV) model and event discrimination rules. However, in this solution, the CV model has a small size, poor generalization ability, and low accuracy in judgment, and cannot accurately identify the event type in the work scene when facing a work scene with complex semantic requirements.

[0028] In other solutions, a multi-modal large model with a large number of parameters can be directly used to understand and determine events in pictures taken during the work process. However, when performing fine-grained target recognition tasks for specific scenes, multi-modal large models may not perform as well as small models designed specifically for such tasks. In addition, multi-modal large models have a large number of parameters, require more data for model fine-tuning, and require more machine resources and higher costs for deployment, which is not conducive to engineering deployment in actual production environments with limited resources.

[0029] To address the above technical problems, in some embodiments of the present application, a solution is provided, and the technical solutions provided by the embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0030] Figure 1 is a flowchart of an event recognition method provided by an exemplary embodiment of the present application. The method can include the steps as shown in Figure 1 .

[0031] Step 101, acquiring scene image data in a work process.

[0032] Step 102, using a target detection model to perform target detection on the scene image data to obtain attribute information of a target object contained in the scene image data.

[0033] Step 103, determining a target work scene corresponding to the scene image data according to the attribute information of the target object.

[0034] Step 104, using a target multi-modal model to identify a target event type corresponding to the scene image data in the target work scene according to the attribute information of the target object and knowledge information corresponding to different event types in the target work scene.

[0035] The embodiment is suitable for a work operation scene, and is used for analyzing scene image data in a work operation process to identify an event type in the work operation process. The work operation refers to an operation involved in industrial production and engineering construction, and is mainly used for tasks related to infrastructure construction, equipment maintenance, process operation, and technical support, etc. The work operation can include, but is not limited to, production equipment installation and debugging, engineering structure construction (such as welding and cutting), infrastructure operation and maintenance (such as railway traction and power inspection), safety hazard investigation, and process standard implementation, etc. The work operation environment is usually complex, and depends on professional technology and standardized processes to ensure safe and efficient implementation.

[0036] For example, in the production equipment installation and debugging scene, the operator needs to wear appropriate personal protective equipment, such as a safety helmet, protective shoes, etc., especially when handling heavy machinery. In the welding operation scene, the operator needs to wear goggles. In the cutting operation scene, the operator needs to wear goggles, and the operator should maintain an appropriate safety distance from the cutting point to avoid injury caused by flying sparks. For example, in the power inspection operation scene, the operator should wear gloves and wear insulated boots to protect the whole body from the risk of electric shock.

[0037] The embodiment can be executed by an event identification device, which can run on a computer device or in a distributed manner on multiple computer devices, and the embodiment is not limited. In some engineering deployment scenarios, the event identification device can run on a local terminal device or an edge side device close to the work operation environment to facilitate rapid response.

[0038] In step 101, the work operation process can be any work operation process. During the work operation process, scene image data can be captured, which is used to describe the operation scene from the visual dimension. In some embodiments, the event identification device can obtain video or images in the work operation process through an image acquisition device deployed in the work operation environment. If a video is obtained, scene image data can be obtained by extracting some static frames from the video.

[0039] In step 102, the event identification device can use a target detection model to perform target detection on the scene image data to obtain attribute information of a target object contained in the scene image data. The attribute information of the target object can include position information and / or category information of the target object in the scene image data.

[0040] The target detection model is a machine learning model for identifying specific targets (such as people, vehicles, animals, actions, etc.) in images or videos, and its core task is to locate the position of the target (output bounding box), or classify the category of the target, or both locate the position of the target and classify the category of the target. In some engineering deployment scenarios, the target detection model can run on a terminal device close to the work operation environment or on an edge side device, so as to quickly respond to the call of the event identification device and meet the real-time early warning demand in the work operation scenario. Of course, in some other optional embodiments, the target detection model can also be deployed on a cloud server for remote calling by the event identification device.

[0041] The event identification device can send a detection request to the target detection model through an interface exposed by the target detection model, and the detection request carries scene image data. The target detection model can take the scene image data as input and perform feature extraction on the scene image. The feature extraction operation can include extracting features in the image using a convolutional neural network (CNN) to capture the shape, texture, and other information of the target. Based on the extracted features, a candidate region (i.e., a potential bounding box) that may contain a target object is generated, or a bounding box is directly predicted based on the extracted features. If a candidate region that may contain a target object is generated, a classification and regression algorithm can be used to classify each candidate region, i.e., to determine the category of the target object corresponding to the candidate region. At the same time, the position and size of the bounding box can be adjusted to more accurately enclose the target object. Then, a non-maximum suppression (NMS) algorithm can be used to remove redundant bounding boxes that overlap, and to retain more accurate detection results. Based on the above implementation, the target detection model can output the category label and the corresponding bounding box coordinates of each target object in the scene image data to the event identification device. The event identification device can take the category label and / or the bounding box coordinates of any target object as the attribute information of the target object, and execute step 103 based on the attribute information.

[0042] In step 103, the event identification device can determine the target work operation scenario corresponding to the scene image data according to the attribute information of the target object. In some optional embodiments, the event identification device can call a neural network model capable of understanding context semantics to determine the target work operation scenario corresponding to the scene image data according to the attribute information of the target object.

[0043] In some optional embodiments, the event recognition apparatus can use a rule engine to perform rule matching on the attribute information of the target object based on the knowledge in the field of work operation to determine the target work operation scenario corresponding to the scene image data. Optionally, different work operation scenarios have different necessary conditions to ensure the safety, efficiency and quality of the operation. For example, the necessary conditions for the production equipment installation and debugging scenario are that the scene contains target objects such as an operator, a safety helmet, protective shoes and the like. The necessary conditions for the welding operation scenario are that the scene contains target objects such as an operator, a welding operation tool, goggles, a face shield, sparks and the like. The necessary conditions for the power inspection operation scenario are that the scene contains target objects such as an operator, gloves, insulating boots and the like.

[0044] The rule engine can traverse the necessary conditions corresponding to different work operation scenarios according to the attribute information of the target object. When any work operation scenario is traversed, the rule engine can determine whether the attribute information of the target object can satisfy the necessary conditions corresponding to the work operation scenario. If yes, it is determined that the work operation scenario is the target work operation scenario corresponding to the scene image data. If not, the next work operation scenario can be traversed, and the judgment operation of the necessary conditions is performed according to the attribute information of the target object, which will not be described again.

[0045] In some optional embodiments, the necessary conditions corresponding to any work operation scenario refer to the categories of objects that should be contained in the work operation scenario. Based on this, one way in which the event recognition apparatus determines the target work operation scenario corresponding to the scene image data according to the attribute information of the target object can include: obtaining category information of the target object according to the attribute information of the target object; comparing the category information of the target object with category information of necessary objects contained in different work operation scenarios respectively; and if the category information of the target object matches the category information of necessary objects contained in any work operation scenario in the different work operation scenarios, determining that the target work operation scenario corresponding to the scene image data. For example, the necessary conditions corresponding to the cutting operation scenario are that the scene should contain objects such as a cutting machine and an operator. If the target objects identified in the scene image data include objects such as a cutting machine and an operator, it is determined that the target work operation scenario corresponding to the scene image data is the cutting operation scenario. Based on this condition matching manner, the scene image data can be accurately associated with the target work operation scenario.

[0046] After the target work operation scenario is determined, in step 104, the event recognition apparatus can use the target multi-modal model to identify the target event type corresponding to the scene image data in the target work operation scenario according to the attribute information of the target object and the knowledge information corresponding to different event types in the target work operation scenario.

[0047] In this embodiment, the target multi-modal model is a machine learning model involving multiple input sources or data types (e.g., visual, auditory, textual), which can comprehensively utilize multiple types of information for analysis and decision-making. The multi-modal model is a neural network model capable of processing and understanding multiple types of data (such as text, images, audio, video, etc.), which can capture the correlation and semantics between different modalities by fusing information of different modalities, thereby improving the understanding and interaction capabilities for complex scenarios. The target multi-modal model can be a neural network model based on the Transformer architecture. In this embodiment, the information of the text modality can at least include: text information in the detection result obtained by target detection and knowledge information corresponding to different event types in the target work scenario; the information of the image modality can include: the detection box of the target object extracted by target detection and / or the original scene image data, which is not limited in this embodiment.

[0048] In this embodiment, the target multi-modal model can be a small and medium-sized multi-modal model suitable for terminal-side deployment or edge-side deployment. Compared with large-scale multi-modal models, small and medium-sized multi-modal models have fewer parameter quantities, for example, the parameter quantity is usually between several million and several hundred million, which are suitable for deployment on devices with limited resources. The training process of the target multi-modal model will be introduced in subsequent embodiments, which will not be described here.

[0049] Among them, the event refers to one or a group of behaviors occurring in the work scenario. The event types corresponding to different work scenarios can be different. In some embodiments, the event types in the work scenario can include but are not limited to: device failure events, safety work events, and illegal work events, etc.

[0050] Among them, the knowledge information corresponding to different event types is used to describe the tools, equipment, materials, safety equipment and skill requirements of the operating personnel, and specific work environment conditions required for performing work of different event types in the work scenario. For example, in the excavation work scenario, the knowledge information corresponding to the safety work event is: the excavation work environment should have excavation tools such as excavators or shovels and excavation site markers; the operator should wear a safety helmet, gloves, and protective boots, and wear a reflective vest as needed. In the excavation work scenario, the knowledge information corresponding to the illegal work event is: the excavation work environment lacks excavation tools or excavation site markers; the operator does not wear at least one of a safety helmet, gloves, protective boots, and a reflective vest.

[0051] For example, in the aerial work scenario, the knowledge information corresponding to the safe work event is: a scaffold, a ladder or an aerial work platform should be equipped in the aerial work environment; the operator must wear a safety belt and a helmet, and ensure that the safety belt is connected to a reliable fixed point; the weather conditions such as strong wind or rainy and snowy weather may not be suitable for work. In the aerial work scenario, the knowledge information corresponding to the illegal work event is: at least one of the scaffold, the ladder and the aerial work platform is not equipped in the aerial work environment; the operator does not wear a safety belt or a helmet, or the safety belt is not connected to a reliable fixed point; or the work environment is exposed to strong wind or rainy and snowy weather.

[0052] For another example, in the electrical installation work scenario, the knowledge information corresponding to the safe work event is: the electrical installation work site should be prepared with electrical components such as wires, sockets, switches and distribution boxes; the operator should carry an electrician's tool kit (screwdriver, wire stripper, etc.), and wear insulating shoes and use insulating gloves. In the electrical installation work scenario, the knowledge information corresponding to the illegal work event is: at least one of the wires, sockets, switches and distribution boxes is not prepared in the electrical installation work site; the operator does not carry the electrician's tool kit (screwdriver, wire stripper, etc.), or the operator does not wear insulating shoes or use insulating gloves.

[0053] In this embodiment, the knowledge information corresponding to different event types can be obtained by the event recognition device through retrieval augmented generation (RAG) from a specified knowledge base. The specified knowledge base stores standardized knowledge information corresponding to different event types in different work service scenarios. The knowledge information can be obtained by sinking related documents such as industry standards and specifications. In some embodiments, the event recognition device can obtain input data of the target multi-modal model according to at least the attribute information of the target object and the knowledge information corresponding to different event types in the target work scenario. The target multi-modal model can understand the attribute information of the target object and the knowledge information corresponding to different event types in the target work scenario by combining the learned semantic understanding ability, so as to recognize the target event type corresponding to the scene image data in the target work scenario.

[0054] In this embodiment, the target detection model based on computer vision is combined with the target multi-modal model, the advantages of the target detection model in fine-grained target detection and the powerful ability of the multi-modal model in advanced semantic understanding and reasoning are fully played, and more accurate event recognition is realized. Secondly, the target detection model is used as the pre-model of the target multi-modal model, so that the target multi-modal model can focus on advanced semantic understanding and reasoning, which is beneficial to reducing the parameter quantity of the target multi-modal model, and can significantly reduce the engineering landing complexity while ensuring the semantic understanding ability, thereby facilitating the engineering deployment of the target multi-modal model. In addition, the target detection model and the target multi-modal model can be optimized independently, which is convenient for flexible version iteration.

[0055] In some optional embodiments, the multi-modal model is trained on a data set in the field of engineering operation, and the optional training method will be exemplarily described below.

[0056] Optionally, in the data collection and preprocessing stage, a data set containing multiple modalities can be collected from the engineering operation scene, such as video data, picture data and text data corresponding to different operation scenes shot by the engineering operation scene. The text data in any operation scene can be used to describe the concept, definition, identification rule, knowledge information, etc. of different event types in the operation scene. For each type of data, preprocessing can be performed separately, which can include text segmentation, image normalization, audio feature extraction, etc.

[0057] In the encoding stage, a special encoder can be used to convert data of different modalities into a unified representation form. For example, a Transformer or other encoder can be used to encode text data, a convolutional neural network (CNN) or visual Transformer (ViT) can be used to encode image data, and a time series model (such as RNN, WaveNet) can be used to extract the spectral features of audio data.

[0058] In the cross-modal alignment and fusion stage, specific mechanisms (such as attention mechanism, contrastive learning) can be used to align the features of different modalities, so that the features of different modalities have comparability in the shared semantic space. After aligning the features of different modalities, the multi-modal features can be fused to form a joint representation, and the fused multi-modal features can be input into the multi-modal model.

[0059] In the model training stage, the task target can be designed as an event type identification task, and the loss function can be defined. After obtaining the prediction result of the multi-modal model for the input data, the prediction loss of the multi-modal model can be calculated according to the loss function. Then, the model parameters can be optimized with the goal of minimizing the prediction loss until the prediction loss converges to a specified range.

[0060] In some optional embodiments, the target multi-modal model can be obtained by compressing a pre-trained large-scale multi-modal model. A multi-modal large model refers to a large-scale model capable of processing and understanding data from multiple input types. These inputs can include text, images, audio, video, etc. The multi-modal large model has strong language understanding ability and can integrate other modal information. Optionally, the compressed target multi-modal model can be deployed on a terminal device or an edge device.

[0061] In some optional embodiments, the target multi-modal model can be obtained by compressing a pre-trained large-scale multi-modal model. A multi-modal large model refers to a large-scale model capable of processing and understanding data from multiple input types. These inputs can include text, images, audio, video, etc. The multi-modal large model has strong language understanding ability and can integrate other modal information. Optionally, the compressed target multi-modal model can be deployed on a terminal device or an edge device.

[0062] Optionally, the knowledge distillation technique can be used to transfer the knowledge of the multi-modal large model to obtain the target multi-modal model. The number of model parameters supported by the multi-modal large model is not limited by the embodiments of the present application.

[0063] Specifically, a multi-modal large model in a general field can be selected as a teacher model, which is usually trained on a large-scale dataset and performs well on multiple tasks. In addition, a small or medium-sized model can be designed as a student model, which has fewer parameters and lower computational complexity, so as to be deployed in a resource-limited environment. In order to effectively distill, relevant data in the construction work scene can be collected to obtain a field training dataset, so as to ensure that the student model can learn the knowledge in the construction work field. During the knowledge transfer process, the teacher model can be used to predict the training samples to obtain the "soft" labels of each sample, which not only contains the correct class information, but also contains the confidence scores of the teacher model for other classes. During the training of the student model, the cross-entropy loss between the predicted labels of the student model and the true labels of the training samples can be minimized, and the difference between the predicted labels output by the student model and the soft labels provided by the teacher model can also be minimized. After training, the student model obtained by knowledge transfer can be further fine-tuned using the training dataset in the construction work scene to obtain a target multi-modal model more suitable for the construction work field.

[0064] Based on this implementation, the knowledge of the large model can be effectively transferred to the small model, thereby obtaining a domain-specific model with certain precision and capable of running in a resource-limited environment, which can improve the deployability of the target multi-modal model and meet the engineering deployment requirements. On the other hand, the small model focuses on a specific task domain and can provide more accurate and efficient recognition results.

[0065] In some optional embodiments, before event recognition is performed by using the target multi-modal model, the event recognition apparatus can further determine whether the recognition of the event type corresponding to the scene image data strongly depends on the target multi-modal model. If the target multi-modal model is strongly depended on, the target multi-modal model is used to recognize the target event type corresponding to the scene image data in the target work scene. Otherwise, other algorithms can be used to recognize the target event type corresponding to the scene image data in the target work scene. The other algorithms refer to algorithms with smaller parameter quantity, smaller calculation quantity, and smaller required resource quantity than the target multi-modal model, such as a logical judgment algorithm or a decision tree algorithm. The following will be exemplarily described.

[0066] Optionally, after the target work scene corresponding to the scene image data is determined, the event recognition apparatus can determine whether the target event type corresponding to the scene image data in the target work scene can be determined according to the attribute information of the target object according to the event recognition rule of the target work scene. The event recognition rule refers to a set of pre-defined logical conditions for describing the conditions on which different event types in the target work scene are recognized.

[0067] In some optional embodiments, the event recognition rule is constructed based on the category of the object and / or the relationship between the objects in the target work scene. For example, the safety work event recognition rule in the tractor work scene includes "operator" and "tractor". The safety work event recognition rule in the welding work scene is that the work scene contains "operator", "cutting machine", "safety glasses", "protective mask", and the distance between "operator" and "cutting machine" is greater than 50 cm, and "operator" needs to wear "safety glasses" and "protective mask".

[0068] Optionally, if the event recognition rule of the target work scene depends on the category of the object, the event recognition apparatus can determine that the target event type corresponding to the scene image data in the target work scene can be determined according to the attribute information of the target object. For example, the safety work event recognition rule in the tractor work scene only depends on the category of the object and does not depend on the relationship between the objects, and the event recognition apparatus can directly determine the target event type corresponding to the scene image data in the tractor work scene according to the attribute information of the target object.

[0069] Optionally, if the event identification rule of the target work scene depends on the category of the object and the relationship between the objects, it is determined that the target event type corresponding to the scene image data in the target work scene cannot be determined according to the attribute information of the target object. For example, the safety work event identification rule in the welding work scene depends not only on the category of the object but also on the relationship between the objects, and the event identification device can determine that the target event type corresponding to the scene image data in the tractor work scene cannot be directly determined according to the attribute information of the target object. Based on the above embodiment, the logical judgment method can be used to judge whether the target event type can be directly determined according to the attribute information of the target object based on the event identification rule of the target work scene. Compared with the neural network model, the logical judgment method is relatively light, and the consumption of computing resources is reduced.

[0070] Optionally, if the target event type corresponding to the image data in the target work scene cannot be determined according to the attribute information of the target object, the target event type corresponding to the scene image data in the target work scene is identified according to the attribute information of the target object and the knowledge information corresponding to different event types in the target work scene. Optionally, if the target event type corresponding to the scene image data in the target work scene can be determined according to the attribute information of the target object, the event identification device can output the target event type. Specifically, the category of the target object can be determined according to the attribute information of the target object, and the category of the target object is matched with the necessary object category relied on by different event types in the target work scene. If the category of the target object is consistent with the necessary object category relied on by any event type in the target work scene, the event type is determined as the target event type. For example, in the tractor work scene, the necessary object category relied on by the safety work event includes the operator and the tractor, and the necessary object category relied on by the illegal work event includes the operator or the tractor. If the category of the target object includes the operator and the tractor, it is determined that the target event type corresponding to the scene image data in the tractor work scene is the safety work type; otherwise, if the category of the target object does not include the operator or the tractor, it is determined that the target event type corresponding to the scene image data in the tractor work scene is the illegal work type.

[0071] That is, in the case where the target event type can be directly determined according to the attribute information of the target object, the calling of the target multi-modal model is not needed, which can improve the overall event identification efficiency on the one hand, and effectively reduce the unnecessary calling of the target multi-modal model on the other hand, thereby reducing the running frequency of the target multi-modal model and reducing the resource consumption.

[0072] In some optional embodiments, the target multi-modal model is a multi-modal model, and the attribute information of the target object includes coordinate information and category information of the target object. Accordingly, one way in which the event recognition apparatus identifies a target event type corresponding to the scene image data in the target work scenario by using the target multi-modal model according to the attribute information of the target object and knowledge information corresponding to different event types in the target work scenario can include: cropping the scene image data according to the coordinate information of the target object to obtain a scene subgraph corresponding to the target object, and constructing a target prompt word according to the scene subgraph, the coordinate information and the category information of the target object, and the knowledge information corresponding to different events in the target work scenario. In this embodiment, a prompt word template can be preset, which can include a task type field, a knowledge information field, a scene subgraph field of the target object, a coordinate field, and a category field, and the like. The event recognition apparatus can set the value of the task type field to an event type identification task, and fill the above information into the fields of the prompt word template to obtain the target prompt word. The event recognition apparatus can call the target multi-modal model according to the target prompt word to identify the target event type corresponding to the scene image data in the target work scenario by using the target multi-modal model.

[0073] In this implementation, the input data of the target multi-modal model not only includes text data but also includes image data such as scene subgraphs, so that the target multi-modal model can not only understand the text description but also more accurately identify the event type by analyzing the logical relationship and context information in the image. In addition, inputting the scene subgraph into the target multi-modal model can prevent errors from being directly passed to the inference stage of the target multi-modal model when the target detection model makes errors in the visual feature extraction stage, which is conducive to reducing error accumulation.

[0074] Optionally, when the event recognition apparatus calls the target multi-modal model according to the target prompt word, the event recognition apparatus can input the target prompt word into the target multi-modal model.

[0075] In this target multimodal model, the coordinate and category information of multiple objects within the target object are obtained from the target prompt, and semantic features of the scene subgraphs of each of the multiple objects are extracted. In some embodiments, if the target multimodal model is based on the Transformer architecture or a variant thereof, the target multimodal model can capture semantic information in the scene subgraphs through an attention mechanism. In other embodiments, the target multimodal model may include a convolutional neural network or a visual transformer, and can extract semantic features in the scene subgraphs based on the convolutional neural network or the visual transformer. Logical relationships between the multiple objects are identified based on their respective coordinate, category, and semantic features. These logical relationships may include, but are not limited to, holding, traction, wearing, or usage relationships. Specifically, the target multimodal model can extract high-dimensional semantic features from the scene subgraph of each object, and determine the logical relationships between different objects based on the high-dimensional semantic features and the coordinate information of each object. For example, if the coordinates of the operator's head and safety helmet are used to determine that the head and safety helmet overlap in position, and the feature matching degree of the scene sub-image of the head and the scene sub-image of the safety helmet is high, then the relationship between the head and the safety helmet can be determined to be a wearing relationship.

[0076] The target multimodal model can match the knowledge information corresponding to different event types based on the logical relationship between the multiple objects, obtain the target knowledge information that is adapted to the logical relationship between the multiple objects, and take the event type corresponding to the target knowledge information as the event type corresponding to the scene image data in the target operation scene.

[0077] For example, if the target work scenario is a mining operation scenario, the target multimodal model can match the knowledge information corresponding to safety operation events and violation operation events in the mining operation scenario based on the logical relationships between the multiple objects. If the logical relationship between the multiple objects is: the operator (object a1) wears (relation r1) a safety helmet (object a2), gloves (object a3), and protective boots (object a4), and wears (relation r2) a reflective vest (object a5), then the event type corresponding to the scene image data in the mining operation scenario is determined to be a safety operation event.

[0078] For example, if the target work scenario is a high-altitude work scenario, the target multimodal model can match the knowledge information corresponding to safe work events and illegal work events in the high-altitude work scenario based on the logical relationships between the multiple objects. If the logical relationship between the multiple objects is: the operator (object b1) wears (relation r3) a safety belt (object b2) and a helmet (object b3), but the safety belt (object b2) is not connected (relation r4) to a reliable fixed point (object b4), then the event type corresponding to the scene image data in the mined work scenario is determined to be an illegal work event.

[0079] Based on the above embodiments, the multi-modal target multi-modal model can understand the logical relationship between multiple objects according to the input data, and accurately map the input data to the corresponding event type according to the logical relationship and the knowledge information of different events, realizing the automatic understanding of complex work scenes.

[0080] The following will be combined Figure 2 Take the identification of abnormal events in the power safety monitoring work scene as an example to further illustrate the event identification method provided by the embodiments of the present application.

[0081] As Figure 2 shown, in the step of inputting images, image data taken in the power inspection process can be obtained. In the step of target detection, a target detection model can be used to identify target objects in the image, such as key equipment (such as welding equipment, safety helmets), workers and their actions. In the step of judging the associated scene, whether the current work scene conforms to a certain common event mode, such as a device failure event or a violation operation event, can be determined according to the information of the target objects detected in the previous step. In the step of event explicit judgment, if the event type cannot be determined directly based on the results of target detection, the steps of scene subgraph generation and prompt word construction are performed. In the steps of scene subgraph generation and prompt word construction, a scene subgraph of the target object is further generated, and a prompt word is generated based on the information of the scene subgraph of the target object, the category, the coordinates, and the work scene to which the target object belongs. In the judgment step of the multi-modal model, the generated prompt word can be input into the multi-modal model for event type judgment, and the judgment result output by the multi-modal model can be obtained. If it is determined that the event type is a violation event, the relevant personnel can be prompted to handle it in a timely manner.

[0082] In this embodiment, the target detection model based on computer vision is combined with the multi-modal model, which fully takes advantage of the advantages of the target detection model in fine-grained target detection and the powerful ability of the multi-modal large model in cross-modal understanding and reasoning, and realizes more accurate event identification. Secondly, the multi-modal model can be a multi-modal large model with less parameters, which can reduce the consumption of computing resources while ensuring the understanding and reasoning effect, and improve the real-time performance and deployability of the system. In addition, in this embodiment, by performing the steps of associated scene judgment and event explicit judgment, unnecessary calls to the multi-modal model can be effectively reduced, and the overall identification efficiency can be improved.

[0083] It should be noted that the execution subject of each step of the method provided in the above embodiments can be the same device, or the method can also be executed by different devices as the execution subject. For example, the execution subject of steps 101 to 104 can be device A; for another example, the execution subject of steps 101 and 102 can be device A, and the execution subject of step 103 can be device B; and the like.

[0084] In addition, in some of the processes described in the above embodiments and the accompanying drawings, a plurality of operations appearing in a certain order are included, but it should be clearly understood that these operations can be executed in the order appearing in the text or in parallel, and the serial numbers of the operations such as 101, 102, etc. are only used to distinguish different operations, and the serial numbers themselves do not represent any execution order. In addition, these processes can include more or fewer operations, and the operations can be executed in sequence or in parallel. It should be noted that the "first", "second", etc. described in the text are used to distinguish different messages, devices, modules, etc., and do not represent the order, nor do "first" and "second" represent different types.

[0085] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present application are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant national and regional laws, regulations and standards, and provide corresponding operation portal for user to choose authorization or refusal.

[0086] Figure 3 The structure of the electronic device provided by an example embodiment of the present application is shown, and the electronic device can run the event recognition device provided by the above embodiments. As shown in the figure, Figure 3 The electronic device includes a memory 301 and a processor 302.

[0087] The memory 301 is used to store computer programs and can be configured to store other various data to support operations on the electronic device. Examples of these data include instructions for any application or method operating on the electronic device, data structures, contact data, phonebook data, messages, pictures, videos, etc.

[0088] The processor 302 is coupled with the memory 301 and is configured to execute a computer program in the memory 301 to acquire scene image data in a work operation process, perform target detection on the scene image data by using a target detection model to obtain attribute information of a target object contained in the scene image data, determine a target work scene corresponding to the scene image data according to the attribute information of the target object, and identify a target event type corresponding to the scene image data in the target work scene according to the attribute information of the target object and knowledge information corresponding to different event types in the target work scene by using a target multi-modal model.

[0089] Optionally, when determining the target work scene corresponding to the scene image data according to the attribute information of the target object, the processor 302 is specifically configured to acquire category information of the target object according to the attribute information of the target object, compare the category information of the target object with category information of necessary objects contained in different work scenes respectively, and determine any work scene in the different work scenes as the target work scene corresponding to the scene image data if the category information of the target object matches the category information of the necessary objects contained in the any work scene.

[0090] Optionally, when identifying the target event type corresponding to the scene image data in the target work scene according to the attribute information of the target object and the knowledge information corresponding to different event types in the target work scene by using the target multi-modal model, the processor 302 is specifically configured to determine whether the target event type corresponding to the scene image data in the target work scene can be determined according to the attribute information of the target object according to an event identification rule of the target work scene, and identify the target event type corresponding to the scene image data in the target work scene according to the attribute information of the target object and the knowledge information corresponding to different event types in the target work scene by using the target multi-modal model if the determination result is no.

[0091] Optionally, the processor 302 is further configured to determine a category of the target object according to the attribute information of the target object if the target event type corresponding to the scene image data in the target work scene can be determined according to the attribute information of the target object, match the category of the target object with necessary object categories relied on by different event types in the target work scene respectively, and determine any event type in the target work scene as the target event type and output the target event type if the category of the target object is consistent with a necessary object category relied on by the any event type in the target work scene.

[0092] Optionally, when the processor 302 determines whether the target event type corresponding to the scene image data in the target work scene can be determined according to the attribute information of the target object according to the event recognition rule of the target work scene, the processor 302 is specifically configured to: if the event recognition rule of the target work scene depends on the category of the object, it is determined that the target event type corresponding to the scene image data in the target work scene can be determined according to the attribute information of the target object; if the event recognition rule of the target work scene depends on the category of the object and the relationship between the objects, it is determined that the target event type corresponding to the scene image data in the target work scene cannot be determined according to the attribute information of the target object.

[0093] Optionally, the attribute information of the target object includes coordinate information and category information of the target object; when the processor 302 identifies the target event type corresponding to the scene image data in the target work scene by using the target multi-modal model according to the attribute information of the target object and the knowledge information corresponding to different event types in the target work scene, the processor 302 is specifically configured to: crop the scene image data according to the coordinate information of the target object to obtain a scene subgraph corresponding to the target object; construct a target prompt word according to the scene subgraph, the coordinate information and the category information of the target object, and the knowledge information corresponding to different events in the target work scene; and call the target multi-modal model according to the target prompt word to identify the target event type corresponding to the scene image data in the target work scene by using the target multi-modal model.

[0094] Optionally, when the processor 302 calls the target multi-modal model according to the target prompt word to identify the target event type corresponding to the scene image data in the target work scene by using the target multi-modal model, the processor 302 is specifically configured to: input the target prompt word into the target multi-modal model; in the target multi-modal model, obtain the coordinate information and the category information of each of the plurality of objects in the target object from the target prompt word, and extract semantic features of a scene subgraph of each of the plurality of objects; identify the logical relationship between the plurality of objects according to the coordinate information, the category information and the semantic features of each of the plurality of objects; match the knowledge information corresponding to the different event types according to the logical relationship between the plurality of objects to obtain target knowledge information adapted to the logical relationship between the plurality of objects; and determine the event type corresponding to the target knowledge information as the event type corresponding to the scene image data in the target work scene.

[0095] Optionally, the target multi-modal model is obtained by lightweight compression of a pre-trained multi-modal large model, and the target multi-modal model is deployed on a terminal device or an edge side device.

[0096] Further, as shown in Figure 3 the electronic device further includes a communication component 303, a power supply component 304, a display component 305, an audio component 306, and other components. Figure 3 Some components are only schematically shown in the electronic device, and it does not mean that the electronic device only includes Figure 3 the components shown. Figure 3 In the embodiment, the components in the dashed box are optional components, rather than mandatory components, and the specific components can be determined according to the product form of the electronic device. The electronic device in the embodiment can be implemented as a terminal device such as a desktop computer, a notebook computer, a smart phone, or an IOT device, or a server device such as a general server, a cloud server, or a server array. If the electronic device in the embodiment is implemented as a terminal device such as a desktop computer, a notebook computer, a smart phone, etc., it can include Figure 3 the components in the dashed box in the embodiment; if the electronic device in the embodiment is implemented as a server device such as a general server, a cloud server, or a server array, it can not include Figure 3 the components in the dashed box in the embodiment.

[0097] The memory 301 can be implemented by any type of volatile or nonvolatile storage devices or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0098] The communication component 303 is configured to facilitate wired or wireless communication between the device where the communication component is located and other devices. The device where the communication component is located can access a wireless network based on a communication standard, such as a 2G (e.g., Global System for Mobile Communications (GSM)), a 3G (e.g., Wideband Code Division Multiple Access (WCDMA)), a 4G (e.g., Long Term Evolution (LTE)), a 4G+ (e.g., LTE-Advanced (LTE-A)), or a 5G (5th Generation Mobile Communication Technology), or a combination thereof. In an example embodiment, the communication component receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel.

[0099] The power component 304 is configured to supply power to various components of the device where the power component is located. The power component can include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power to the device where the power component is located.

[0100] The display component includes a screen, which can include a Liquid Crystal Display (LCD) and a Touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touch screen to receive an input signal from a user. The touch panel includes one or more touch sensors to sense a touch, a slide, and a gesture on the touch panel. The touch sensor can not only sense a boundary of a touching or a sliding action, but also detect duration and pressure related to the touching or sliding action.

[0101] The audio component can be configured to output and / or input audio signals. For example, the audio component includes a microphone (MIC) that is configured to receive an external audio signal when the device where the audio component is located is in an operating mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signal can be further stored in the memory or transmitted via the communication component. In some embodiments, the audio component also includes a speaker for outputting an audio signal.

[0102] In this embodiment, the target detection model based on computer vision is combined with the target multi-modal model, the advantages of the target detection model in fine-grained target detection and the powerful ability of the multi-modal model in advanced semantic understanding and reasoning are fully played, and more accurate event recognition is realized. Secondly, the target detection model is used as the pre-model of the target multi-modal model, so that the target multi-modal model can focus on advanced semantic understanding and reasoning, which is beneficial to reducing the parameter quantity of the target multi-modal model, can significantly reduce the engineering landing complexity while ensuring the semantic understanding ability, and thus facilitates the engineering deployment of the target multi-modal model. In addition, the target detection model and the target multi-modal model can be independently optimized, which is convenient for flexible version iteration.

[0103] Correspondingly, the embodiment of the application further provides a computer readable storage medium storing a computer program, when the computer program is executed by a processor, the processor can realize each step in the above method embodiment. The computer readable storage medium includes volatile or non-volatile or their combination, and can be removable or non-removable. Examples of the computer readable storage medium include, but are not limited to, phase-change random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital video disc (DVD) or other optical storage, magnetic cassette, magnetic tape disk storage or other magnetic storage device, or any other non-transmission medium

[0104] Correspondingly, the embodiment of the application further provides a computer program product, the computer program product includes a computer program or instructions, when the computer program or instructions are executed by a processor, the processor can realize each step in the above method embodiment. It should be understood that each process or combination of multiple processes in the above method flow can be realized by the computer program or instructions. In addition, these computer programs or instructions can be applied to the processor of a general-purpose computer, a special-purpose computer, an embedded processor or other programmable data processing device, so that the processor of the general-purpose computer, the special-purpose computer, the embedded processor or other programmable data processing device can be realized as an apparatus for realizing the corresponding functions in the above method embodiment.

[0105] It should also be noted that the terms "comprising", "comprises", "including", "includes" or any other variation thereof are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but can include other elements not expressly listed or inherent to such process, method, article, or apparatus. An element proceeded by "comprises a... " does not, without more constraints, exclude the existence of additional identical elements in the process, method, article, or apparatus that comprises the element.

[0106] The above description is merely illustrative of the application, and not restrictive. Various modifications and changes can become apparent to those skilled in the art. Incorporating any modification, equivalent substitution, improvement, etc. within the spirit and principle of the application, shall be included in the scope of the claims of the application.

Claims

1. An event recognition method characterized by, The method comprises the following steps: acquiring scene image data in a work operation process; performing target detection on the scene image data by using a target detection model to obtain attribute information of a target object contained in the scene image data, the attribute information of the target object comprising coordinate information and category information of the target object; determining a target operation scene corresponding to the scene image data according to the attribute information of the target object; identifying a target event type corresponding to the scene image data in the target operation scene by using a target multi-modal model according to the attribute information of the target object and knowledge information corresponding to different event types in the target operation scene, comprising: cropping the scene image data according to the coordinate information of the target object to obtain a scene sub-image corresponding to the target object; constructing a target prompt word according to the scene sub-image, the coordinate information and the category information of the target object, and the knowledge information corresponding to different events in the target operation scene; and calling the target multi-modal model according to the target prompt word to identify the target event type corresponding to the scene image data in the target operation scene by using the target multi-modal model; wherein the target prompt word is obtained by setting a task type field in a prompt word template to an event type identification task, and filling the scene sub-image, the coordinate information and the category information of the target object, and the knowledge information corresponding to different events in the target operation scene into a scene sub-image field, a coordinate field, a category field, and a knowledge information field of the target object in the prompt word template.

2. The method of claim 1, wherein, The method further comprises the following steps: acquiring the category information of the target object according to the attribute information of the target object; comparing the category information of the target object with category information of necessary objects contained in different operation scenes respectively; if the category information of the target object matches the category information of necessary objects contained in any operation scene in the different operation scenes, determining the any operation scene as the target operation scene corresponding to the scene image data.

3. The method of claim 1, wherein, The method further comprises the following steps: determining whether the target event type corresponding to the scene image data in the target operation scene can be determined according to the attribute information of the target object according to an event identification rule of the target operation scene; if not, identifying the target event type corresponding to the scene image data in the target operation scene by using the target multi-modal model according to the attribute information of the target object and the knowledge information corresponding to different event types in the target operation scene.

4. The method of claim 3, wherein, The method further comprises the following steps: if the target event type corresponding to the scene image data in the target operation scene can be determined according to the attribute information of the target object, determining the category of the target object according to the attribute information of the target object. Match the category of the target object with the necessary object category relied by different event types in the target job scene respectively; If the category of the target object is consistent with the necessary object category relied by any event type in the target job scene, determine the any event type as the target event type, and output the target event type.

5. The method of claim 3, wherein, According to the event identification rule of the target job scene, determine whether the target event type corresponding to the scene image data in the target job scene can be determined according to the attribute information of the target object, including: If the event identification rule of the target job scene relies on the category of the object, it is determined that the target event type corresponding to the scene image data in the target job scene can be determined according to the attribute information of the target object; If the event identification rule of the target job scene relies on the category of the object and the relationship between the objects, it is determined that the target event type corresponding to the scene image data in the target job scene cannot be determined according to the attribute information of the target object.

6. The method according to any one of claims 1 to 5, characterized in that, According to the target prompt word, the target multi-modal model is called to identify the target event type corresponding to the scene image data in the target job scene by using the target multi-modal model, including: Input the target prompt word into the target multi-modal model; In the target multi-modal model, the coordinate information and category information of each of the plurality of objects in the target object are obtained from the target prompt word, and the semantic features of the scene sub-graphs of each of the plurality of objects are extracted; According to the coordinate information, category information and semantic features of each of the plurality of objects, the logical relationship between the plurality of objects is identified; According to the logical relationship between the plurality of objects, the knowledge information corresponding to the different event types is matched respectively to obtain target knowledge information adapted to the logical relationship between the plurality of objects; The event type corresponding to the target knowledge information is determined as the event type corresponding to the scene image data in the target job scene.

7. The method according to any one of claims 1 to 5, characterized in that, The target multi-modal model is obtained by lightweight compression of a pre-trained multi-modal large model, and the target multi-modal model is deployed on a terminal device or an edge side device.

8. An electronic device, comprising: Including: Memory and processor; The memory is used to store one or more computer instructions; The processor is used to execute the one or more computer instructions to execute the steps in the method of any one of claims 1-7.

9. A computer readable storage medium storing a computer program, characterized in that, The computer program is executed by the processor to implement the event identification method of any one of claims 1-7.

10. A computer program product, characterised in that, Including: Computer program / instruction, which is executed by the processor to implement the event identification method of any one of claims 1-7. Including: Computer program / instruction, which is executed by the processor to implement the event identification method of any one of claims 1-7.

Images