A data processing method, device, program product, storage medium

Abstract

The application provides a data processing method, device, program product and storage medium. The method comprises: acquiring multi-modal data in a first area, wherein the multi-modal data comprises video data, sensor data and text data; fusing the multi-modal data to generate a multi-modal fusion vector; determining an image sub-vector and a sensor semantic sub-vector according to the multi-modal fusion vector; determining a first text vector in the text data according to the sensor semantic sub-vector, wherein the first text vector is related to the sensor semantic sub-vector; and constructing a training data set according to the image sub-vector, the sensor semantic sub-vector and the first text vector; wherein the training data set is used for training a first recognition model, and the first recognition model is used for identifying a plurality of types of behaviors in the first area and performers of the plurality of types of behaviors.

Description

Technical Field

[0001] This application relates to artificial intelligence technology, and more particularly to a data processing method, device, program product, and storage medium. Background Technology

[0002] Multimodal violation analysis refers to the use of artificial intelligence technology to identify, classify, and issue warnings about violations in public places or online environments by integrating data from multiple modalities such as video, audio, and text. It can be widely applied in fields such as security monitoring, content moderation, and intelligent transportation, aiming to improve the accuracy and scenario adaptability of violation identification and provide intelligent support for security management. However, the accuracy of existing identification methods needs improvement. Summary of the Invention

[0003] To address the aforementioned technical problems, embodiments of this application provide a data processing method, apparatus, program product, and storage medium.

[0004] The data processing method provided in this application includes: Acquire multimodal data within a first region, the multimodal data including video data, sensor data, and text data; The multimodal data is fused to generate a multimodal fusion vector; The image sub-vector and sensor semantic sub-vector are determined based on the multimodal fusion vector; A first text vector in the text data is determined based on the sensor semantic sub-vector, and the first text vector is related to the sensor semantic sub-vector; A training data set is constructed based on the image sub-vectors, sensor semantic sub-vectors, and the first text vector; the training data set is used to train a first recognition model, which is used to identify multiple types of behaviors and the performers of the multiple types of behaviors within the first region.

[0005] The data processing apparatus provided in this application includes a processor and a memory, wherein the memory is used to store computer programs, and the processor is used to call and run the computer programs stored in the memory to perform the above-described data processing method.

[0006] This application provides a computer program product, comprising: a computer program that, when executed by a processor, implements the above-described data processing method.

[0007] The computer-readable storage medium provided in this application is used to store a computer program that causes a computer to perform the above-described data processing method.

[0008] In the technical solution of this application, multimodal data within a first region is acquired, including video data, sensor data, and text data. The multimodal data is fused to generate a multimodal fusion vector. An image sub-vector and a sensor semantic sub-vector are determined based on the multimodal fusion vector. A first text vector in the text data is determined based on the sensor semantic sub-vector, and the first text vector is related to the sensor semantic sub-vector. A training data set is constructed based on the image sub-vector, the sensor semantic sub-vector, and the first text vector. The training data set is used to train a first recognition model, which is used to recognize multiple types of behaviors and the performers of these behaviors within the first region. Thus, when acquiring training data for the first recognition model, multiple dimensions of information and their correlations are considered simultaneously, improving the recognition accuracy of the first recognition model. Attached Figure Description

[0009] The accompanying drawings, which are provided to further illustrate this application and form part of this application, illustrate exemplary embodiments of this application and are used to explain this application, but do not constitute an undue limitation of this application.

[0010] Figure 1 This is a flowchart illustrating the data processing method provided in an embodiment of this application; Figure 2 This is a flowchart illustrating the behavior analysis method provided in an embodiment of this application; Figure 3 This is a schematic diagram of the structural composition of the data processing apparatus provided in the embodiments of this application; Figure 4 This is a schematic structural diagram of a data processing device provided in an embodiment of this application; Figure 5 This is a schematic structural diagram of the chip according to an embodiment of this application. Detailed Implementation

[0011] The technical solutions of the embodiments of this application will now be described with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0012] In the following description, the term "some embodiments" refers to a subset of all possible embodiments. However, it is understood that "some embodiments" can be the same or different subsets of all possible embodiments and can be combined with each other without conflict. It should also be noted that the terms "first," "second," and "third" used in the embodiments of this application are only used to distinguish similar objects and do not represent a specific ordering of objects. It is understood that "first," "second," and "third" can be interchanged in a specific order or sequence where permissible, so that the embodiments of this application described herein can be implemented in an order other than that illustrated or described herein. The term "and / or" in this document is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. Additionally, the character " / " in this document generally indicates that the preceding and following related objects have an "or" relationship. It should also be understood that the "instruction" mentioned in the embodiments of this application can be a direct instruction, an indirect instruction, or an indication of an association relationship. For example, A instructing B can mean that A directly instructs B, for example, B can obtain information through A; it can also mean that A indirectly instructs B, for example, A instructs C, and B can obtain information through C; it can also mean that there is an association between A and B. It should also be understood that the term "correspondence" mentioned in the embodiments of this application can mean that there is a direct or indirect correspondence between the two, or that there is an association between the two, or that there is an instruction and being instructed, configuration and being configured, etc.

[0013] To facilitate understanding of the technical solutions of the embodiments of this application, the relevant technologies of the embodiments of this application are described below. The following relevant technologies are optional solutions and can be combined with the technical solutions of the embodiments of this application in any way, and they all fall within the protection scope of the embodiments of this application.

[0014] Multimodal violation analysis refers to the use of artificial intelligence technology to identify, classify, and issue warnings about violations in public places or online environments by integrating data from multiple modalities such as video, audio, and text. It can be widely used in fields such as security monitoring, content moderation, and intelligent transportation, aiming to improve the accuracy and scenario adaptability of violation identification and provide intelligent support for security management.

[0015] To address the problem of multimodal violation detection, existing technologies typically employ independent processing methods. For example, they might use convolutional neural networks to extract image features based on video analytics, or set simple threshold alarm mechanisms based on sensor data. Furthermore, some solutions attempt shallow data fusion, such as directly concatenating information from multiple sources, but fail to delve into the potential correlations between modalities. While these methods can partially address single-modal challenges, they struggle to handle complex violations due to a lack of advanced semantic integration.

[0016] The summary and analysis of the above-mentioned multimodal violation recognition technologies reveal three main shortcomings: First, existing multimodal data fusion methods are insufficient, lacking a way to integrate monitoring images, IoT sensor data, and security management text samples to train violation recognition models, resulting in low accuracy. Second, in multimodal data fusion, existing technologies lack a method for deep integration of image and IoT sensor data. During cross-modal data alignment, the relationships between different modal data are isolated, leading to low accuracy in violation recognition models. Third, in monitoring image feature extraction, image features are extracted only through image rotation and scaling operations, lacking a feature extraction method that integrates global and local image features. This results in an incomplete understanding of image feature vector representation, further contributing to low accuracy in violation recognition models.

[0017] Therefore, how to maximize the accuracy of behavior recognition becomes a problem that needs to be considered. To this end, the following technical solutions are proposed according to embodiments of this application.

[0018] Figure 1 This is a flowchart illustrating the data processing method provided in an embodiment of this application, as shown below. Figure 1 As shown, the data processing method includes the following steps: Step 101: Acquire multimodal data within the first region. The multimodal data includes video data, sensor data, and text data.

[0019] In some implementations, the first region is the region to be identified by the first network model.

[0020] For example, if the first area is a computer room and the first network model is used to identify violations, then the first network model is used to identify violations in the computer room and the executors of the violations.

[0021] In some implementations, acquiring multimodal data within a first region includes: Video data within the first area is collected from the camera equipment within the first area; Sensor data within the first area is collected from sensor devices within the first area; Collect text data within the first region based on the text information within the first region.

[0022] In some implementations, the camera device can be a high-definition camera, which captures video data within a first area. Specifically, the video data is captured by a high-definition camera, stored and transmitted using H.264 encoding format, and has a resolution of 1024×1024 pixels to ensure clear capture of personnel movements and equipment status.

[0023] In some implementations, the sensor device includes a temperature sensor, a humidity sensor, and a hazardous gas sensor. Temperature data within the first area is collected by the temperature sensor within the first area; humidity data within the first area is collected by the humidity sensor within the first area; and hazardous gas data within the first area is collected by the hazardous gas sensor within the first area. The temperature data, humidity data, and hazardous gas data are sensor data.

[0024] It is understood that the sensor devices in the first area are not limited to temperature sensors, humidity sensors, and harmful gas sensors; the specific sensor devices can be determined according to the actual situation.

[0025] In some implementations, the sensor data is in CSV or JSON format for easier subsequent processing.

[0026] In some implementations, the text data in the first area includes all text data within the first area, including both paper and electronic text data. The text data includes management documents, rule documents, and attendance records, etc. This application does not specifically limit the specific text data.

[0027] For example, if the first area is a computer room in a certain location, then the text materials include the safety management regulations and computer room inspection sign-in documents for all computer rooms in that area. Such texts include "Open flames are prohibited in the computer room" and "If the temperature in the computer room exceeds 35 degrees Celsius, personnel must evacuate safely."

[0028] It is understandable that when the first area is a computer room, the first network model is used to identify violations in the computer room, and the multimodal data comes from video surveillance systems, IoT sensor environmental detection devices, and area operation safety management texts within the violation identification area.

[0029] Step 102: Fuse the multimodal data to generate a multimodal fusion vector.

[0030] In some implementations, fusing multimodal data to generate a multimodal fusion vector includes: The video data is input into the second network model, and the second network model outputs an initial set of multi-scale feature maps based on the video data. Determine the multi-scale fused feature map set based on the initial multi-scale feature map set; The multimodal fusion vector is determined based on the multiscale fusion feature map set and sensor data.

[0031] It is understandable that the acquired video data includes N frames of images, where N is a positive integer greater than or equal to 1.

[0032] In some implementations, the second network model is ResNet50. After the acquired video data within the first region is input into ResNet50, ResNet50 outputs multiple initial multi-scale feature maps corresponding to each frame of the input video data. These initial multi-scale feature maps are the feature maps output by each stage of the ResNet50 model, and the feature maps output by each stage constitute the initial multi-scale feature map set. Then, the initial multi-scale feature map sets are fused to obtain a multi-scale fused feature map set. The multi-scale fused feature map set includes new feature maps of global and detailed features. Finally, the multi-scale fused feature map set is fused with sensor data to determine the multimodal fusion vector.

[0033] In some implementations, the input video frame image has a resolution of 1024×1024. The ResNet50 backbone network outputs five scale feature maps (C1-C5), with initial multi-scale feature maps at resolutions of 256×256, 128×128, 64×64, 32×32, and 16×16 for each stage. The first stage contains low-level visual features such as edges, textures, and colors; the second stage contains mid-level features such as simple shapes and local structures; the third stage contains mid-to-high-level features such as the overall shape of components; the fourth stage contains high-level features such as target category information; and the fifth stage includes the highest-level features such as global semantic features. Thus, compared to the input image, the scale of the resulting multi-scale feature maps gradually decreases with the convolutional operations of the network layers. This allows for the capture of targets of different sizes using different sized feature maps, effectively balancing detailed and global information.

[0034] For example, when the first area is a server room, the first-stage feature map C1 (resolution 256×256) contains feature information such as server room floor lines and equipment shell textures, which can clearly capture the color changes of equipment indicator lights and details of cable connectors, such as red alarm lights; the second-stage feature map C2 (resolution 128×128) contains feature information such as equipment panel outlines and indicator light positions; the third-stage feature map C3 (resolution 64×64) contains feature information such as complete server racks and monitor screens, which can identify the overall status of a single server rack, such as whether the rack door is closed or whether there are foreign objects; the fourth-stage feature map C4 (resolution 32×32) contains abstract features of "personnel", "switches", and "power cabinets"; the fifth-stage feature map C5 (resolution 16×16) contains scene information such as "personnel approaching power cabinets" and "abnormal equipment arrangement", which can reflect the overall layout of the entire monitoring area, such as whether personnel have entered restricted areas and whether the equipment arrangement conforms to regulations. These feature maps clearly describe the image content from fine to coarse and from local to global, providing multi-layered information for subsequent violation identification.

[0035] In some implementations, the multi-scale feature maps corresponding to each stage in the initial multi-scale feature map set are fused to obtain a multi-scale fused feature map set.

[0036] In some implementations, a set of multi-scale fused feature maps is obtained from a multi-scale network, which is used to achieve feature map fusion by combining information from the feature maps to be fused.

[0037] In some implementations, a fifth multi-scale fusion feature map is determined based on the initial multi-scale feature map output from the fifth stage of the second network model; a fourth multi-scale fusion feature map is determined based on the fifth multi-scale fusion feature map and the initial multi-scale feature map output from the fourth stage of the second network model; a third multi-scale fusion feature map is determined based on the fourth multi-scale fusion feature map and the initial multi-scale feature map output from the third stage of the second network model; a second multi-scale fusion feature map is determined based on the third multi-scale fusion feature map and the initial multi-scale feature map output from the second stage of the second network model; a first multi-scale fusion feature map is determined based on the second multi-scale fusion feature map and the initial multi-scale feature map output from the first stage of the second network model; and a set of multi-scale fusion feature maps is constructed based on the first, second, third, fourth, and fifth multi-scale fusion feature maps.

[0038] It can be understood that the first multi-scale fusion feature map, the second multi-scale fusion feature map, the third multi-scale fusion feature map, the fourth multi-scale fusion feature map, and the fifth multi-scale fusion feature map are respectively the first layer feature map, the second layer feature map, the third layer feature map, the fourth layer feature map, and the fifth layer feature map output by the multi-scale network.

[0039] In some implementations, the method for obtaining the fourth multi-scale fusion feature map is described with reference to formula (1): , (1) Here, G4 represents the fourth layer feature map output by the multi-scale network, also known as the fourth multi-scale fusion feature map, with a resolution of 64×64. G5 represents the fifth layer feature map output by the multi-scale network, also known as the fifth multi-scale fusion feature map, with a resolution of 32×32. C4 represents the fourth layer feature map output by the second network model, also known as the initial multi-scale feature map output in the fourth stage, with a resolution of 32×32. upS() uses bilinear interpolation with a scaling factor of 2 to double the size of the feature map. Conv() is a 3×3 convolution operation (stance 1, padding 1). Here, the enlarged G5 (64×64) and C4 (64×64) are weighted by mean. After fusion, the high-level semantics of G5 (such as "abnormal behavior of personnel") are combined with the detailed features of C4 (such as "detailed information of personnel"), while preserving local and detailed information.

[0040] For example, in data center monitoring, feature map (G5) provides detailed information on "whether personnel are near the power cabinet," while feature map (C4) provides detailed information on "whether personnel's hands are touching the power switch." The fused G4 can accurately identify "personnel's unauthorized operation of the power cabinet" and pinpoint the specific location of the operation, providing precise features for subsequent violation analysis. Thus, by combining high-level semantics with low-level details through multi-scale fusion, the problem of weak semantics or poor detail in single-scale feature maps is solved, providing higher-quality image features for multimodal fusion.

[0041] It is understandable that the first multi-scale fusion feature map G1 (resolution 512×512), the second multi-scale fusion feature map G2 (resolution 256×256), and the third multi-scale fusion feature map G3 (resolution 128×128) are obtained according to the aforementioned method for obtaining the fourth multi-scale fusion feature map. Here, the fifth multi-scale fusion feature map can be determined solely based on the initial multi-scale feature map output from the fifth stage, and can be referenced in formula (2): , (2) Wherein, G5 is the fifth layer feature map output by the multi-scale network, that is, the fifth multi-scale fusion feature map, with a resolution of 32×32, C5 represents the fifth layer feature map output by the second network model, with a resolution of 16×6, upS() is to use bilinear interpolation with a scaling factor of 2 to double the size of the feature map, and Conv() is a 3×3 convolution operation (stance 1, padding 1).

[0042] It is understandable that each multi-scale fused feature map in the multi-scale fused feature map set includes both global and detailed information.

[0043] In some implementations, determining a multimodal fusion vector based on a multi-scale fused feature map set and sensor data includes: Generate the first vector corresponding to the multi-scale fusion feature map set based on the multi-scale fusion feature map set; A first text set is generated based on sensor data. The first text set is then input into a large text model. The large text model generates a second vector corresponding to the sensor data based on the first text set. The first and second vectors are input into the third network model, and the third network model outputs a multimodal fusion vector based on the first and second vectors.

[0044] Here, the image and sensor data text are vectorized, and a multimodal fusion vector is generated based on the vectorized image and sensor text.

[0045] In some implementations, generating a first vector corresponding to the multi-scale fused feature map set based on the multi-scale fused feature map set includes: Each multi-scale fusion feature map in the set of multi-scale fusion feature maps is input into the second network model. The second network model generates an initial vector corresponding to each multi-scale fusion feature map based on each multi-scale fusion feature map, and determines the first vector based on the initial vector corresponding to each multi-scale fusion feature map.

[0046] In some implementations, the formula for generating the initial vector is shown in equation (3): , (3) in, For the i-th multi-scale fused feature map, For the second network model, Let be the initial vector of the i-th multi-scale fused feature map. .

[0047] In some implementations, the first vector is determined based on the initial vector corresponding to each multi-scale fused feature map in the manner shown in equation (4): , (4) in, Let be the first vector. These are the initial vectors corresponding to G1-G5.

[0048] In some implementations, the sensor data is used to generate a corresponding first set of files. For example, the text generated from the gas sensor data is "The real-time CO2 concentration in the monitored area is 20%", "The real-time temperature in the monitored area is 20 degrees", and "The real-time humidity in the monitored area is 60%".

[0049] In some implementations, the text from the first text set is concatenated and then input into a large text model to vectorize the text, generating a second vector. Here, the second vector is a vector with dimension 512. .

[0050] In some implementations, the third network model is a fusion model. A first vector obtained by vectorizing the image and a second vector obtained by vectorizing the sensor data are input into the fusion model. The fusion model uses multi-head self-attention to fuse the multimodal features of the image and sensor data, resulting in a multimodal fusion vector. This is used for subsequent behavior analysis. Here, .

[0051] In some embodiments, the method further includes: acquiring a first historical vector, a second historical vector, and a multimodal fusion historical vector; constructing a training set for the fusion model based on the first historical vector, the second historical vector, and the multimodal fusion historical vector; and training the fusion model using the training set. Here, the first historical vector is a vector obtained from images in historical sample data through the aforementioned process of obtaining the first vector; the second historical vector is a vector obtained from sensor data through the aforementioned process of obtaining the second vector; and the multimodal fusion historical vector is a vector obtained by fusing the first historical vector and the second historical vector.

[0052] In this way, potential correlations between different modalities of data can be captured, such as the relationship between operator actions and sensor environmental conditions. Through multi-head attention mechanisms, the model can extract features from different perspectives, improving the accuracy of behavior recognition.

[0053] Step 103: Determine the image sub-vector and sensor semantic sub-vector based on the multimodal fusion vector.

[0054] Here, the multimodal fusion vector Based on the correlation of the attention head, it is further split into image sub-vectors and sensor semantic sub-vectors.

[0055] In some implementations, the third network model includes multiple attention heads; determining image sub-vectors and sensor semantic sub-vectors based on the multimodal fusion vector includes: Obtain the first correlation degree between each of the multiple attention heads and the first vector; Obtain the second correlation degree between each of the multiple attention heads and the second vector; The input vector associated with each attention head is determined based on the first correlation degree and the second correlation degree. The input vector is either the first vector or the second vector. Image subvectors and sensor semantic subvectors are determined based on the input vector and multimodal fusion vector associated with each attention head.

[0056] In some implementations, the multimodal fusion vector is formed by concatenating the outputs of all attention heads in the third network model using a linear projection matrix WO. WO is a projection matrix used to quantify the contribution of each attention head to each dimension of the multimodal fusion vector. It is generated by the third network model, and different inputs will have different parameter matrices WO. The shape of WO is typically... ,in, It's the number of heads that attract attention. It is the dimension of attention head. It is the output dimension.

[0057] In some implementations, WO is expressed as shown in equation (5): , (5) Here, pay attention to the number of heads. It is 6. It has 128 dimensions. Since the dimension is 1024, the dimension of WO in equation (5) is 768×1024.

[0058] In some implementations, the first correlation degree between each multi-attention head and the first vector is determined based on the projection matrix WO.

[0059] For example, according to equation (5), the first correlation degree between attention head 1 and the first vector is calculated as shown in equation (6): , (6) in, For the first degree of attention, is the parameter in the i-th row and j-th column of the projection matrix WO.

[0060] In some implementations, the second correlation degree between each multi-attention and the second vector is determined based on the projection matrix WO.

[0061] For example, the second correlation degree between attention head 1 and the second vector is calculated as shown in equation (7): , (7) in, The second degree of relevance for attention head 1, is the parameter in the i-th row and j-th column of the projection matrix WO.

[0062] It is understandable that in the calculation of the first correlation degree between attention head 2 and the first vector and the second correlation degree with the second vector, i ranges from 129 to 256, and so on.

[0063] In some implementations, the magnitudes of the first and second relevance of each attention head are compared, and the input vector with the higher relevance is selected as the one more strongly associated with that attention head. Specifically, if If the input vector is more strongly associated with attention head 1, then the first vector is used; otherwise, the second vector is used.

[0064] In some implementations, image subvectors and sensor semantic subvectors are determined based on the projection matrix WO, the input vector associated with each attention head, and the multimodal fusion vector.

[0065] In some implementations, because the dimension of the multimodal fusion vector is the same as the number of columns in the projection matrix WO, the parameters in the k-th column of the projection matrix are used to determine the fusion vector. Determine whether the k-th dimension in the multimodal fusion vector is part of the image sub-vector or the sensor semantic sub-vector, and then determine the final image sub-vector and sensor semantic sub-vector.

[0066] Specifically, taking equation (5) as an example, if the aforementioned process determines that attention heads 1, 3, and 5 have a high degree of correlation with the image input, that is, a high degree of correlation with the first vector, and attention heads 2, 4, and 6 have a higher degree of correlation with the sensor semantic input, that is, a high degree of correlation with the second vector, then the parameters of the k-th column of the WO matrix are obtained, and the parameters corresponding to attention heads 1, 3, and 5 are compared. The sum of the parameters of self-attention points 2, 4, and 6 The sum of the parameters in the k-th column of the projection matrix is ​​used to select the largest set of parameters as the vectors with high relevance to the input. Specifically, the sum of parameters in rows 1-128, 257-384, and 513-640, and rows 129-256, 385-512, and 641-768 is calculated. If the sum of parameters for sub-attention heads 1, 3, and 5 is greater than the sum of parameters for self-attention heads 2, 4, and 6, then the k-th dimension of the multimodal fusion vector has high relevance to the first vector, and is thus used as a component of the image sub-vectors. This method is used to sequentially select image sub-vectors from each dimension of the output multimodal fusion vector. IoT sensor semantic sub-vectors .

[0067] Step 104: Determine the first text vector in the text data based on the sensor semantic sub-vector. The first text vector is related to the sensor semantic sub-vector.

[0068] In some implementations, the method further includes, before determining a first text vector in the text data based on the sensor semantic subvector: The text data is transformed into a first set of text vectors, and the first set of text vectors is stored in a first database.

[0069] In some implementations, the first database is a Faiss vector database.

[0070] In some implementations, text data is input into a fourth network model, which generates a first set of text vectors corresponding to the text data and stores the first set of text vectors in the Faiss necklace database.

[0071] In some implementations, the text data is segmented into words, and the BERT model is used to vectorize the text data into 512 dimensions on a sentence-by-sentence basis; then the vector is stored in the Faiss vector database.

[0072] In some implementations, determining a first text vector in the text data based on sensor semantic sub-vectors includes: Using sensor semantic sub-vectors as query conditions, retrieve first text vectors related to the query conditions from the first database.

[0073] In some implementations, the sensor semantic sub-vector is used as a query condition. The text vector most similar to the sensor semantic sub-vector is retrieved from the Faiss database and used as supplementary input to the first network model.

[0074] For example, if the query condition is "the temperature in the monitored area is 30 degrees", the most relevant text retrieved from the Faiss database is the text vector "the safe production temperature in the monitored area is 10-25 degrees, and an alert will be issued if it exceeds 25 degrees".

[0075] Step 105: Construct a training dataset based on the image sub-vectors, sensor semantic sub-vectors, and the first text vector; the training dataset is used to train the first recognition model, which is used to identify multiple types of behaviors and the performers of multiple types of behaviors within the first region.

[0076] In some implementations, after obtaining the image sub-vectors, sensor semantic sub-vectors, and the first text vector, a training data set is constructed and used as input to the first recognition model to train the first recognition model, which is used to identify the behavior and the actor in the first region.

[0077] It can be understood that the training dataset includes the image sub-vector, sensor semantic sub-vector, and first text vector obtained from each frame of video image in the acquired video data through the aforementioned process.

[0078] In some implementations, the network model of the first recognition model uses 512 256 A 128-layer, three-layer fully connected network is used. The first recognition model is trained iteratively using backpropagation to obtain the violation recognition model. For any input camera surveillance image, sensor data, or region text, the first recognition model can effectively identify the executor and the operation.

[0079] The technical solution of this application embodiment involves acquiring multimodal data within a first region, the multimodal data including video data, sensor data, and text data; fusing the multimodal data to generate a multimodal fusion vector; determining image sub-vectors and sensor semantic sub-vectors based on the multimodal fusion vector; determining a first text vector in the text data based on the sensor semantic sub-vector, the first text vector being related to the sensor semantic sub-vector; constructing a training data set based on the image sub-vector, sensor semantic sub-vector, and the first text vector; the training data set being used to train a first recognition model, the first recognition model being used to recognize multiple types of behaviors and the performers of the multiple types of behaviors within the first region. Thus, when acquiring the training data for the first recognition model, multiple dimensions of information and the correlation information of these multiple dimensions are considered simultaneously, improving the recognition accuracy of the first recognition model.

[0080] Based on the foregoing embodiments, taking the first area as an example of a computer room for identifying violations, the data processing method provided by the embodiments of this application will be further explained.

[0081] Figure 2 This is a flowchart illustrating the behavior analysis method provided in the embodiments of this application; as follows: Figure 2 As shown, it includes the following steps: Step 201: Multimodal data preprocessing.

[0082] The multimodal data mentioned in the embodiments of this application mainly comes from video surveillance systems, IoT sensor environmental monitoring devices, and regional operation safety management texts within the violation identification area (equivalent to the aforementioned first area), such as video surveillance systems, IoT sensor environmental monitoring devices, and regional operation safety management texts in the computer room.

[0083] Video data is captured by a high-definition camera, stored and transmitted using the H.264 encoding format, with a resolution of 1024×1024 pixels to ensure clear capture of personnel movements and equipment status.

[0084] The equipment sensor data includes parameters such as humidity, temperature, and harmful gases, and the data is in CSV or JSON format for easy subsequent processing and fusion.

[0085] Collect regional operation safety management documents, such as safety management regulations for each computer room, such as "Open flames are prohibited in the computer room" and "If the temperature in the computer room exceeds 35 degrees Celsius, personnel need to be evacuated safely."

[0086] Step 202: Feature extraction from area monitoring images.

[0087] Based on the multi-scale feature maps output by the ResNet50 network, cross-layer feature fusion is performed through bilinear interpolation amplification and 3x3 convolution weighting to generate a new feature map that simultaneously contains details and global semantics. In general, the multi-scale feature maps output by the ResNet50 network are executed, combined with a multi-scale feature fusion mechanism. The feature map size is amplified through bilinear interpolation, and weighted fusion is performed using 3x3 convolution to generate a new feature map that retains both current and upper-layer features. This addresses the shortcomings of image features lacking local and global feature fusion, resulting in incomplete representation and understanding. On one hand, a strategy of fusing the upper-layer multi-scale fused image with the current layer's interpolated amplified image is adopted to increase the hierarchical correlation dimension of feature extraction, improving the comprehensiveness and accuracy of feature representation. On the other hand, in the image feature extraction process, bilinear interpolation is applied to the output multi-scale feature maps to solve the problem of difficulty in directly fusing features at different levels in traditional extraction methods. It should be noted that the aforementioned extracted video surveillance images include N images; here, a single frame image is used as an example for further explanation. The specific steps are as follows: 1. Use the ResNet50 network to output feature maps.

[0088] Taking a frame of an image from a carrier's data center monitoring station as an example, the monitoring image captured by the aforementioned high-definition camera is input into ResNet50. The five scale feature maps (C1-C5) output by the ResNet50 backbone network are equivalent to the aforementioned initial multi-scale feature map set. Compared to the input image, the scale of the feature maps gradually decreases with the convolutional operations of the network layers. Using feature maps of different sizes captures targets of different sizes, effectively balancing detailed and global information. Based on the carrier's data center monitoring example, one feature map is output for each of the five main stages of the ResNet50 network, and their resolution and meaning are shown in Table 1. Table 1 Output of ResNet50 network in five main stages of carrier data center monitoring scenario

[0089] For example, C1 (resolution 256×256) can clearly capture the color changes of device indicator lights and the details of cable connectors, such as red alarm lights; C3 (resolution 64×64) can identify the overall status of a single server rack, such as whether the rack door is closed and whether there are foreign objects; C5 (resolution 16×16) can reflect the global layout of the entire monitoring area, such as whether personnel have entered restricted areas and whether the equipment arrangement conforms to the specifications. These feature maps clearly describe the image content from fine to coarse and from local to global, providing multi-level information for subsequent identification of violations.

[0090] 2. Multi-scale feature fusion This step, based on the C1-C5 feature maps output in step 1 above, further fuses feature maps of different scales to generate new feature maps (G1-G5) that simultaneously contain global and detailed features. The specific steps are as follows: (1) Definition of each symbol in the formula.

[0091] Taking the calculation method of G4 as an example, the specific method of multi-scale feature fusion is explained as shown in formula (1). G4 represents the 4th layer feature map output by the multi-scale network with a resolution of 64×64. G5 is the 5th layer feature map output by the multi-scale network with a resolution of 32×32. C4 is the 4th layer feature map output by ResNet50 with a resolution of 32×32. upS() is a bilinear interpolation with a scaling factor of 2 to double the size of the feature map. Conv() is a 3×3 convolution operation (stance 1, padding 1). Here, the magnified G5 (64×64) and C4 (64×64) are weighted by mean.

[0092] (2) Multi-scale feature fusion calculation process.

[0093] Processing Feature Map G5: G5 has the smallest resolution (32×32) in the multi-scale feature fusion mechanism and contains global information such as "data center area layout". Through bilinear interpolation with a scaling factor of 2, the size of G5 is enlarged from 32×32 to 64×64 to match the scale of the scaled feature map of C4 that will be fused.

[0094] Processing feature map C4: C4 is a 32×32 feature map output by ResNet (containing mid-to-high-level features such as "device type"). Through bilinear interpolation, the size of C4 is enlarged from 32×32 to 64×64 with a scaling factor of 2.

[0095] Feature fusion: The magnified G5 (64×64) and C4 (64×64) are weighted by mean. After fusion, the high-level semantics of G5 (such as "abnormal personnel behavior") are combined with the detailed features of C4 (such as "personnel detailed information"), while preserving local and detailed information.

[0096] (3) Case study of multi-scale feature fusion.

[0097] In the example of operator data center monitoring, feature map (G5) provides detailed information on "whether personnel are near the power cabinet," while feature map (C4) provides detailed information on "whether personnel's hands are touching the power switch." The fused G4 can accurately identify "personnel's unauthorized operation of the power cabinet" and pinpoint the specific location of the operation, providing precise features for subsequent violation analysis. The multi-scale feature fusion method proposed in this solution (such as the generation of G4) combines high-level semantics with low-level details through multi-scale fusion, solving the problem of weak semantics or poor detail in single-scale feature maps, and providing higher-quality image features for multimodal fusion.

[0098] (4) Generate G1, G2 and G3 according to the above method.

[0099] According to the method of multi-scale feature fusion calculation process in (2) above, G1, G2 and G3 are generated, where G4 (64×64), G3 (128×128), G2 (256×256) and G1 (512×512). The fused features include both global information and detailed information.

[0100] Step 203: Regional image and sensor device modal fusion.

[0101] Multi-scale image features are vectorized and input with sensor text vectors into a multi-head self-attention mechanism to train a cross-modal fusion model to capture potential correlations. Overall, based on monitoring area images and multi-source heterogeneous data from IoT sensors, firstly, multi-scale images are vectorized separately, and a weighted average is applied to ensure the fused vector includes both local and global image features. Simultaneously, sensor text is generated and vectorized using a large text model. Secondly, multi-head self-attention is used to fuse the image and sensor multi-modal features. The specific steps are as follows: 1. Image vectorization.

[0102] The G1-G5 feature maps output in step 202 above are transformed into 512-dimensional vectors using ResNet. Taking the G1 feature map as an example, the vectorization method is shown in formula (8): , (8) generated Using the same method, G2-G5 are successively transformed into vectorized forms using the ResNet network. , , , ,calculate , , , , The vector mean is shown in formula (4).

[0103] 2. Sensor data text vectorization.

[0104] Data from gas, temperature, and humidity sensors in the monitored area is used to generate text. For example, the text generated by the gas sensors might be "Real-time CO2 concentration in the monitored area is 20%", "Real-time temperature in the monitored area is 20 degrees Celsius", or "Real-time humidity in the monitored area is 60%". This text is then concatenated and integrated, and a large text model is used to vectorize the text, generating a 512-dimensional vector. This application does not impose specific limitations on the large text model used; it can be determined based on the actual situation.

[0105] 3. Fusion of image and sensor vectors.

[0106] Based on the historical sample data from steps 1 (image vectorization) and 2 (sensor data text vectorization) above, and using the fused vectors as training data, multi-head self-attention is used to fuse the multimodal features of images and sensors, and a fusion model is trained. The fused feature vector output by the trained fusion model has a dimension of [missing information]. Multimodal fusion vector This data is used for subsequent violation analysis. This method can capture potential correlations between different modalities, such as the relationship between operator actions and sensor environmental conditions. Through a multi-head attention mechanism, the model can extract features from different perspectives, improving the accuracy of identifying complex violations. It can be understood that the multi-modal fusion vector here is the fusion vector corresponding to the frame of image input in step 202 above.

[0107] Step 204: Dual-stream intent understanding, multimodal fusion vector feature splitting.

[0108] By utilizing attention head contribution analysis and quantifying modal correlation through a projection matrix, the fused vector is precisely decomposed into image sub-vectors and sensor semantic sub-vectors. Overall, the multimodal fusion vector output in step 203 above... , will fuse vector Based on the correlation of the attention heads, the vectors are further split into two sub-vectors: image and sensor semantic flow. First, the contribution of each attention head to each dimension of the output vector is quantified using the attention projection matrix, yielding the modality vector strongly correlated with each attention head. Second, the sum of the self-attention parameters corresponding to the projection matrix of each output vector is calculated sequentially, and the set with the largest value is selected as the output vector for that modality, thus separating the image and IoT semantic flow vectors. Image vector exist Extract image-related features, such as human behavior and the appearance of the computer room. Sensor semantic flow vectors. exist This process extracts features related to the status of IoT sensors, such as temperature and humidity in the monitored area. It addresses the drawback of isolated relationships between different modalities during cross-modal data alignment, which leads to low accuracy in violation detection models. The specific implementation method is as follows: 1. Tracing back to the attention head of image vectors and semantic flow vectors.

[0109] It is formed by concatenating the outputs of all attention heads through a linear projection WO, and it is trained according to different inputs, as shown in formula (9): , (9) in, Represents splicing, Let i be the i-th attention head, and there are a total of h attention heads.

[0110] As shown in formula (9). Each dimension is influenced by the outputs of all attention heads, but the influence varies among different attention heads. The final projection matrix WO is used to quantify the contribution of each attention head to each dimension of the final output. The shape of WO is typically... ,in It's the number of heads that attract attention. It is the dimension of attention head. This refers to the output dimension. For example, The expression can be shown in equation (5), where the number of attention heads is in equation (5). It is 6. It has 128 dimensions. It has 1024 dimensions.

[0111] 2. Calculate the influence of the attention head on the image input and the IoT text vector.

[0112] The correlation between each attention head and the image and sensor semantic input is calculated separately. Taking attention head 1 as an example, the correlation between attention head 1 and the image input and sensor semantic input are calculated separately. The correlation depends on the size of the attention parameter. The larger the parameter, the greater the correlation.

[0113] The correlation degree sim1 between attention head 1 and the image input is calculated as shown in formula (6), and the correlation degree sim2 between attention head 1 and the sensor semantic input is calculated as shown in formula (7). Compare the values ​​of sim1 and sim2, and select the input vector with the larger value as the one more strongly correlated with the attention head. Calculate the strongly correlated input vector for each attention head separately.

[0114] 3. The k-th dimension of the output vector is considered to have a high correlation with the input image or sensor semantic input vector. Assuming the final result in the preceding process is that attention heads 1, 3, and 5 have a high correlation with the image input, while attention heads 2, 4, and 6 have a higher correlation with the sensor semantic input, the k-th column of the WO matrix is ​​compared. The sum of the parameters of attention heads 1, 3, and 5 is compared with the sum of the parameters of attention heads 2, 4, and 6, and the larger sum is selected as the vector with high correlation to the input. Using this method, the output vectors are sequentially filtered to select the image. IoT sensor vector .

[0115] Step 205: Associate sensor semantic vectors with area security management text.

[0116] Sensor semantic vectors Using this as the core input, regional security management text is associated with the vector database to further supplement the data dimensions of the violation identification model.

[0117] First, the regional security management text is segmented into words, and then the BERT model is used to vectorize the text into 512-dimensional vectors, sentence by sentence. Second, the sentence vectors are stored in the Faiss vector database. Finally, the text is processed using... Using the Faiss database as the query criteria, retrieve the most similar regional security management text vectors. (Equivalent to the aforementioned first text vector) serves as a supplementary input to the violation identification model. For example, if the query condition is "the temperature in the monitored area is 30 degrees Celsius", the most relevant text retrieved from the Faiss database is "the safe production temperature in the monitored area is 10-25 degrees Celsius, and a warning will be issued if it exceeds 25 degrees Celsius".

[0118] Here, the split sensor data is associated with safety management knowledge text through Retrieval Enhancement Generation (RAG) to generate safety management text vectors.

[0119] Step 206: Train the violation detection model.

[0120] A multimodal violation recognition model is trained based on multimodal data including image subvectors, sensor semantic subvectors, and regional security management text subvectors.

[0121] Model input: as image subvectors Sensor semantic sub-vectors Regional security management text vector Historical data samples are used as input.

[0122] Network structure: The network results use 512 256 128 three-layer fully connected network.

[0123] Training method: The violation detection model is obtained by iterative training using backpropagation.

[0124] Thus, by inputting any camera surveillance images, sensor data, or area security management text into the violation identification model, the violators and their actions can be effectively identified.

[0125] This application proposes a violation analysis method based on multimodal data. First, during image feature extraction, multi-scale feature fusion technology is used to ensure that image features possess both global and local characteristics. Second, during the fusion of image and IoT sensor data, a self-attention mechanism is used to fuse the two types of modal data. Then, combining the data correlation between the attention head and the fused vector across different dimensions, the data is split into image and sensor vectors. This fusion and subsequent splitting of the multimodal vectors results in deep fusion of the split vectors, improving the accuracy of the recognition model during subsequent training. Third, the split sensor data is associated with security management knowledge text through RAG (Related Aspects of Data), generating security management text vectors. Finally, based on historical multimodal data samples, using image feature vectors, sensor status data, and security management text vectors as input, an iterative training method is used to obtain the violation recognition model. Here, RAG technology is an artificial intelligence technology that combines retrieval and generation capabilities. Its core logic is to accurately retrieve information relevant to the current task from a massive knowledge base through a retrieval module, and then combine this external knowledge with the model's own generation capabilities to improve the accuracy, reliability, and relevance of the output content. In the multimodal violation recognition scenario of this application embodiment, RAG technology is mainly used to establish the association between sensor data and safety management text: by retrieving the violation judgment rules corresponding to real-time sensor data (such as temperature, concentration of harmful gases, etc.) in the safety management text library, the static text rules are transformed into dynamic text vectors that can participate in model training, so that the violation recognition model can fully integrate safety knowledge semantic information and avoid the problem of insufficient recognition accuracy due to lack of text modality support.

[0126] Furthermore, in the multimodal violation identification process, a three-step core processing approach is adopted: multi-scale feature fusion to extract global and local image features, self-attention mechanism-driven image and sensor data fusion and splitting, and RAG-linked sensor data and security management text to generate text vectors. Based on multimodal data including image flow vectors, sensor semantic flow vectors, and security management text vectors, a multimodal violation identification model is trained, addressing the problem of insufficient multimodal data dimensionality in traditional methods and improving the accuracy of violation identification. For the multimodal data fusion process, a self-attention mechanism is introduced for deep fusion of image and sensor data. First, a multi-head self-attention mechanism is introduced to increase the dimension of capturing potential correlations between different modal data, breaking the limitation of isolated modal data relationships in traditional fusion methods. Second, in the multimodal fusion vector splitting stage, the steps of quantifying the attention head contribution using an attention projection matrix and calculating the sum of self-attention parameters to filter modal input vectors are used to construct a precise splitting logic based on attention weights, improving the accuracy of fusion vector splitting, effectively increasing the correlation dimension between image and sensor data, solving the problem of semantic separation between the two types of data in traditional fusion methods, and improving the depth and effectiveness of multimodal data fusion. In the image feature data fusion process, a strategy of fusing upper-layer multi-scale fused images with the interpolated and enlarged images of the current layer is adopted. This increases the hierarchical correlation dimension of feature extraction, breaking the limitation that single-scale features can only capture local or global information, and improving the comprehensiveness and accuracy of feature representation. In the image feature extraction process, bilinear interpolation is added to the output feature maps of five sizes, providing a unified dimensional basis for the accurate fusion of features at different scales, and solving the problem that it is difficult to directly fuse features at different levels in traditional extraction methods. By fusing samples from three dimensions—surveillance images, IoT sensor data, and security management text—a full-dimensional multimodal data processing link is constructed, enabling the violation behavior recognition model to more comprehensively understand scene information, thereby significantly improving recognition accuracy.

[0127] Figure 3 This is a schematic diagram of the structural composition of the data processing device provided in the embodiments of this application, as shown below. Figure 3 As shown, the data processing device includes: The acquisition unit 301 is used to acquire multimodal data within a first region, the multimodal data including video data, sensor data and text data; Processing unit 302 is configured to fuse the multimodal data to generate a multimodal fusion vector; determine an image sub-vector and a sensor semantic sub-vector based on the multimodal fusion vector; determine a first text vector in the text data based on the sensor semantic sub-vector, wherein the first text vector is related to the sensor semantic sub-vector; construct a training data set based on the image sub-vector, the sensor semantic sub-vector, and the first text vector; the training data set is used to train a first recognition model, wherein the first recognition model is used to recognize multiple types of behaviors and the performers of the multiple types of behaviors within the first region.

[0128] In some embodiments, the acquisition unit 301 is configured to acquire video data in the first area based on a camera device in the first area; acquire sensor data in the first area based on a sensor device in the first area; and acquire text data in the first area based on text data in the first area.

[0129] In some embodiments, the processing unit 302 is configured to input the video data into a second network model, the second network model outputting an initial multi-scale feature map set based on the video data; determining a multi-scale fusion feature map set based on the initial multi-scale feature map set; and determining the multi-modal fusion vector based on the multi-scale fusion feature map set and the sensor data.

[0130] In some embodiments, the processing unit 302 is configured to generate a first vector corresponding to the multi-scale fusion feature map set based on the multi-scale fusion feature map set; generate a first text set based on the sensor data; input the first text set into a large text model; the large text model generates a second vector corresponding to the sensor data based on the first text set; input the first vector and the second vector into a third network model; and the third network model outputs the multimodal fusion vector based on the first vector and the second vector.

[0131] In some implementations, the third network model includes multiple attention heads; the processing unit 302 is configured to obtain a first correlation degree between each of the multiple attention heads and the first vector; obtain a second correlation degree between each of the multiple attention heads and the second vector; determine an input vector associated with each attention head based on the first correlation degree and the second correlation degree, wherein the input vector is either a first vector or a second vector; and determine the image sub-vector and the sensor semantic sub-vector based on the input vector associated with each attention head and the multimodal fusion vector.

[0132] In some embodiments, the processing unit 302 is configured to convert the text data into a first set of text vectors and store the first set of text vectors in a first database before determining the first text vector in the text data based on the sensor semantic sub-vector.

[0133] In some implementations, the processing unit 302 is configured to query a first text vector related to the query condition from the first database, using the sensor semantic sub-vector as the query condition.

[0134] Those skilled in the art should understand that Figure 3 The functions of each unit in the data processing device shown can be understood by referring to the relevant description of the aforementioned method. Figure 3 The functions of each unit in the data processing device shown can be implemented by a program running on a processor or by specific logic circuits.

[0135] Figure 4 This is a schematic structural diagram of a data processing device 400 provided in an embodiment of this application. Figure 4 The data processing device 400 shown includes a processor 410, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0136] Optionally, such as Figure 4 As shown, the data processing device 400 may further include a memory 420. The processor 410 can retrieve and run computer programs from the memory 420 to implement the methods described in this embodiment.

[0137] The memory 420 can be a separate device independent of the processor 410, or it can be integrated into the processor 410.

[0138] Optionally, such as Figure 4 As shown, the data processing device 400 may also include a transceiver 430, which the processor 410 can control to communicate with other devices. Specifically, it can send information or data to other devices or receive information or data sent by other devices.

[0139] The transceiver 430 may include a transmitter and a receiver. The transceiver 430 may further include an antenna, and the number of antennas may be one or more.

[0140] The data processing device 400 can implement the corresponding processes implemented by the data processing apparatus in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0141] Figure 5 This is a schematic structural diagram of the chip according to an embodiment of this application. Figure 5 The chip 500 shown includes a processor 510, which can call and run computer programs from memory to implement the methods in the embodiments of this application.

[0142] Optionally, such as Figure 5 As shown, chip 500 may further include memory 520. Processor 510 can retrieve and run computer programs from memory 520 to implement the methods described in this embodiment.

[0143] The memory 520 can be a separate device independent of the processor 510, or it can be integrated into the processor 510.

[0144] Optionally, the chip 500 may also include an input interface 530. The processor 510 can control the input interface 530 to communicate with other devices or chips; specifically, it can acquire information or data sent by other devices or chips.

[0145] Optionally, the chip 500 may also include an output interface 540. The processor 510 can control the output interface 540 to communicate with other devices or chips, specifically, to output information or data to other devices or chips.

[0146] This chip can implement the corresponding processes implemented by the data processing device in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0147] It should be understood that the chip mentioned in the embodiments of this application may also be referred to as a system-on-a-chip, system chip, chip system, or system-on-a-chip, etc.

[0148] It should be understood that the processor in the embodiments of this application may be an integrated circuit chip with signal processing capabilities. In implementation, the steps of the above method embodiments can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components. It can implement or execute the methods, steps, and logic block diagrams disclosed in the embodiments of this application. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the methods disclosed in the embodiments of this application can be directly embodied in the execution of a hardware decoding processor, or executed by a combination of hardware and software modules in the decoding processor. The software modules can be located in random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, or other mature storage media in the art. The storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method.

[0149] It is understood that the memory in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous DRAM (SDRAM), Double Data Rate SDRAM (DDR SDRAM), Enhanced Synchronous DRAM (ESDRAM), Synchlink DRAM (SLDRAM), and Direct Rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0150] It should be understood that the above-described memory is exemplary and not a limiting description. For example, the memory in the embodiments of this application may also be static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM), and direct memory bus RAM (DR RAM), etc. That is to say, the memory in the embodiments of this application is intended to include, but is not limited to, these and any other suitable types of memory.

[0151] This application also provides a computer program product, including a computer program.

[0152] When executed by a processor, the computer program implements the corresponding processes implemented by the data processing device in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0153] This application also provides a computer-readable storage medium for storing computer programs.

[0154] The computer program causes the computer to execute the corresponding processes implemented by the data processing device in the various methods of the embodiments of this application, which will not be described in detail here for the sake of brevity.

[0155] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0156] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0157] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0158] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0159] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0160] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0161] The preferred embodiments of this application have been described in detail above with reference to the accompanying drawings. However, this application is not limited to the specific details of the above embodiments. Within the scope of the technical concept of this application, various simple modifications can be made to the technical solutions of this application, and these simple modifications all fall within the protection scope of this application. For example, the various specific technical features described in the above specific embodiments can be combined in any suitable manner without contradiction. To avoid unnecessary repetition, this application will not describe the various possible combinations separately. Furthermore, various different embodiments of this application can also be arbitrarily combined, as long as they do not violate the spirit of this application, they should also be considered as the content disclosed in this application. Moreover, without conflict, the various embodiments and / or the technical features in the various embodiments described in this application can be arbitrarily combined with the prior art, and the resulting technical solutions should also fall within the protection scope of this application.

[0162] It should be understood that in the various method embodiments of this application, the sequence number of each process does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0163] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A data processing method, characterized in that, The method includes: Acquire multimodal data within a first region, the multimodal data including video data, sensor data, and text data; The multimodal data is fused to generate a multimodal fusion vector; The image sub-vector and sensor semantic sub-vector are determined based on the multimodal fusion vector; A first text vector in the text data is determined based on the sensor semantic sub-vector, and the first text vector is related to the sensor semantic sub-vector; A training data set is constructed based on the image sub-vectors, sensor semantic sub-vectors, and the first text vector; the training data set is used to train a first recognition model, which is used to identify multiple types of behaviors and the performers of the multiple types of behaviors within the first region.

2. The method according to claim 1, characterized in that, The acquisition of multimodal data within the first region includes: Video data within the first area is collected from the camera devices within the first area; Sensor data within the first area is collected from sensor devices within the first area; Collect text data within the first region based on the text information within the first region.

3. The method according to claim 1, characterized in that, The step of fusing the multimodal data to generate a multimodal fusion vector includes: The video data is input into the second network model, and the second network model outputs an initial set of multi-scale feature maps based on the video data. A multi-scale fused feature map set is determined based on the initial multi-scale feature map set; The multimodal fusion vector is determined based on the multiscale fusion feature map set and the sensor data.

4. The method according to claim 3, characterized in that, Determining the multimodal fusion vector based on the multi-scale fusion feature map set and the sensor data includes: Generate a first vector corresponding to the multi-scale fused feature map set based on the multi-scale fused feature map set; A first text set is generated based on the sensor data, and the first text set is input into a large text model. The large text model generates a second vector corresponding to the sensor data based on the first text set. The first vector and the second vector are input into a third network model, and the third network model outputs the multimodal fusion vector based on the first vector and the second vector.

5. The method according to claim 4, characterized in that, The third network model includes multiple attention heads; the step of determining the image sub-vector and sensor semantic sub-vector based on the multimodal fusion vector includes: Obtain the first correlation degree between each of the plurality of attention heads and the first vector; Obtain the second correlation degree between each of the plurality of attention heads and the second vector; The input vector associated with each attention head is determined based on the first correlation degree and the second correlation degree, wherein the input vector is either the first vector or the second vector; The image subvector and the sensor semantic subvector are determined based on the input vector associated with each attention head and the multimodal fusion vector.

6. The method according to any one of claims 1 to 5, characterized in that, Before determining the first text vector in the text data based on the sensor semantic sub-vector, the method further includes: The text data is converted into a first set of text vectors, and the first set of text vectors is stored in a first database.

7. The method according to claim 6, characterized in that, Determining the first text vector in the text data based on the sensor semantic sub-vector includes: Using the sensor semantic sub-vector as the query condition, a first text vector related to the query condition is queried from the first database.

8. A data processing device, characterized in that, include: A processor and a memory for storing a computer program, the processor for calling and running the computer program stored in the memory to perform the method as described in any one of claims 1 to 7.

9. A computer program product, characterized in that, include: A computer program that, when executed by a processor, implements the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store a computer program that causes a computer to perform the method as described in any one of claims 1 to 7.

Images