Multi-dimensional feature information monitoring method and device, computer equipment and storage medium

Abstract

The invention relates to the technical field of compliance monitoring and analysis, and discloses a multi-dimensional feature information monitoring method and device, computer equipment and a storage medium. According to the invention, a multi-dimensional feature information monitoring model is adopted to determine information monitoring conditions corresponding to multiple pieces of to-be-monitored information of a target person, wherein the to-be-monitored information includes but is not limited to appearance image information, audio information, video information and position information of the target person; and based on the information monitoring condition, adopting a preset monitoring analysis standard to monitor and analyze the target person, and determining a monitoring analysis result corresponding to the target person, thereby realizing multi-dimensional monitoring of multiple pieces of to-be-monitored information of the target person, and improving comprehensiveness and accuracy of information monitoring of the target person.

Description

Technical Field

[0001] The present invention relates to the technical field of compliance monitoring and analysis, and in particular to a multi-dimensional feature information monitoring method, device, computer equipment and storage medium. Background Art

[0002] With the development and progress of science and technology, there are already some information monitoring and analysis technologies for target personnel; however, existing technologies usually only use a single aspect of the target personnel as a criterion for monitoring and analysis, for example, only monitoring the target personnel by identifying and analyzing the voice content. Therefore, there are problems such as insufficient data collection for monitoring and analysis and imperfect analysis models. Summary of the invention

[0003] In view of this, the present invention provides a multi-dimensional feature information monitoring method, device, computer equipment and storage medium to improve the comprehensiveness and accuracy of information monitoring on target personnel.

[0004] On the one hand, the present invention provides a multi-dimensional feature information monitoring method, which includes using a multi-dimensional feature information monitoring model to determine information monitoring situations corresponding to multiple monitored information of a target person, wherein the monitored information includes but is not limited to: appearance image information, audio information, video information and location information of the target person; based on the information monitoring situation, using preset monitoring and analysis standards to monitor and analyze the target person, and determining the monitoring and analysis results corresponding to the target person.

[0005] On the other hand, the present invention also provides a multi-dimensional feature information monitoring device, which includes: a determination module, which is used to use a multi-dimensional feature information monitoring model to determine the information monitoring status corresponding to multiple monitored information of a target person, and the monitored information includes but is not limited to: the appearance image information, audio information, video information and location information of the target person; a monitoring and analysis module, which is used to monitor and analyze the target person based on the information monitoring status and using preset monitoring and analysis standards to determine the monitoring and analysis results corresponding to the target person.

[0006] On the other hand, the present invention also provides a computer device, which includes: a memory and a processor, the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor implements the above-mentioned multi-dimensional feature information monitoring method by executing the computer instructions.

[0007] On the other hand, the present invention further provides a computer-readable storage medium, on which computer instructions are stored, and the computer instructions are used to enable a computer to implement the above-mentioned multi-dimensional feature information monitoring method.

[0008] In this process, the information monitoring situations corresponding to the multiple monitored information of the target personnel are obtained, and based on the information monitoring situations, the target personnel are monitored and analyzed using the preset monitoring and analysis standards, and the monitoring and analysis results corresponding to the target personnel are determined, so as to realize multi-dimensional feature monitoring and analysis of the multiple monitored information of the target personnel, thereby improving the comprehensiveness and accuracy of the information monitoring of the target personnel. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the related technologies, the drawings required for use in the specific embodiments or the related technical descriptions will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0010] Figure 1 It is a flowchart of a multi-dimensional feature information monitoring method provided by an embodiment of the present invention;

[0011] Figure 2 It is a schematic diagram of the structure of a back-end analysis system of a multi-dimensional feature information monitoring method provided by an embodiment of the present invention;

[0012] Figure 3 It is a flowchart of another multi-dimensional feature information monitoring method provided by an embodiment of the present invention;

[0013] Figure 4 is a structural schematic diagram of a recorder provided by an embodiment of the present invention;

[0014] Figure 5 It is a structural schematic diagram of a multi-dimensional feature information monitoring device provided by an embodiment of the present invention;

[0015] Figure 6 It is a structural schematic diagram of another multi-dimensional feature information monitoring device provided by an embodiment of the present invention. DETAILED DESCRIPTION

[0016] With the development of science and technology, there are already some related solutions for civilized language voice recognition or video action recognition in the existing technology, but the application scenarios of the existing technology are currently only concentrated on the identification of traffic violation cases and the monitoring and analysis of the language used by online car-hailing drivers; the identification of traffic violation cases in the existing technology obtains traffic violation case images, identifies the key elements therein, and determines the compliance of the handling; in the monitoring and analysis of the language used by online car-hailing drivers in the existing technology, by monitoring and identifying the voice of the online car-hailing driver, it can be determined whether uncivilized language is used during the transportation process; there is currently existing technology that has realized video recognition of some actions, but there is no more comprehensive multi-dimensional feature information monitoring method.

[0017] Therefore, existing technologies usually only use a single aspect of the target person as a criterion for monitoring and analysis, and there are still problems such as insufficient data collection for monitoring and analysis and imperfect analysis models.

[0018] In order to solve the above problems, various embodiments of the present invention provide a multi-dimensional feature information monitoring method, including: using a multi-dimensional feature information monitoring model to determine the information monitoring status corresponding to multiple monitored information of the target person, the monitored information includes but is not limited to: the target person's appearance image information, audio information, video information and location information; based on the information monitoring status, using preset monitoring and analysis standards to monitor and analyze the target person, and determine the monitoring and analysis results corresponding to the target person.

[0019] In order to make the purpose, technical solution and advantages of the embodiments of the present invention clearer, the technical solution in the embodiments of the present invention will be clearly and completely described below in conjunction with the drawings in the embodiments of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present invention.

[0020] According to an embodiment of the present invention, a multi-dimensional feature information monitoring method is provided. Figure 1 is a flow chart of a multi-dimensional feature information monitoring method provided by an embodiment of the present invention, such as Figure 1 As shown, the process includes the following steps:

[0021] Step S101, using a multi-dimensional feature information monitoring model to determine information monitoring conditions corresponding to multiple pieces of information to be monitored of a target person, the information to be monitored including but not limited to: appearance image information, audio information, video information and location information of the target person;

[0022] Step S102: Based on the information monitoring situation, the target personnel are monitored and analyzed using the preset monitoring and analysis standards to determine the monitoring and analysis results corresponding to the target personnel.

[0023] Among them, the appearance image information may refer to the appearance image of the target person, through which the head, upper limbs, legs and feet of the target person can be fully displayed; the audio information may refer to the full recording of the target person; the video information may be the full video recording of the target person; the location information may refer to the determined full positioning position of the target person.

[0024] Exemplarily, the target personnel may include, but are not limited to, personnel of various occupations and types of work.

[0025] In one possible implementation, the multidimensional feature information monitoring model can be a model that can comprehensively process and analyze multiple different types of feature information. Multidimensional can refer to different types of data or feature information from multiple dimensions. Here, multidimensional can include: graphics, audio, video, and location coordinates.

[0026] In a possible implementation, based on the information monitoring situation, the target person is monitored and analyzed using preset monitoring and analysis standards, which may include:

[0027] The number of people who do not meet the preset monitoring and analysis standards in the appearance image information monitoring, audio information monitoring, video information monitoring and location information monitoring is counted, and the target personnel are monitored and analyzed based on the number and the preset monitoring and analysis standards; the preset monitoring and analysis standards can be set according to specific needs.

[0028] Among them, the information monitoring situation may refer to the preliminary processing results of the monitored information; the monitoring and analysis of the target personnel may include but is not limited to: behavioral analysis, emotional analysis, identity recognition analysis, health status analysis, etc. of the target personnel.

[0029] Through the above method, by determining the information monitoring situations corresponding to multiple pieces of information to be monitored of the target person and performing monitoring analysis, the comprehensiveness and accuracy of information monitoring on the target person can be improved.

[0030] In a specific embodiment, a multi-dimensional feature information monitoring model is used to determine the information monitoring status corresponding to multiple monitored information of the target person, and the monitored information includes but is not limited to: appearance image information, audio information, video information and location information of the target person, based on the following steps:

[0031] Using a neural network-based target detection model to identify preset parts in the appearance image information, determining monitoring results corresponding to the preset parts, and determining a first information monitoring situation corresponding to the appearance image information based on the monitoring results;

[0032] A pre-trained deep speech learning model based on an encoder-decoder structure and a pre-trained named entity recognition model are used to encode and decode the audio information, determine the text information corresponding to the audio information, identify the standard terms and uncivilized terms in the text information, and determine the monitoring status of the second information corresponding to the audio information based on the number of standard terms and uncivilized terms;

[0033] A video recognition model based on mask reconstruction and multimodal learning is used to identify the types and frequencies of illegal actions in video information, and based on the types and frequencies, the third information monitoring status corresponding to the video information is determined; illegal actions include: uncivilized actions and violent actions;

[0034] A preset boundary deviation rule is used to determine whether the location information of the target person meets the preset location requirements, determine the degree of deviation of the location information, and determine the fourth information monitoring situation corresponding to the location information based on the degree of deviation.

[0035] In a possible implementation, a target detection model based on a neural network is used to identify preset parts in the appearance image information, and monitoring results corresponding to the preset parts are determined. Based on the monitoring results, a first information monitoring condition corresponding to the appearance image information is determined, including:

[0036] A target detection model based on a neural network is used to identify preset body parts of the target person contained in the appearance image information, and each preset body part is monitored and analyzed respectively to determine the monitoring results corresponding to each preset body part;

[0037] Based on the monitoring result, the first information monitoring situation corresponding to the appearance image information is determined.

[0038] Among them, the neural network-based target detection model can refer to a type of deep learning model used to automatically detect, locate and identify specific targets in images or videos.

[0039] Furthermore, the training process of the neural network-based object detection model may include:

[0040] Obtain appearance image data, annotate preset parts in the image, and use data enhancement technology to generate more training samples;

[0041] Divide the labeled data set into training set, validation set and test set, design a preset loss function, use the back propagation algorithm to train the model, and adjust the model parameters to minimize the loss function;

[0042] Use the validation set to monitor the model output and fine-tune the model;

[0043] Use the validation set to evaluate the accuracy and recall of the model, and adjust the model architecture or parameters to improve the target detection performance.

[0044] Among them, the preset parts in the image are marked to determine the facial features, hair, ears, etc. included in the appearance image data; the target detection model used may include but is not limited to: YOLO (You Only Look Once) model, Convolutional Neural Networks (CNN) model, etc.

[0045] In one possible implementation, the encoder-decoder structure in the pre-trained deep speech learning model based on the encoder-decoder structure includes an encoder and a decoder.

[0046] Among them, the encoder can convert the input audio signal into an audio feature sequence, and the encoder can include but is not limited to: CNN or Recurrent Neural Network (RNN) structure; the decoder can generate a text sequence from the audio feature sequence output by the encoder, and the decoder can include but is not limited to: an autoregressive model or a non-autoregressive model; the pre-trained deep speech learning model can refer to a deep learning model that is used for specific speech tasks after preliminary training on a large-scale speech data set, and the preliminary training includes pre-training and fine-tuning.

[0047] In one possible implementation, preliminary training of a pre-trained deep speech learning model based on an encoder-decoder structure includes:

[0048] Collect audio data containing multiple languages ​​and accents, annotate the audio data, and identify standard terms, uncivilized terms, and other entity information in the audio data;

[0049] Preprocess the audio data, input the preprocessed audio features into the encoder-decoder structure, obtain the output text sequence, and calculate the loss based on the model's predicted output and the actual text label;

[0050] The gradient is calculated through the back-propagation algorithm and the optimizer is used to update the parameters of the model to minimize the loss;

[0051] Fine-tuning technology is used to perform additional training on the model using task-specific datasets to adapt the model to the recognition of specific standard and uncivilized language.

[0052] Among them, the back propagation algorithm calculates the gradient of the loss function relative to the model parameters, and determines the adjustment method of the parameters based on the gradient to minimize the loss function.

[0053] Here, we can improve the model's recognition of specific standard terms and uncivilized terms through fine-tuning.

[0054] Furthermore, a pre-trained named entity recognition (NER) model can be used to recognize and mark specific types of entities in the text, such as standard terms, uncivilized terms, names of people, names of places, etc.; the named entity recognition can be a BERT (Bidirectional Encoder Representations from Transformers) model.

[0055] In one possible implementation, a training method for a video recognition model based on mask reconstruction and multimodal learning is as follows:

[0056] Obtain video data in different scenarios, annotate the video data, determine the uncivilized and violent actions contained in the video, and the time when the actions appear, and divide the video data into multiple small clips;

[0057] Randomly select some areas in multiple small clips of video data for masking, so as to train the model to reconstruct the masked parts based on the context information of the video data;

[0058] Extract multiple modal information from video data and adopt multimodal learning methods to enable the model to better classify the action types in video data.

[0059] Among them, masked reconstruction can refer to the model intentionally masking part of the input data and training the model to reconstruct the masked part, so that the model can learn the internal features and structure of the input data; multimodal learning can refer to the model being able to simultaneously process and integrate data from different sources or forms, including but not limited to images, audio, and text.

[0060] In one possible implementation, the preset position requirement may refer to determining a position range where the target person should be located. When the target person exceeds a distance threshold outside the specified range of the preset position requirement, a preset boundary deviation rule is used to determine the degree of deviation of the target person.

[0061] Among them, the degree of deviation may include but is not limited to: distance deviation degree, time deviation degree, and direction deviation degree; the distance deviation degree may refer to the specific distance between the target person and the preset position, the time deviation degree may refer to the specific time when the target person deviates from the preset position, and the direction deviation degree may refer to the specific deviation angle of the target person from the preset position.

[0062] Further, obtaining the initial location area of ​​the target person and the location area of ​​the preset object;

[0063] Based on the preset boundary deviation rules, determine whether the location information of the target person belongs to the location area that meets the preset requirements;

[0064] If the location information does not belong to the location area that meets the preset requirements, the fourth information monitoring situation of the target person is determined to be unqualified;

[0065] The online status, offline statistics and longest continuous offline time of the target person's recorder are detected. If the longest continuous offline time of the recorder is greater than the preset offline threshold, the fourth information monitoring status of the target person is determined to be unqualified.

[0066] Here, the location information of the target person is obtained and the comparison information of the location area of ​​the preset object is obtained; based on the preset boundary deviation rule, it is determined whether the target person is in a position that meets the preset requirements. If it exceeds the distance range defined by the preset boundary deviation rule, it is determined to be unqualified; the online status, offline statistical times and the longest continuous offline time of the recorder detecting the target person are obtained. If the longest continuous offline time of the recorder is greater than 30 minutes, the fourth information monitoring status of the target person is determined to be unqualified.

[0067] Here, the recorder may refer to a recording terminal used to obtain appearance image information, audio information, video information and location information of a target person.

[0068] Through the above method, by simultaneously monitoring the target person's appearance image information, audio information, video information and location information, the system can comprehensively evaluate the target person's behavior and status from multiple dimensions; the monitoring of video information adopts multimodal learning technology, which combines the visual and behavioral characteristics of the video, which helps to more accurately identify and classify the target person's illegal actions; the processing of audio information uses a pre-trained deep speech learning model and a named entity recognition model, which can efficiently extract text information and identify standard terms and uncivilized terms; the monitoring of location information adopts a preset boundary deviation rule, which can flexibly adjust the target person's location information requirements according to actual conditions.

[0069] In a specific embodiment, a target detection model based on a neural network is used to identify preset parts in the appearance image information, and monitoring results corresponding to the preset parts are determined. Based on the monitoring results, the first information monitoring situation corresponding to the appearance image information is determined, which is implemented based on the following steps:

[0070] Pre-training the first model to obtain a neural network-based target detection model;

[0071] A target detection model based on a neural network is used to identify preset parts in the appearance image information, the preset parts include but are not limited to: head, upper limbs, legs and feet, and based on the preset unqualified conditions, the monitoring results corresponding to the head, upper limbs, legs and feet are determined respectively;

[0072] Based on at least one monitoring result, a first information monitoring situation corresponding to the appearance image information is determined.

[0073] In a possible implementation, a neural network-based target detection model is used to identify preset parts in the appearance image information, and monitoring results corresponding to the preset parts are determined. The method further includes:

[0074] Adopting a neural network-based target detection model, based on preset parts, it can identify head features, upper limb features, leg features, and foot features in the appearance image information;

[0075] According to the head features, upper limb features, leg features and foot features, the appearance positions of the head, upper limbs, legs and feet are obtained respectively.

[0076] Among them, head features may include but are not limited to: hairstyle, hat, glasses; upper limb features may include but are not limited to: clothing worn, clothing sleeve length; leg features may include but are not limited to: clothing trouser length; foot features may include but are not limited to: shoe type, sock color.

[0077] In a possible implementation, based on the monitoring result, determining the first information monitoring situation corresponding to the appearance image information is implemented based on the following steps:

[0078] Determine the preset appearance scoring standards based on the pre-classified unqualified situations;

[0079] According to the preset appearance scoring standard, the preset parts identified by the neural network-based target detection model are scored, and the first information monitoring conditions corresponding to the preset parts are determined by calculating the appearance instrument qualification scores.

[0080] Among them, the pre-classified unqualified situations may include but are not limited to: not wearing protective clothing, gloves, work badges, etc.; wearing different clothes together; wearing or hanging signs that are not related to the environment corresponding to the preset process; draping clothes, opening the chest, rolling up sleeves, and rolling up trouser legs; weird hairstyles, and disheveled hair.

[0081] Here, pre-training the first model can be achieved based on the following steps:

[0082] Obtain sufficient images of the target personnel's appearance for training;

[0083] Set labels according to the preset parts and monitoring results that need to be marked. Labels can include head, upper limbs, legs and feet. The monitoring results of each part are divided into qualified, partially qualified and unqualified;

[0084] Use image annotation tools to annotate all collected appearance images;

[0085] According to the characteristics of the data set and the requirements of the target detection task, the parameters of the first model are adjusted, and the first model is trained using the labeled image data. The performance of the model is monitored during the training process, and the optimal weights when the performance is best are recorded.

[0086] Furthermore, the optimal weights are loaded into the model, and a neural network-based target detection model based on the optimal weights is used to predict each preset part respectively, and output the corresponding first information monitoring situation.

[0087] In a possible implementation, by calculating the eligibility score of the appearance and instrument, determining the first information monitoring conditions corresponding to the preset parts respectively can be implemented based on the following steps:

[0088] According to the preset appearance scoring standard, the appearance pass score of each preset part in the appearance image of the target person is calculated and determined; the preset appearance scoring standard may include: 3 points for pass, 2 points for partly pass, and 0 points for fail;

[0089] By calculating the appearance and manners qualification score, the first information monitoring conditions corresponding to the preset parts are determined. The first information monitoring conditions may include: the appearance and manners qualification score is greater than or equal to the first threshold score for qualified, the appearance and manners qualification score is greater than the second threshold score and less than the first threshold score for partially qualified, and the appearance and manners qualification score is less than the second threshold score for unqualified.

[0090] Here, the threshold scores may be specifically set according to requirements. For example, the first threshold score may be 80 points, and the second threshold score may be 60 points.

[0091] Furthermore, a target detection model based on a neural network is used to identify the number of target persons in the appearance image information. If the number of target persons identified in the appearance image information is less than 2, the first information monitoring situation is output as unqualified.

[0092] Through the above method, the appearance evaluation is carried out using the neural network-based target detection model, and the preset appearance scoring standards and monitoring analysis are adopted, which can reduce the influence of human subjective factors, enhance the transparency and fairness of the evaluation process, and increase the fairness of the first information monitoring inspection of the target personnel; through automated appearance image evaluation, unqualified situations can be quickly discovered and handled, and corrected and managed in time, thereby improving the efficiency and effectiveness of monitoring and analysis.

[0093] In a specific embodiment, pre-training the first model to obtain a neural network-based target detection model includes:

[0094] Acquire the appearance image data of the target person for training, perform a first labeling process on the appearance image data of the target person based on a first preset label, and determine the corresponding preset part in the appearance image data of the target person;

[0095] Based on the second preset label, the target person's appearance image data is subjected to a second labeling process to determine the monitoring result labels corresponding to the preset parts in the target person's appearance image data, and the monitoring result labels include but are not limited to: qualified, partially qualified and unqualified;

[0096] The first model is trained using the labeled appearance image data of the target person, the weight of the first model is adjusted during the training process, and the first model with the optimal weight is determined to be used as the target detection model based on the neural network.

[0097] In a possible implementation, based on the first preset tag, performing a first labeling process on the appearance image data of the target person to determine the corresponding preset part in the appearance image data of the target person includes:

[0098] Determine the preset parts in the appearance image data of the target person, mark each preset part with a corresponding first preset label, and determine the corresponding preset parts in the appearance image data of the target person.

[0099] The first preset tags include but are not limited to: head tags, upper limb tags, leg tags and foot tags.

[0100] In a possible implementation, based on the second preset label, the second labeling process is performed on the appearance image data of the target person to determine the monitoring result labels corresponding to the preset parts in the appearance image data of the target person, including:

[0101] Based on the result of the first labeling process, a second labeling process is further performed on each preset part to analyze whether each preset part meets the preset specification requirements, and a monitoring result label is assigned to each preset part according to the qualified status.

[0102] In a possible implementation, the first model is trained using labeled target person appearance image data, the weight of the first model is adjusted during the training process, and the first model with the optimal weight is determined as the target detection model based on the neural network, including:

[0103] Determine the initialization weight of the first model, and input the labeled target person appearance image data set into the first model as training data;

[0104] The cross entropy loss function and the regression loss function are used as optimization objectives, the back propagation algorithm and the gradient descent method are used to update the weights of the first model, and the first model with the optimal weight is determined as the target detection model based on the neural network.

[0105] Here, the cross entropy loss function is used to minimize the difference between the probability distribution predicted by the first model and the true category distribution; the regression loss function is used to train the first model by minimizing the error between the predicted value of the first model and the true value.

[0106] Through the above method, a neural network-based target detection model is adopted to accurately identify the preset parts in the target person's appearance image, thereby improving the reliability of the overall monitoring and analysis; the first preset label and the second preset label used can be adjusted according to specific application scenarios and needs, thereby improving the flexibility of monitoring and analysis.

[0107] In a specific embodiment, a pre-trained deep speech learning model based on an encoder-decoder structure and a pre-trained named entity recognition model are used to encode and decode audio information, determine text information corresponding to the audio information, identify standard terms and uncivilized terms in the text information, and determine the monitoring status of the second information corresponding to the audio information based on the number of standard terms and uncivilized terms, which is implemented based on the following steps:

[0108] Pre-training the second model to obtain a pre-trained deep speech learning model based on an encoder-decoder structure;

[0109] A pre-trained deep speech learning model based on an encoder-decoder structure is used to resample the audio information, determine the audio information with a uniform sampling rate, convert the audio information with a uniform sampling rate into a logarithmic Mel spectrogram, and encode and embed the preset feature information in the logarithmic Mel spectrogram; the logarithmic Mel spectrogram is used to characterize the audio frequency and audio amplitude of the audio information of the target person;

[0110] The decoder in the pre-trained deep speech learning model based on the encoder-decoder structure is used to convert the logarithmic Mel spectrogram into text information;

[0111] Use a pre-trained named entity recognition model to identify standard and uncivilized terms in text information based on preset text recognition tags;

[0112] Determine the monitoring status of the second information corresponding to the audio information based on the number of standard terms and uncivilized terms;

[0113] The second model is pre-trained to obtain a pre-trained deep speech learning model based on an encoder-decoder structure, including:

[0114] Acquire preset audio segment information and corresponding text information of standard terms and uncivilized terms, and determine audio recognition corpus based on the combination of the audio segment information and the text information;

[0115] The audio recognition corpus is used to fine-tune the weights of the second model, and the second model with the optimal weights is determined to be used as a pre-trained deep speech learning model based on an encoder-decoder structure.

[0116] Among them, the resampling process may refer to unifying the sampling rate of the audio information to ensure the consistency of subsequent processing; the embedded preset feature information may include but is not limited to: speech feature information, context feature information, speech emotion feature information, etc.

[0117] In one possible implementation, the weight of the second model is fine-tuned using audio recognition corpus, and the second model with the optimal weight is determined to be used as a pre-trained deep speech learning model based on an encoder-decoder structure, which is implemented based on the following steps:

[0118] The encoder in the second model is trained, including:

[0119] The encoder is pre-trained using audio recognition corpus, and the model weights are adjusted through the back-propagation algorithm to gradually optimize the performance of the encoder part so that the encoder minimizes the difference between the predicted text and the real text;

[0120] The decoder in the second model is trained, including:

[0121] The collected audio recognition corpus and text information of standard and uncivilized language are used as a training dataset to fine-tune the model and adjust the weights of the decoder to ensure that the decoder accurately maps the audio features output by the encoder into text.

[0122] In a possible implementation, determining the monitoring status of the second information corresponding to the audio information based on the number of standard terms and uncivil terms includes:

[0123] An audio similarity analysis model is used to compare and analyze the identified standardized terms with standard standardized terms, calculate the audio similarity, determine whether there are situations where standardized terms are not used in specific scenarios, and count the number of times standardized terms are not used.

[0124] Here, the audio similarity analysis model may be a model for evaluating and comparing the similarity between two audio segments; for example, the audio similarity analysis model may include but is not limited to: CNN, RNN.

[0125] Furthermore, the target personnel's use of standard and uncivilized language is counted, and the audio clips of uncivilized language, language content, language frequency information and the use frequency of non-standard language are uploaded to the back-end analysis system;

[0126] The second information monitoring situation corresponding to the audio clip of the target person is determined through the back-end analysis system.

[0127] Here, the backend analysis system may refer to a system at least used to view and analyze the target personnel monitoring analysis results.

[0128] Through the above method, based on the use of logarithmic Mel spectrogram and the powerful modeling ability of the encoder-decoder structure, the effective capture and translation of speech features are ensured, thereby improving the accuracy of speech recognition; by encoding multi-dimensional features such as audio frequency and audio amplitude, the model can understand the audio information more comprehensively and improve the recognition effect of standard language and uncivilized language.

[0129] In a specific embodiment, a video recognition model based on mask reconstruction and multimodal learning is used to identify the type and frequency of illegal actions in video information, and based on the type and frequency, the monitoring status of the third information corresponding to the video information is determined, which is implemented based on the following steps:

[0130] The first layer model in the video recognition model based on mask reconstruction and multimodal learning is used to partially occlude the video information and determine the embedded coding of the video action with mask information; the first layer model is used to characterize the masked video coding layer of the encoder and decoder structure, and the mask information is used to characterize the occluded part of the video information;

[0131] The second layer model in the video recognition model based on masked reconstruction and multimodal learning is used to perform encoding and decoding based on the video action embedding coding with masked information and other preset coding to detect the type and frequency of illegal actions in the video information; the second layer model is used to characterize the multimodal learning layer, and other preset coding includes but is not limited to: video frames, single-frame video images, and text description information in the video information;

[0132] Based on the type and frequency, determine the monitoring status of the third information corresponding to the video information.

[0133] In a possible implementation, the first layer model in the video recognition model based on mask reconstruction and multimodal learning is used to perform partial occlusion processing on the video information, and determine the video action embedding code with the mask information, including:

[0134] Perform partial occlusion processing on the video information, randomly occlude the pixel information of a specific area in the video information, and generate a masked image;

[0135] The preprocessed video frame sequence is input to the encoder-decoder structure, and the encoder is used to obtain the video action embedding code; the decoder is used to try to reconstruct the original video frame, restore the occluded part, and output the reconstructed video frame.

[0136] Furthermore, the second layer model in the video recognition model based on masked reconstruction and multimodal learning is adopted, and the video action embedding coding with masked information and other preset coding are used for encoding and decoding processing to detect the type and frequency of illegal actions in the video information, including:

[0137] A multimodal learning layer is used to encode the video stream, single-frame image and semantic description text respectively to determine the global features, local features and text features of the video stream;

[0138] Jointly encode the global features, local features, and text features of the video stream to generate a joint feature representation, jointly decode the joint feature representation, and output the illegal actions in the video information

[0139] Furthermore, the fine-tuning process of the video recognition model based on masked reconstruction and multimodal learning may include: fine-tuning the model using preset normative action and uncivilized and violent action video data to ensure that the model has higher accuracy in recognizing these specific actions; and adjusting the model parameters through supervised learning methods to optimize the recognition performance of illegal actions.

[0140] Through the above method, a video recognition model based on masked reconstruction and multimodal learning is used to identify illegal actions. The actions of the target person in the video information can be identified and analyzed in real time and accurately during the preset process to ensure the standardization of the behavior.

[0141] In a specific embodiment, the method further comprises:

[0142] When the monitoring and analysis results indicate that the target personnel are unqualified, a voice reminder broadcast is sent to the target personnel, and the voice reminder broadcast includes but is not limited to: the first information monitoring situation is unqualified, the second information monitoring situation is unqualified, the third information monitoring situation is unqualified, the fourth information monitoring situation is unqualified, and basic information and precautions.

[0143] Here, basic information and precautions may refer to the information that the target personnel need to obtain when performing the preset process; for example, if the target personnel are production line workers, the basic information may include: work area, work requirements, working hours, and precautions may refer to complying with safety operating procedures, avoiding accidents, wearing protective equipment as required, and not removing them at will, etc.

[0144] Through the above method, after detecting that one or more information monitoring conditions of the target person are unqualified, a voice reminder can be immediately generated and broadcast. Such an instant feedback mechanism helps the target person quickly understand his or her unqualified situation and take corresponding corrective measures immediately; the content of the voice reminder broadcast is customized according to the specific unqualified situation to ensure that the information received by the target person is closely related to his or her actual problem. This highly targeted reminder method can increase the target person's attention to the reminder information.

[0145] In a specific embodiment, the back-end analysis system can be used to comprehensively manage the monitoring and analysis results of the target personnel. The back-end analysis system can include: a model training module, a video analysis module and a personnel overview module.

[0146] For example, Figure 2 It is a schematic diagram of the structure of a back-end analysis system of a multi-dimensional feature information monitoring method provided in an embodiment of the present invention; wherein, the back-end analysis system 20 includes: a model training module 201, a video analysis module 202 and a personnel overview module 203.

[0147] In one possible implementation, the model training module 201 can be used to obtain model training data and train an optimized model; obtain labeled image, audio, and video training data, and randomly divide the data into a training set, a validation set, and a test set according to a preset ratio; input the training set, the validation set, and the test set into a preset image recognition model, a preset audio recognition model, and a preset visual model for iterative training, respectively, to obtain the model accuracy, recall rate, and F1 value for each round; dynamically adjust the model parameters according to the model training situation, determine the optimal training parameters, and pass the optimal model based on the optimal training parameters to the video analysis module 202.

[0148] Here, accuracy can refer to the ratio of the number of samples correctly predicted by the model to the total number of samples, recall can refer to the ratio of positive samples that the model can correctly identify, and F1 value can be the harmonic mean of accuracy and recall, which is used to find a balance between accuracy and recall. Among them, positive samples can refer to samples marked as the target category in a binary classification problem.

[0149] In a possible implementation, the video analysis module 202 uses a video recognition model based on mask reconstruction and multimodal learning to perform behavioral action analysis on the video information; detects whether there are uncivilized and violent actions in the video information, and identifies the type and frequency of unqualified actions, and transmits the video clips, types and frequency statistics of the unqualified actions identified to the personnel overview module 203, and at the same time transmits the third information monitoring situation back to the voice reminder module;

[0150] The video analysis module 202 can use an audio and video similarity analysis model to compare and analyze the identified standard actions with the standard standard actions after the target person completes the preset process, determine whether there are situations where standard terms are not used in specific scenarios, and upload the statistical number of times the standard actions are not used to the personnel overview module 203.

[0151] In one possible implementation, the personnel overview module 203 can be used to view the basic information and monitoring and analysis results of the target personnel; the basic information includes but is not limited to: name, identification number, age, etc.; the monitoring and analysis results include but are not limited to: statistics on the number of unqualified terms and actions, and statistics on the number of times standard terms and standard actions are not used.

[0152] Through the above-mentioned back-end analysis system 20, it is possible to implement a comprehensive analysis of the information monitoring situations corresponding to multiple monitored information of the target personnel, realize comprehensive management and inspection of the target personnel, and thus solve the problems of weak inspection force, high cost, and inadequate and incomplete inspection results for the target personnel.

[0153] Figure 3 FIG. 1 is a flow chart of another multi-dimensional feature information monitoring method provided by an embodiment of the present invention; Figure 3 As shown, the process includes the following steps:

[0154] Step S301, obtaining appearance image information;

[0155] Here, a recorder is used to obtain the appearance image information of the target person;

[0156] Step S302, determining whether it is qualified;

[0157] Here, a target detection model based on a neural network is used to determine whether the appearance image information is qualified; if so, the process proceeds to step S304; if not, the process returns to step S301;

[0158] Step S303, obtaining audio information;

[0159] Here, a recorder is used to obtain audio information of the target person;

[0160] Step S304, determining whether it is qualified;

[0161] Here, a pre-trained deep speech learning model based on an encoder-decoder structure and a pre-trained named entity recognition model are used to determine whether the audio information is qualified; if so, proceed to step S305, if not, proceed to step S306;

[0162] Step S305, obtaining video information;

[0163] Here, a recorder is used to obtain video information of the target person;

[0164] Step S306, uploading to the background analysis system;

[0165] Here, the number of times uncivilized language is used and the number of times standard language is not used in the audio information are uploaded to the background analysis system;

[0166] Step S307, determining whether it is qualified;

[0167] Here, a video recognition model based on mask reconstruction and multimodal learning is used to determine whether the video information is qualified; if so, proceed to step S308, if not, proceed to step S309;

[0168] Step S308, obtaining location information;

[0169] Here, a recorder is used to obtain the location information of the target person;

[0170] Step S309, uploading to the background analysis system;

[0171] Here, the number of times uncivilized actions are used and the number of times standard actions are not used in the video information are uploaded to the background analysis system;

[0172] Step S310, determining whether it is qualified;

[0173] Here, based on the preset boundary deviation rule, it is determined whether the location information of the target person belongs to a location area that meets the preset requirements; if so, it proceeds to step S311; if not, it proceeds to step S312;

[0174] Step S311, obtaining online status;

[0175] Here, the online status of the target person's recorder is obtained;

[0176] Step S312, uploading to the background analysis system;

[0177] Here, the location information is uploaded to the background analysis system;

[0178] Step S313, determining whether it is qualified;

[0179] Here, it is detected whether the longest continuous offline time of the recorder is greater than the preset offline threshold; if so, the process ends, if not, it goes to step S314;

[0180] Step S314, uploading to the background analysis system;

[0181] Here, the fourth information monitoring situation of the online situation is uploaded to the background analysis system.

[0182] In a specific embodiment, Figure 4 4 is a schematic diagram of the structure of a recorder provided in an embodiment of the present invention; wherein the recorder 40 includes a collection module 401, an analysis module 402 and a voice reminder module 403;

[0183] The acquisition module 401 includes: an appearance image information acquisition module 4011, an audio information acquisition module 4012, a video information acquisition module 4013 and a location information acquisition module 4014; the analysis module 402 includes: a target detection module 4021, a voice recognition module 4022 and a route location analysis module 4023; the voice reminder module 403 includes: a monitoring and analysis result reminder module 4031 and an information reminder module 4032.

[0184] Specifically, the appearance image information acquisition module 4011 is used to obtain the appearance image information of the target person;

[0185] The audio information acquisition module 4012 is used to obtain the audio information of the target person;

[0186] The video information acquisition module 4013 is used to obtain the video information of the target person;

[0187] The location information acquisition module 4014 is used to obtain the location information of the target person;

[0188] The target detection module 4021 is used to analyze the appearance and instrument image by using a target detection model based on a neural network, determine the first information monitoring situation, and transmit the first information monitoring situation back to the voice reminder module 403;

[0189] The speech recognition module 4022 is used to recognize the target person's language using a pre-trained deep speech learning model based on an encoder-decoder structure and a pre-trained named entity recognition model, identify the standard language and uncivilized language therein, upload the uncivilized language audio clip, language content, and language frequency information to the personnel overview module 203 in the back-end analysis system 20, and at the same time return the second information monitoring situation to the speech reminder module 403;

[0190] The route and position analysis module 4023 is used to analyze the real-time position and route of the target personnel. If it is detected that the target personnel deviates from the route for a long time and is not in the specified position, it is uploaded to the personnel overview module 203 in real time, and the fourth information monitoring situation is sent back to the voice reminder module 403 at the same time;

[0191] The monitoring and analysis result reminder module 4031 is used to remind the target person in voice form whether the first information monitoring situation is unqualified, the second information monitoring situation is unqualified, the third information monitoring situation is unqualified, and the fourth information monitoring situation is unqualified;

[0192] The information reminder module 4032 is used to provide the target person with the basic information and precautions that need to be obtained.

[0193] It should be noted that: when the recorder provided in the above embodiment implements the corresponding multi-dimensional feature information monitoring method, only the division of the above program modules is used as an example. In actual application, the above processing can be assigned to different program modules as needed, that is, the internal structure of the device can be divided into different program modules to complete all or part of the above-described processing. Figure 1 The embodiments of the method shown belong to the same concept, and the specific implementation process is detailed in the method embodiments, which will not be repeated here.

[0194] The embodiment of the present invention also provides a multi-dimensional feature information monitoring device, such as Figure 5 As shown, the device includes: an acquisition module 501 and an analysis module 502; wherein,

[0195] The acquisition module 501 is used to determine the information monitoring status corresponding to multiple monitored information of the target person by using a multi-dimensional feature information monitoring model, and the monitored information includes but is not limited to: appearance image information, audio information, video information and location information of the target person;

[0196] The analysis module 502 is used to monitor and analyze the target personnel based on the information monitoring situation and adopt the preset monitoring and analysis standards to determine the monitoring and analysis results corresponding to the target personnel.

[0197] In one embodiment, the acquisition module 501 is used to identify the preset parts in the appearance image information by using the target detection model based on the neural network, determine the monitoring results corresponding to the preset parts respectively, and determine the first information monitoring situation corresponding to the appearance image information based on the monitoring results;

[0198] A pre-trained deep speech learning model based on an encoder-decoder structure and a pre-trained named entity recognition model are used to encode and decode the audio information, determine the text information corresponding to the audio information, identify the standard terms and uncivilized terms in the text information, and determine the monitoring status of the second information corresponding to the audio information based on the number of standard terms and uncivilized terms;

[0199] A video recognition model based on mask reconstruction and multimodal learning is used to identify the types and frequencies of illegal actions in video information, and based on the types and frequencies, the third information monitoring status corresponding to the video information is determined; illegal actions include: uncivilized actions and violent actions;

[0200] A preset boundary deviation rule is used to determine whether the location information of the target person meets the preset location requirements, determine the degree of deviation of the location information, and determine the fourth information monitoring situation corresponding to the location information based on the degree of deviation.

[0201] In one embodiment, the acquisition module 501 is used to pre-train the first model to obtain a target detection model based on a neural network;

[0202] A target detection model based on a neural network is used to identify preset parts in the appearance image information, the preset parts include but are not limited to: head, upper limbs, legs and feet, and based on the preset unqualified conditions, the monitoring results corresponding to the head, upper limbs, legs and feet are determined respectively;

[0203] Based on at least one monitoring result, a first information monitoring situation corresponding to the appearance image information is determined.

[0204] In one embodiment, the acquisition module 501 is used to acquire the appearance image data of the target person for training, perform a first labeling process on the appearance image data of the target person based on the first preset label, and determine the corresponding preset part in the appearance image data of the target person;

[0205] Based on the second preset label, the target person's appearance image data is subjected to a second labeling process to determine the monitoring result labels corresponding to the preset parts in the target person's appearance image data, and the monitoring result labels include but are not limited to: qualified, partially qualified and unqualified;

[0206] The first model is trained using the labeled appearance image data of the target person, the weight of the first model is adjusted during the training process, and the first model with the optimal weight is determined to be used as the target detection model based on the neural network.

[0207] In one embodiment, the acquisition module 501 is used to pre-train the second model to obtain a pre-trained deep speech learning model based on an encoder-decoder structure;

[0208] A pre-trained deep speech learning model based on an encoder-decoder structure is used to resample the audio information, determine the audio information with a uniform sampling rate, convert the audio information with a uniform sampling rate into a logarithmic Mel spectrogram, and encode and embed the preset feature information in the logarithmic Mel spectrogram; the logarithmic Mel spectrogram is used to characterize the audio frequency and audio amplitude of the audio information of the target person;

[0209] The decoder in the pre-trained deep speech learning model based on the encoder-decoder structure is used to convert the logarithmic Mel spectrogram into text information;

[0210] Use a pre-trained named entity recognition model to identify standard and uncivilized terms in text information based on preset text recognition tags;

[0211] Determine the monitoring status of the second information corresponding to the audio information based on the number of standard terms and uncivilized terms;

[0212] The second model is pre-trained to obtain a pre-trained deep speech learning model based on an encoder-decoder structure, including:

[0213] Acquire preset audio segment information and corresponding text information of standard terms and uncivilized terms, and determine audio recognition corpus based on the combination of the audio segment information and the text information;

[0214] The audio recognition corpus is used to fine-tune the weights of the second model, and the second model with the optimal weights is determined to be used as a pre-trained deep speech learning model based on an encoder-decoder structure.

[0215] In one embodiment, the acquisition module 501 is used to use the first layer model in the video recognition model based on mask reconstruction and multimodal learning to perform partial occlusion processing on the video information and determine the video action embedding code with mask information; the first layer model is used to represent the masked video coding layer of the encoder and decoder structure, and the mask information is used to represent the occluded part of the video information;

[0216] The second layer model in the video recognition model based on masked reconstruction and multimodal learning is used to perform encoding and decoding based on the video action embedding coding with masked information and other preset coding to detect the type and frequency of illegal actions in the video information; the second layer model is used to characterize the multimodal learning layer, and other preset coding includes but is not limited to: video frames, single-frame video images, and text description information in the video information;

[0217] Based on the type and frequency, determine the monitoring status of the third information corresponding to the video information.

[0218] In one embodiment, the device further includes a reminder module 503;

[0219] Reminder module 503 is used to send a voice reminder broadcast to the target person when the monitoring and analysis results indicate that the target person is unqualified. The voice reminder broadcast includes but is not limited to: the first information monitoring situation is unqualified, the second information monitoring situation is unqualified, the third information monitoring situation is unqualified, the fourth information monitoring situation is unqualified and basic information and precautions.

[0220] The embodiment of the present invention also provides a computer device having the above Figure 5 The multi-dimensional feature information monitoring device shown.

[0221] See also Figure 6 , Figure 6 is a schematic diagram of the structure of a multi-dimensional feature information monitoring device provided by an embodiment of the present invention, such as Figure 6As shown, the computer device includes: one or more processors 100, a memory 200, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process the instructions executed in the computer device, including instructions stored in or on the memory to display the graphical information of the GUI on an external input / output device (such as, a display device coupled to the interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 6 A processor 100 is taken as an example.

[0222] The processor 100 may be a central processing unit, a network processor or a combination thereof. The processor 100 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable logic gate array, a general purpose array logic or any combination thereof.

[0223] The memory 200 stores instructions executable by at least one processor 100, so that the at least one processor 100 executes the method shown in the above embodiment.

[0224] The memory 200 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required by at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 200 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 200 may optionally include a memory remotely arranged relative to the processor 100, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0225] The memory 200 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0226] The computer device further includes a communication interface 300 for the computer device to communicate with other devices or a communication network.

[0227] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0228] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0229] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A multi-dimensional feature information monitoring method, characterized in that: The method comprises: A multi-dimensional feature information monitoring model is used to determine information monitoring conditions corresponding to multiple pieces of information to be monitored of a target person, wherein the information to be monitored includes but is not limited to: appearance image information, audio information, video information and location information of the target person; Based on the information monitoring situation, the target personnel are monitored and analyzed using preset monitoring and analysis standards to determine the monitoring and analysis results corresponding to the target personnel.

2. The method according to claim 1, characterized in that A multi-dimensional feature information monitoring model is used to determine the information monitoring status corresponding to multiple monitored information of the target person, wherein the monitored information includes but is not limited to: appearance image information, audio information, video information and location information of the target person, based on the following steps: Using a neural network-based target detection model to identify preset parts in the appearance image information, determining monitoring results corresponding to the preset parts respectively, and determining a first information monitoring situation corresponding to the appearance image information based on the monitoring results; A pre-trained deep speech learning model based on an encoder-decoder structure and a pre-trained named entity recognition model are used to encode and decode the audio information, determine the text information corresponding to the audio information, identify the standard terms and uncivilized terms in the text information, and determine the monitoring status of the second information corresponding to the audio information based on the number of the standard terms and the uncivilized terms; A video recognition model based on mask reconstruction and multimodal learning is used to identify the type and frequency of illegal actions in the video information, and based on the type and frequency, the third information monitoring status corresponding to the video information is determined; the illegal actions include: uncivilized actions and violent actions; A preset boundary deviation rule is used to determine whether the location information of the target person meets the preset location requirements, determine the degree of deviation of the location information, and determine the fourth information monitoring status corresponding to the location information based on the degree of deviation.

3. The method according to claim 2, characterized in that The preset parts in the appearance image information are identified by using a target detection model based on a neural network, the monitoring results corresponding to the preset parts are determined, and based on the monitoring results, the first information monitoring situation corresponding to the appearance image information is determined, which is achieved based on the following steps: Pre-training the first model to obtain the neural network-based target detection model; The neural network-based target detection model is used to identify preset parts in the appearance image information, the preset parts including but not limited to: head, upper limbs, legs and feet, and based on preset unqualified conditions, the monitoring results corresponding to the head, upper limbs, legs and feet are determined respectively; Based on at least one of the monitoring results, a first information monitoring situation corresponding to the appearance image information is determined.

4. The method according to claim 3, characterized in that The pre-training of the first model to obtain the neural network-based target detection model includes: Acquire the appearance image data of the target person for training, perform a first labeling process on the appearance image data of the target person based on a first preset label, and determine the preset part corresponding to the appearance image data of the target person; Based on the second preset label, the target person's appearance image data is subjected to a second labeling process to determine the monitoring result labels corresponding to the preset parts in the target person's appearance image data, wherein the monitoring result labels include but are not limited to: qualified, partially qualified and unqualified; The first model is trained using the labeled appearance image data of the target person, and the weight of the first model is adjusted during the training process to determine the first model with the optimal weight as the neural network-based target detection model.

5. The method according to claim 4, characterized in that A pre-trained deep speech learning model based on an encoder-decoder structure and a pre-trained named entity recognition model are used to encode and decode the audio information, determine the text information corresponding to the audio information, identify the standard terms and uncivilized terms in the text information, and determine the monitoring status of the second information corresponding to the audio information based on the number of the standard terms and the uncivilized terms, which is achieved based on the following steps: Pre-training the second model to obtain the pre-trained deep speech learning model based on the encoder-decoder structure; The pre-trained deep speech learning model based on the encoder-decoder structure is used to resample the audio information to determine the audio information with a uniform sampling rate, and the audio information with the uniform sampling rate is converted into a logarithmic Mel spectrogram, and the preset feature information is encoded and embedded in the logarithmic Mel spectrogram; the logarithmic Mel spectrogram is used to characterize the audio frequency and audio amplitude of the audio information of the target person; Using a decoder in the pre-trained deep speech learning model based on the encoder-decoder structure, converting the logarithmic Mel spectrogram into the text information; Using a pre-trained named entity recognition model, based on preset text recognition tags, to identify standard terms and uncivilized terms in the text information; Determining a monitoring status of the second information corresponding to the audio information based on the number of the standard terms and the uncivilized terms; The pre-training of the second model to obtain the pre-trained deep speech learning model based on the encoder-decoder structure includes: Acquire preset audio segment information and corresponding text information of standard terms and uncivilized terms, and determine audio recognition corpus based on the combination of the audio segment information and the text information; The audio recognition corpus is used to fine-tune the weights of the second model, and the second model with the optimal weights is determined to be used as the pre-trained deep speech learning model based on the encoder-decoder structure.

6. The method according to claim 5, characterized in that A video recognition model based on masked reconstruction and multimodal learning is used to identify the type and frequency of illegal actions in the video information, and based on the type and frequency, the monitoring status of the third information corresponding to the video information is determined, which is achieved based on the following steps: Using the first layer model in the video recognition model based on mask reconstruction and multimodal learning, partially occlude the video information to determine the video action embedding code with mask information; The first layer model is used to represent the masked video coding layer of the encoder and decoder structure, and the mask information is used to represent the portion of the video information that is blocked; The second layer model in the video recognition model based on masked reconstruction and multimodal learning is adopted to perform encoding and decoding processing based on the video action embedding coding with masked information and other preset coding to detect the type and frequency corresponding to the illegal action in the video information; the second layer model is used to characterize the multimodal learning layer, and other preset coding includes but is not limited to: video frames, single-frame video images, and text description information in the video information; Based on the type and frequency, determine the monitoring status of the third information corresponding to the video information.

7. The method according to claim 6, characterized in that The method further comprises: When the monitoring and analysis results indicate that the target person is unqualified, a voice reminder broadcast is sent to the target person, and the voice reminder broadcast includes but is not limited to: the first information monitoring situation is unqualified, the second information monitoring situation is unqualified, the third information monitoring situation is unqualified, the fourth information monitoring situation is unqualified, and basic information and precautions.

8. A multi-dimensional feature information monitoring device, characterized in that: The multi-dimensional feature information monitoring device comprises: A determination module, used to determine information monitoring conditions corresponding to a plurality of to-be-monitored information of a target person by using a multi-dimensional feature information monitoring model, wherein the to-be-monitored information includes but is not limited to: appearance image information, audio information, video information and location information of the target person; The monitoring and analysis module is used to monitor and analyze the target personnel based on the information monitoring situation and adopt preset monitoring and analysis standards to determine the monitoring and analysis results corresponding to the target personnel.

9. A computer device, characterized in that: include: A memory and a processor, wherein the memory and the processor are communicatively connected to each other, the memory stores computer instructions, and the processor executes the multi-dimensional feature information monitoring method according to any one of claims 1 to 7 by executing the computer instructions.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the multi-dimensional feature information monitoring method according to any one of claims 1 to 7.

Images