Object behavior monitoring method, object behavior monitoring device, and electronic device

Abstract

Embodiments of the present disclosure disclose an object behavior monitoring method, an object behavior monitoring device and an electronic device. A specific implementation of the method comprises: in response to receiving multi-modal monitoring data sent by a monitoring device, performing feature extraction on each image in a monitoring video or a sequence of monitoring images to obtain a sequence of image features; performing user behavior prediction according to the sequence of image features and generating text description data; generating unified text data based on the text description data of the multi-modal monitoring data; in response to determining that there is a security problem according to the unified text data, generating a prompt information and sending the prompt information to a corresponding object. The implementation is related to home security monitoring technology, can predict future behavior of a user according to multi-modal monitoring data, thereby realizing safety warning for behavior about to occur. Not only can the accuracy of the prediction result be improved, but also the occurrence of safety accidents can be reduced.

Description

Technical Field

[0001] The embodiments disclosed herein relate to the field of home safety monitoring technology, specifically to object behavior monitoring methods, object behavior monitoring devices, and electronic devices. Background Technology

[0002] With the development of technology, more and more research institutions and industries are beginning to study human behavior recognition. Behavior recognition technology is often used in various applications such as mobile health care, personal motion monitoring, and motion-sensing games. Most related behavior recognition technologies adopt computer vision-based behavior perception technology, such as camera surveillance equipment.

[0003] However, the inventors discovered that related behavior recognition technologies have limited observation range and are easily affected by factors such as lighting conditions and occlusion, thus failing to fully meet the needs of daily behavior perception. Furthermore, they typically analyze actual, ongoing user behavior and cannot predict potential security risks.

[0004] The information disclosed in this background section is only intended to enhance the understanding of the background of the inventive concept, and therefore may contain information that does not constitute prior art known to those skilled in the art. Summary of the Invention

[0005] The summary portion of this disclosure is intended to provide a brief overview of the concepts, which will be described in detail in the detailed description portion. This summary portion is not intended to identify key or essential features of the claimed technical solutions, nor is it intended to limit the scope of the claimed technical solutions.

[0006] Some embodiments of this disclosure provide object behavior monitoring methods, object behavior monitoring devices, electronic devices, computer-readable media, and computer program products to solve one or more of the technical problems mentioned in the background section above.

[0007] In a first aspect, some embodiments of this disclosure provide an object behavior monitoring method, comprising: in response to receiving multimodal monitoring data sent by a monitoring device, extracting features from each image in a monitoring video or monitoring image sequence to obtain an image feature sequence, wherein the multimodal monitoring data includes at least one of the following: monitoring video, monitoring image sequence; predicting object behavior based on the image feature sequence and generating text description data; generating unified text data based on the text description data of the multimodal monitoring data; and in response to determining that a security problem exists based on the unified text data, generating a prompt message and sending the prompt message to the corresponding object.

[0008] In some embodiments, object behavior prediction is performed based on image feature sequences, and text description data is generated, including: inputting the image feature sequences into a pre-trained time series model, predicting the behavior that the object will exhibit, and generating text description data of the monitoring data indicated by the image feature sequences based on the image feature sequences and the predicted behavior; wherein the time series model is trained using sample data, and the sample data includes sample videos and corresponding sample descriptions.

[0009] In some embodiments, the multimodal monitoring data further includes monitoring audio; and the method further includes: converting the monitoring audio to generate corresponding text description data.

[0010] In some embodiments, generating unified text data based on the text description data of multimodal monitoring data includes: using a natural language processing toolkit to concatenate and process the text description data of each monitoring data in the multimodal monitoring data to generate unified text data.

[0011] In some embodiments, the method further includes: processing unified text data using a language model that supports multiple languages ​​and multiple modalities to generate enhanced text data, wherein the enhanced text data is used to highlight the behavior of the target object and security risk events; extracting features from the enhanced text data to generate text feature vectors, and predicting security levels based on the text feature vectors; and generating a prompt message in response to determining that a security problem exists based on the unified text data, including: generating a prompt message using a language model based on the enhanced text data and the predicted security level in response to determining that a security problem exists based on the predicted security level.

[0012] In some embodiments, the security level prediction based on the text feature vector includes: performing similarity analysis between the text feature vector and each feature vector in a pre-built vector database, wherein the vector database stores feature vectors and corresponding security level identifiers; using the feature vector with the highest similarity as the matching vector; and determining the security level corresponding to the matching vector as the predicted security level of the text feature vector.

[0013] In some embodiments, the vector database is constructed by: extracting features from the acquired raw text data to obtain feature vectors; storing the feature vectors using a similarity search library; and marking the feature vectors with security level identifiers, wherein the security level identifiers represent security levels including at least one of the following: safe, general, non-hazardous, potentially hazardous, and high-risk; and the method further includes: in response to receiving feedback data from the object regarding the prompt information, updating the vector database based on the feedback data to adjust the security level identifiers corresponding to the matching vectors.

[0014] In some embodiments, the monitoring device further includes a robot that interacts with the object via voice; and sending prompts to the corresponding object, including: locating the target object indicated by the multimodal monitoring data through the robot, sending the prompts to the target object via voice broadcast, and obtaining the voice feedback data of the target object.

[0015] Secondly, some embodiments of this disclosure provide an object behavior monitoring device, including: an image feature extraction unit configured to extract features from each image in a monitoring video or monitoring image sequence in response to receiving multimodal monitoring data sent by a monitoring device, to obtain an image feature sequence, wherein the multimodal monitoring data includes at least one of the following: monitoring video, monitoring image sequence; a text data generation unit configured to predict object behavior based on the image feature sequence and generate text description data; a unified text generation unit configured to generate unified text data based on the text description data of the multimodal monitoring data; and a prompt information generation unit configured to generate prompt information and send the prompt information to the corresponding object in response to determining that a security problem exists based on the unified text data.

[0016] In some embodiments, the text data generation unit is further configured to input an image feature sequence into a pre-trained time series model, predict the behavior that the object will exhibit, and generate text description data of the monitoring data indicated by the image feature sequence based on the image feature sequence and the predicted behavior; wherein the time series model is trained using sample data, which includes sample videos and corresponding sample descriptions.

[0017] In some embodiments, the multimodal monitoring data further includes monitoring audio; and the text data generation unit is further configured to convert the monitoring audio to generate corresponding text description data.

[0018] In some embodiments, the unified text generation unit is further configured to use a natural language processing toolkit to concatenate and process the text description data of each monitoring data in the multimodal monitoring data to generate unified text data.

[0019] In some embodiments, the object behavior monitoring device further includes a prediction unit configured to process unified text data using a language model that supports multiple languages ​​and multiple modalities to generate enhanced text data, wherein the enhanced text data is used to highlight and characterize object behavior and security risk events; to extract features from the enhanced text data to generate text feature vectors; and to predict the security level based on the text feature vectors; and a prompt information generation unit further configured to generate prompt information based on the language model in response to determining that a security problem exists based on the predicted security level.

[0020] In some embodiments, the prediction unit is further configured to perform similarity analysis between the text feature vector and each feature vector in a pre-built vector database, wherein the vector database stores feature vectors and corresponding security level identifiers; the feature vector with the highest similarity is used as the matching vector, and the security level corresponding to the matching vector is determined as the predicted security level of the text feature vector.

[0021] In some embodiments, the object behavior monitoring device further includes: a database construction unit configured to extract features from the acquired raw text data to obtain feature vectors; store the feature vectors using a similarity search library; and label the feature vectors with security level identifiers, wherein the security level identifiers represent security levels including at least one of the following: safe, general, non-hazardous, potentially hazardous, and high-risk; and a database update unit configured to update the vector database based on feedback data received from the object regarding the prompt information, so as to adjust the security level identifiers corresponding to the matching vectors.

[0022] In some embodiments, the monitoring device further includes a robot that interacts with the object via voice; and the prompt information generation unit is further configured to locate the target object indicated by the multimodal monitoring data through the robot, send the prompt information to the target object via voice broadcast, and obtain the voice feedback data of the target object.

[0023] Thirdly, some embodiments of this disclosure provide an electronic device, including: one or more processors; and a storage device having one or more programs stored thereon, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the object behavior monitoring method described in any of the implementations of the first aspect above.

[0024] Fourthly, some embodiments of this disclosure provide a computer-readable medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the object behavior monitoring method described in any of the implementations of the first aspect above.

[0025] Fifthly, some embodiments of this disclosure provide a computer program product, including a computer program that, when executed by a processor, implements the object behavior monitoring method described in any of the implementations of the first aspect above.

[0026] The various embodiments disclosed above have the following beneficial effects: The object behavior monitoring methods of some embodiments of this disclosure can predict future user behavior and assess potential security risks associated with upcoming behaviors. This can prevent security incidents in advance and reduce or avoid their occurrence. Specifically, most related human behavior recognition technologies employ computer vision-based behavior perception technology. This technology has a limited observation range and is easily affected by various factors such as lighting conditions and occlusion. Furthermore, most existing monitoring technologies analyze actual, ongoing user behavior and provide judgment results, but cannot predict future user behavior based on the on-site situation. Therefore, they cannot achieve early prevention of security risks to reduce the occurrence of such incidents.

[0027] Based on this, some embodiments of the object behavior monitoring method disclosed herein can acquire multimodal monitoring data. This enriches the monitoring data, reduces or avoids the influence of factors such as lighting and occlusion, and improves the accuracy of subsequent data analysis. Furthermore, the multimodal monitoring data can be converted into uniform modal text data. This achieves data format uniformity, facilitating subsequent processing. Subsequently, security risks can be identified based on the text data of different modal monitoring data. In addition, the method of this disclosure can also predict object behavior based on monitoring video or monitoring image sequences in the multimodal monitoring data, thereby knowing the behavior the object will exhibit. If a security problem is identified, a prompt message can be generated and sent to the corresponding object. This enables early prevention of security risks, eliminates security problems, or prevents the object from performing the intended behavior, thereby reducing or avoiding the occurrence of security incidents. Attached Figure Description

[0028] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and elements are not necessarily drawn to scale.

[0029] Figure 1 These are flowcharts of some embodiments of the object behavior monitoring method disclosed herein;

[0030] Figure 2 These are flowcharts of some other embodiments of the object behavior monitoring method disclosed herein;

[0031] Figure 3A yes Figure 2 A schematic diagram illustrating the generation process of the vector database;

[0032] Figure 3B yes Figure 2 A detailed flowchart of each stage in the process shown.

[0033] Figure 4 These are schematic diagrams illustrating the structure of some embodiments of the object behavior monitoring device disclosed herein;

[0034] Figure 5 This is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed Implementation

[0035] Embodiments of this disclosure will now be described in more detail with reference to the accompanying drawings. While some embodiments of this disclosure are shown in the drawings, it should be understood that this disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of this disclosure. It should be understood that the accompanying drawings and embodiments of this disclosure are for illustrative purposes only and are not intended to limit the scope of protection of this disclosure.

[0036] It should also be noted that, for ease of description, only the parts relevant to the invention are shown in the accompanying drawings. Unless otherwise specified, the embodiments and features described in this disclosure can be combined with each other.

[0037] It should be noted that the concepts of "first" and "second" mentioned in this disclosure are used only to distinguish different devices, modules or units, and are not used to limit the order of functions performed by these devices, modules or units or their interdependencies.

[0038] Furthermore, the terms “a” and “a plurality” used in this disclosure are illustrative rather than restrictive, and those skilled in the art should understand that, unless otherwise expressly indicated in the context, they should be understood as “one or more”.

[0039] Figure 1 A flow 100 of some embodiments of an object behavior monitoring method according to the present disclosure is shown. The method may include the following steps:

[0040] Step 101: In response to receiving multimodal monitoring data sent by the monitoring device, feature extraction is performed on each image in the monitoring video or monitoring image sequence to obtain an image feature sequence.

[0041] In some embodiments, the execution entity (such as a server) of the object behavior monitoring method disclosed herein can communicate with other electronic devices (such as security monitoring equipment like surveillance cameras, home robots, user terminals, etc.) via wired or wireless connections. Here, the monitoring equipment can be installed in the environment requiring monitoring, such as homes, convenience stores, banks, etc. The monitoring equipment can include traditional security monitoring equipment, such as surveillance cameras, recorders, etc. The surveillance camera can capture images or video. The video may contain only video images or may contain audio.

[0042] Here, the executing entity can collect monitoring data in multiple modalities through monitoring equipment, such as monitoring video, monitoring image sequences, and monitoring audio. After receiving the multimodal monitoring data sent by the monitoring equipment, the executing entity can extract features from each image in the monitoring video or monitoring image sequence to obtain an image feature sequence.

[0043] Understandably, for surveillance video, the executing entity can first extract video frames, treating each frame as an independent image. For each image, the executing entity can use a pre-trained image extraction model to extract features, thus obtaining the image features. The image extraction model can be any machine learning model with image feature extraction capabilities, such as a Convolutional Neural Network (CNN). CNNs are deep learning models particularly suitable for processing data with a grid structure, such as images and videos. CNNs typically extract and learn data features through convolutional operations, pooling operations, and fully connected layers. The training process of the image extraction model will not be elaborated here.

[0044] In some embodiments, the monitoring device disclosed herein may further include a robot, such as a home robot, capable of voice interaction with an object. A home robot is typically an intelligent robot designed to perform various tasks and provide services in a home environment. These robots typically possess advanced technologies such as artificial intelligence, autonomous navigation, voice recognition, and visual recognition, enabling them to sense and control various smart home devices, interact with family members, and assist them in completing various daily activities. Here, the robot, as an interactive device, can acquire audio through interaction with the user, thereby enriching the monitoring data. Furthermore, based on the on-site conditions, the robot can be used to acquire on-site images at closer range. This reduces or avoids the influence of factors such as lighting conditions and obstructions, improving the quality of the acquired images and contributing to the accuracy of subsequent behavior prediction.

[0045] Step 102: Predict object behavior based on image feature sequence and generate text description data.

[0046] In some embodiments, based on the image feature sequence obtained in step 101, the executing entity can predict object behavior and generate corresponding text description data. This involves processing surveillance video or surveillance image sequences into single-modal text data. For example, the executing entity can input the image feature sequence into a pre-trained time series model to predict the behavior of an object. The object here is generally the object indicated by the surveillance data, such as a family member, a stranger, or a pet. Then, the executing entity can generate corresponding text description data for the surveillance data based on the events represented by the predicted object behavior.

[0047] In some embodiments, to improve the accuracy of the generated text description data, the executing entity can also generate text description data of the input data using the aforementioned time series model. This involves inputting the image feature sequence into a pre-trained time series model to predict the behavior of the object, and generating text description data of the monitoring data indicated by the image feature sequence based on the image feature sequence and the predicted behavior. The time series model can be trained using sample data. Sample data can include sample videos and corresponding sample descriptions, such as the widely used video description dataset MSR-VTT (Microsoft Research Video to Text). This dataset aims to advance research in video understanding and description generation, particularly in multimodal learning, video retrieval, and video description generation. The time series model is not limited here; for example, a Long Short-Term Memory (LSTM) network or an autoregressive moving average (ARMA) model can be used.

[0048] It should be noted that during the training of the time series model, the model parameters can be adjusted by weighting the behavior error and the description error. The behavior error is typically the loss function value between the predicted behavior of a sample object at a given moment in the video and the actual behavior of the sample object. The description error is typically the loss function value between the predicted description of the sample and the corresponding description of the sample video.

[0049] In some application scenarios, where multimodal monitoring data also includes monitoring audio, the executing entity can further process the monitoring audio to generate corresponding text description data. This involves converting monitoring data from different modalities into single-modal text data, achieving modality unification and facilitating subsequent processing. As an example, the executing entity can use a third-party tool (such as Speech-to-Text) via an interface to convert monitoring audio into text.

[0050] Step 103: Generate unified text data based on the text description data of the multimodal monitoring data.

[0051] In some embodiments, based on the text description data of the multimodal monitoring data generated in step 102, the executing entity can generate unified text data. For example, the executing entity can concatenate the text description data of each modal monitoring data in the order of video, image, and audio to obtain unified text data.

[0052] Optionally, the execution entity can use a Natural Language Toolkit (NLTK) to concatenate and process the textual description data of each monitoring data point in the multimodal monitoring data, thereby generating unified text data. NLTK is typically a powerful Python library for processing and analyzing natural language text. It provides a wealth of tools and resources for tasks such as text processing, annotation, parsing, semantic analysis, and text classification.

[0053] Step 104: In response to determining that a security issue exists based on the unified text data, generate a prompt message and send the prompt message to the corresponding object.

[0054] In some embodiments, based on the unified text data generated in step 103, the executing entity can generate a prompt message when it determines that a security issue exists. This prompt message can then be sent to the corresponding object. For example, the executing entity can send the prompt message to a user terminal connected to it. Alternatively, to avoid security risks, the executing entity can also use monitoring equipment to directly send prompt messages to the object indicated by the monitoring data, thereby achieving timely early warning of security risks and reducing or avoiding the occurrence of security incidents.

[0055] In some embodiments, where the monitoring equipment includes a robot, the executing entity can also send prompts to the corresponding object via the robot. Specifically, after receiving instructions from the executing entity, the robot can first locate the target object indicated by the multimodal monitoring data. Then, it can send the prompts to the target object via voice broadcast. Furthermore, the robot can obtain the target object's voice feedback data regarding the prompts.

[0056] As described above, some embodiments of the object behavior monitoring method disclosed herein can acquire multimodal monitoring data. This enriches the monitoring data, reduces or avoids the influence of factors such as lighting and occlusion, and improves the accuracy of subsequent data analysis. Furthermore, the multimodal monitoring data can be converted into uniform modal text data. This achieves data format uniformity, facilitating subsequent processing. Subsequently, security risks can be identified based on the text data of different modal monitoring data. In addition, the method disclosed herein can also predict object behavior based on monitoring video or monitoring image sequences in the multimodal monitoring data, thereby knowing the behavior the object will exhibit. If a security problem is identified, a prompt message can be generated and sent to the corresponding object. This enables early prevention of security risks, eliminates security problems, or prevents the object from performing the intended behavior, thereby reducing or avoiding the occurrence of security incidents.

[0057] In some application scenarios, the executing entity can use a language model that supports multiple languages ​​and multiple modalities of data, such as Figure 3B In the prediction phase, GPT-4o (Generative Pre-trained Transformer 4 Optimized) is used to predict security issues. GPT-4o is typically based on the GPT-4 framework, supports multiple languages ​​and modalities, and can be used to handle complex language tasks and generate user-friendly contextual information.

[0058] Specifically, firstly, the aforementioned language model can be used to process the unified text data to generate enhanced text data. This enhanced text data can be used to highlight the behavior of the target object and security risks. For example, data representing target behavior and security risks in the unified text data can be labeled to focus attention on these data. The labeling method is not limited. Next, feature extraction can be performed on the enhanced text data, such as using a pre-trained BERT (Bidirectional Encoder Representations from Transformers) model to generate text feature vectors. Then, security level prediction can be performed based on the text feature vectors. Different security levels typically represent different degrees of security risks. If a security problem is determined based on the predicted security level, a warning message can be generated using the language model based on the enhanced text data and the predicted security level. This warning message can include alerts and suggestions regarding the security risks.

[0059] In some embodiments, such as Figure 2 As shown, the executing entity can predict security levels using a vector database. Specifically, as... Figure 3BAs shown, a similarity analysis can be performed between the text feature vector and each feature vector in a pre-built vector database. The vector database can store feature vectors and their corresponding security level identifiers (or tags). Therefore, the feature vector with the highest similarity can be used as the matching vector. Furthermore, the security level corresponding to this matching vector can be determined as the predicted security level of the text feature vector, thus achieving security level matching.

[0060] Here, as Figure 3A As shown, a vector database can be constructed using a standard text dataset. First, feature extraction can be performed on the acquired raw text data, such as using the BERT model mentioned above, to obtain feature vectors. The raw text data here is usually standardized text data. This helps improve the accuracy of security level matching. Next, a similarity search library (such as Faiss, Facebook AI Similarity Search) can be used to store the feature vectors and label them with security level identifiers. The security level identifier can represent at least one of the following: safe, general, no risk, potentially dangerous, high risk, etc. Faiss is typically a high-efficiency similarity search library designed for similarity searches of large-scale vector data. It is mainly used for fast searching and retrieving high-dimensional vector data and is widely used in recommender systems, image retrieval, text search, and other fields.

[0061] Understandably, by combining Faiss with the storage and security level labeling of extracted feature vectors, an index can be built. The stored feature vectors can then be used for similarity matching. The vector database can match the input text feature vectors with the output vectors based on their similarity ranking, and use the ranking priority to obtain the label (i.e., security level identifier) ​​of the similar vectors. This label can then be returned as the label for the input text feature vectors.

[0062] Furthermore, upon receiving feedback data from the object regarding the prompt information, the executing entity can update the vector database based on the feedback data to adjust the security level identifier corresponding to the matching vector. For example, such as... Figure 3B As shown, if a security issue is identified requiring preventative measures, the robot can trigger post-security processing logic. Through a search function, it can interact with the user to deliver the alert to the specific user and their guardian. User feedback is gathered during the interaction between the user and the robot. The identifiers in the vector database are updated based on this feedback. For example, if the robot alerts the user to a potential danger, and the user acknowledges and takes immediate action, the identifiers in the vector database (similar vectors matched in this text) can be updated based on the user's voice feedback (security level upgrade or downgrade).

[0063] It's important to note that as more people focus on home security, home-based elderly care and child protection have become key concerns. Home security scenarios are ubiquitous: outdoors, there may be fights, strangers loitering, signs of fire, or the accumulation of hazardous materials; indoor security concerns include spilled medicine, water on the floor, items placed within children's reach, children playing alone on sofas or balconies (posing a fall risk), and children accessing household appliances; for elderly people with mobility issues, there may be slippery floors or them trying to reach out of reach. If these scenarios are not prevented in advance, potentially dangerous behaviors could lead to irreversible consequences. Therefore, the awareness of user safety behavior both indoors and outdoors needs to be taken seriously.

[0064] However, most existing monitoring equipment analyzes and judges actual user behavior in progress, but cannot predict future user behavior based on the on-site situation, nor can it assess the security level of upcoming behaviors. Furthermore, existing technologies often lack feedback logic and accurate security level information when classifying security information resulting from user behavior.

[0065] Based on this, such as Figure 2 As shown, the object behavior monitoring method disclosed herein can collect multimodal data such as images, videos, and audio, and process the multimodal data using relevant algorithms, that is, convert the collected data into text, and then into vectors. It can also match a vector database with security level classification (i.e., a vector database with security level identifiers) to identify the security level and generate warning interaction scripts. Furthermore, it can obtain user feedback based on the voice interaction function of a home robot, and update the security level identifiers in the vector database according to the feedback information. That is, the security level is iteratively updated. This vector based on standard text can better unify the expression of multimodal text information. Combined with feedback logic, the security level information can be enriched during use, thus closely approximating the user's actual usage scenario.

[0066] In other words, the object behavior monitoring method disclosed herein can predict the security level of multimodal user behavior based on a vector database. It mainly includes four stages: collection, processing, prediction, and feedback. The specific processing flow for each stage can be found in [reference needed]. Figure 3BAs shown, this involves connecting security monitoring devices such as cameras to an IoT platform. These devices can report data in real time, including image sets and videos. The robot also reports user voice interaction information in real time. The collected multimodal data is converted and merged into text information. GPT-4o refines and expands this text information to generate enhanced text. This enhanced text is further converted into vectors, which are used to match a vector database to predict security levels. GPT-4o uses the matched security level and enhanced text to generate response text. The robot interacts with the user via voice, updating the vector database based on the user's voice feedback (affirmation, questioning, negation).

[0067] Further reference Figure 4 As a response to the above Figures 1 to 3B The implementation of the method shown in this disclosure provides some embodiments of an object behavior monitoring device. These device embodiments are similar to... Figures 1 to 3B The methods and embodiments shown correspond to those described. This device can be specifically applied to various electronic devices.

[0068] like Figure 4 As shown, the object behavior monitoring device 400 in some embodiments may include: an image feature extraction unit 401, configured to extract features from each image in a monitoring video or monitoring image sequence in response to receiving multimodal monitoring data sent by a monitoring device, to obtain an image feature sequence, wherein the multimodal monitoring data includes at least one of the following: monitoring video, monitoring image sequence; a text data generation unit 402, configured to predict object behavior based on the image feature sequence and generate text description data; a unified text generation unit 403, configured to generate unified text data based on the text description data of the multimodal monitoring data; and a prompt information generation unit 404, configured to generate prompt information and send the prompt information to the corresponding object in response to determining that a security problem exists based on the unified text data.

[0069] In some embodiments, the text data generation unit 402 may be further configured to input image feature sequences into a pre-trained time series model, predict the behavior that the object will exhibit, and generate text description data of the monitoring data indicated by the image feature sequences based on the image feature sequences and the predicted behavior; wherein the time series model is trained using sample data, and the sample data includes sample videos and corresponding sample descriptions.

[0070] In some embodiments, the multimodal monitoring data may further include monitoring audio; and the text data generation unit 402 is further configured to convert the monitoring audio to generate corresponding text description data.

[0071] In some embodiments, the unified text generation unit 403 may be further configured to use a natural language processing toolkit to concatenate and process the text description data of each monitoring data in the multimodal monitoring data to generate unified text data.

[0072] In some embodiments, the object behavior monitoring device 400 may further include a prediction unit (not shown in the figure), configured to process unified text data using a language model that supports multiple languages ​​and multiple modalities, to generate enhanced text data, wherein the enhanced text data is used to highlight and characterize object behavior and security risk events; to extract features from the enhanced text data to generate text feature vectors, and to predict the security level based on the text feature vectors; and the prompt information generation unit 404 may be further configured to generate prompt information based on the enhanced text data and the predicted security level, in response to determining that a security problem exists based on the predicted security level, through the language model.

[0073] In some embodiments, the prediction unit may be further configured to perform similarity analysis between the text feature vector and each feature vector in a pre-built vector database, wherein the vector database stores feature vectors and corresponding security level identifiers; the feature vector with the highest similarity is used as the matching vector, and the security level corresponding to the matching vector is determined as the predicted security level of the text feature vector.

[0074] In some embodiments, the object behavior monitoring device 400 may further include: a database construction unit (not shown in the figure), configured to extract features from the acquired raw text data to obtain feature vectors; store the feature vectors using a similarity search library; and mark the security level identifiers of the feature vectors, wherein the security level identifiers represent at least one of the following: safe, general, non-hazardous, potentially hazardous, and high-risk; and a database update unit (not shown in the figure), configured to update the vector database according to the feedback data received from the object in response to the prompt information, so as to adjust the security level identifiers corresponding to the matching vectors.

[0075] In some embodiments, the monitoring device may further include a robot that interacts with the object via voice; and the prompt information generation unit 404 may be further configured to locate the target object indicated by the multimodal monitoring data through the robot, send the prompt information to the target object via voice broadcast, and obtain the voice feedback data of the target object.

[0076] It is understandable that the units recorded in the object behavior monitoring device 400 are related to the reference. Figures 1 to 3BThe steps in the described method correspond to each other. Therefore, the operations, features, and beneficial effects described above for the method also apply to the object behavior monitoring device 400 and the units contained therein, and will not be repeated here.

[0077] The following is for reference. Figure 5 It shows a schematic diagram of the structure of an electronic device 500 suitable for implementing some embodiments of the present disclosure. Figure 5 The electronic device shown is merely an example and should not be construed as limiting the functionality and scope of the embodiments of this disclosure.

[0078] like Figure 5 As shown, the electronic device 500 may include a processing unit 501 (e.g., a central processing unit, a graphics processor, etc.), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. The RAM 503 also stores various programs and data required for the operation of the electronic device 500. The processing unit 501, ROM 502, and RAM 503 are interconnected via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0079] Typically, the following devices can be connected to I / O interface 505: input devices 506 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 507 including, for example, speakers, vibrators, etc.; storage devices 508 including, for example, disks, hard drives, memory cards, etc.; and communication devices 509. Communication device 509 allows electronic device 500 to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 5 An electronic device 500 with various devices is shown; however, it should be understood that it is not required to implement or possess all of the devices shown. More or fewer devices may be implemented or possessed alternatively. Figure 5 Each box shown can represent a device or multiple devices as needed.

[0080] In particular, according to some embodiments of this disclosure, the processes described above with reference to the flowcharts can be implemented as computer software programs. For example, some embodiments of this disclosure include a computer program product comprising a computer program carried on a computer-readable medium, the computer program containing program code for performing the methods shown in the flowcharts. In such embodiments, the computer program can be downloaded and installed from a network via communication device 509, or installed from storage device 508, or installed from ROM 502. When the computer program is executed by processing device 501, it performs the functions defined in the methods of some embodiments of this disclosure.

[0081] It should be noted that, in some embodiments of this disclosure, the computer-readable medium may be a computer-readable signal medium or a computer-readable storage medium, or any combination thereof. A computer-readable storage medium may be, for example,—but not limited to—an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any combination thereof. More specific examples of a computer-readable storage medium may include, but are not limited to: an electrical connection having one or more wires, a portable computer disk, a hard disk, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage device, magnetic storage device, or any suitable combination thereof. In some embodiments of this disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in conjunction with an instruction execution system, apparatus, or device. In some embodiments of this disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, carrying computer-readable program code. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium can be any computer-readable medium other than a computer-readable storage medium, which can send, propagate, or transmit a program for use by or in connection with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium can be transmitted using any suitable medium, including but not limited to: wires, optical fibers, RF (radio frequency), etc., or any suitable combination thereof.

[0082] In some implementations, clients and servers can communicate using any currently known or future-developed network protocol such as HTTP (Hypertext Transfer Protocol) and can interconnect with digital data communication (e.g., communication networks) of any form or medium. Examples of communication networks include local area networks (“LANs”), wide area networks (“WANs”), the Internet (e.g., the Internet of Things), and end-to-end networks (e.g., ad hoc end-to-end networks), as well as any currently known or future-developed networks.

[0083] The aforementioned computer-readable medium may be included in the aforementioned electronic device; or it may exist independently and not assembled into the electronic device. The aforementioned computer-readable medium carries one or more programs that, when executed by the electronic device, cause the electronic device to: in response to receiving multimodal monitoring data sent by a monitoring device, extract features from each image in the monitoring video or monitoring image sequence to obtain an image feature sequence, wherein the multimodal monitoring data includes at least one of the following: monitoring video, monitoring image sequence; predict object behavior based on the image feature sequence and generate text description data; generate unified text data based on the text description data of the multimodal monitoring data; and in response to determining a security issue based on the unified text data, generate a prompt message and send the prompt message to the corresponding object.

[0084] Furthermore, computer program code for performing operations of some embodiments of this disclosure can be written in one or more programming languages ​​or a combination thereof. Programming languages ​​include object-oriented programming languages—such as Java, Smalltalk, and C++—and conventional procedural programming languages—such as the "C" language or similar programming languages. The program code can be executed entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving remote computers, the remote computer can be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or can be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0085] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of this disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of code containing one or more executable instructions for implementing a specified logical function. It should also be noted that in some alternative implementations, the functions indicated in the blocks may occur in a different order than those indicated in the drawings. For example, two consecutively indicated blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, can be implemented using a dedicated hardware-based system that performs the specified function or operation, or using a combination of dedicated hardware and computer instructions.

[0086] The units described in some embodiments of this disclosure can be implemented in software or hardware. The described units can also be housed in a processor; for example, a processor may be described as including an image feature extraction unit, a text data generation unit, a unified text generation unit, and a prompt information generation unit. The names of these units do not necessarily limit the specific unit; for example, the image feature extraction unit may also be described as "a unit that extracts features from each image in a surveillance video or a sequence of surveillance images."

[0087] The functions described above in this document can be performed, at least in part, by one or more hardware logic components. For example, exemplary types of hardware logic components that can be used, without limitation, include: Field Programmable Gate Arrays (FPGAs), Application-Specific Integrated Circuits (ASICs), Application Standard Products (ASSPs), System-on-Chip (SoCs), Complex Programmable Logic Devices (CPLDs), and so on.

[0088] Some embodiments of this disclosure also provide a computer program product, including a computer program that, when executed by a processor, implements any of the object behavior monitoring methods described above.

[0089] The above description is merely a selection of preferred embodiments of this disclosure and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of the invention involved in the embodiments of this disclosure is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the above-described inventive concept. For example, technical solutions formed by substituting the above-described features with (but not limited to) technical features with similar functions disclosed in the embodiments of this disclosure.

Claims

1. A method for monitoring object behavior, comprising: In response to receiving multimodal monitoring data sent by a monitoring device, feature extraction is performed on each image in the monitoring video or monitoring image sequence to obtain an image feature sequence, wherein the multimodal monitoring data includes at least one of the following: monitoring video, monitoring image sequence; Based on the image feature sequence, object behavior is predicted, and text description data is generated; Based on the text description data of the multimodal monitoring data, unified text data is generated; In response to determining a security issue based on the unified text data, a prompt message is generated and sent to the corresponding object.

2. The object behavior monitoring method according to claim 1, wherein, The step of predicting object behavior based on the image feature sequence and generating text description data includes: The image feature sequence is input into a pre-trained time series model to predict the behavior that the object will exhibit, and based on the image feature sequence and the predicted behavior, textual description data of the monitoring data indicated by the image feature sequence is generated. The time series model is trained using sample data, which includes sample videos and corresponding sample descriptions.

3. The object behavior monitoring method according to claim 1, wherein, The multimodal monitoring data also includes monitoring audio; as well as The method further includes: The monitored audio is converted to generate corresponding text description data.

4. The object behavior monitoring method according to claim 1, wherein, The text description data based on the multimodal monitoring data is used to generate unified text data, including: A natural language processing toolkit is used to concatenate and process the text description data of each monitoring data in the multimodal monitoring data to generate unified text data.

5. The object behavior monitoring method according to claim 1, wherein, The method further includes: A language model that supports multiple languages ​​and multiple modalities is used to process the unified text data to generate enhanced text data, wherein the enhanced text data is used to highlight the behavior of the target object and security risk events. Feature extraction is performed on the enhanced text data to generate a text feature vector, and a security level prediction is performed based on the text feature vector; and The response, based on the unified text data, determines that a security issue exists and generates a prompt message, including: In response to determining that a security issue exists based on the predicted security level, a prompt message is generated using the language model based on the enhanced text data and the predicted security level.

6. The object behavior monitoring method according to claim 5, wherein, The step of predicting the security level based on the text feature vector includes: The text feature vector is compared with each feature vector in a pre-built vector database for similarity analysis. The vector database stores feature vectors and their corresponding security level identifiers. The feature vector with the highest similarity is used as the matching vector, and the security level corresponding to the matching vector is determined as the predicted security level of the text feature vector.

7. The object behavior monitoring method according to claim 6, wherein, The vector database was constructed using the following method: Feature extraction is performed on the acquired raw text data to obtain feature vectors; The feature vectors are stored using a similarity search library, and a security level identifier is used to label the feature vectors. The security level identifier represents a security level including at least one of the following: safe, general, no risk, possible risk, and high risk. The method further includes: In response to receiving feedback data from the object regarding the prompt information, the vector database is updated based on the feedback data to adjust the security level identifier corresponding to the matching vector.

8. The object behavior monitoring method according to any one of claims 1-7, wherein, The monitoring equipment also includes a robot that can interact with the object via voice. as well as Sending the prompt information to the corresponding object includes: The robot locates the target object indicated by the multimodal monitoring data, sends the prompt information to the target object via voice broadcast, and obtains the target object's voice feedback data.

9. An object behavior monitoring device, comprising: An image feature extraction unit is configured to extract features from each image in a monitoring video or monitoring image sequence in response to receiving multimodal monitoring data sent by a monitoring device, thereby obtaining an image feature sequence, wherein the multimodal monitoring data includes at least one of the following: monitoring video or monitoring image sequence; The text data generation unit is configured to predict object behavior based on the image feature sequence and generate text description data; A unified text generation unit is configured to generate unified text data based on text description data of the multimodal monitoring data; The prompt message generation unit is configured to generate a prompt message in response to determining that a security problem exists based on the unified text data, and to send the prompt message to the corresponding object.

10. An electronic device, comprising: One or more processors; Storage device, on which one or more programs are stored, When the one or more programs are executed by the one or more processors, the one or more processors implement the object behavior monitoring method as described in any one of claims 1-8.

11. A computer-readable medium having a computer program stored thereon, wherein, When the computer program is executed by the processor, it implements the object behavior monitoring method as described in any one of claims 1-8.

12. A computer program product comprising a computer program that, when executed by a processor, implements the object behavior monitoring method as described in any one of claims 1-8.

Images