A monitoring method, apparatus and vehicle

Abstract

The application provides a monitoring method, device and vehicle. The method comprises: acquiring a first image collected by a camera, the first image comprising a first object; acquiring first feature information according to the first image, the first feature information being used to describe a first state of the first object; generating a second image according to the first feature information, the second image being used to display the first state; and controlling a display device to display the second image. The application can provide timely and effective feedback on the monitoring result, accurately analyze the reasons corresponding to different monitoring results, and provide corresponding specific measures, thereby helping the user to more quickly understand the monitoring situation and quickly respond.

Description

Technical Field

[0001] This application relates to the field of computer technology, and more specifically, to a monitoring method, apparatus, and vehicle. Background Technology

[0002] In recent years, family driving scenarios have become increasingly common. During driving, parents, as the drivers, need to maintain a high level of concentration to ensure driving safety, especially at high speeds or in noisy environments. When children in the back seat cry or encounter dangerous situations, parents may not be able to monitor their children's condition in time. Therefore, there is an urgent need to find a method to monitor and provide real-time feedback on the status of children in the back seat to ensure driving safety in family driving scenarios. Summary of the Invention

[0003] This application provides a monitoring method. It can provide timely and effective feedback on monitoring results, accurately analyze the causes of different monitoring results, and thus provide corresponding specific measures to help users understand the monitoring situation more quickly and react swiftly.

[0004] In a first aspect, a monitoring method is provided, comprising: acquiring a first image captured by a camera, the first image including a first object; acquiring first feature information based on the first image, the first feature information being used to describe a first state of the first object; generating a second image based on the first feature information, the second image being used to display the first state; and controlling a display device to display the second image.

[0005] The embodiments of this application do not limit the first object, which can be a person, an animal, or an article.

[0006] In one possible implementation, after acquiring the first image captured by the camera module, the method further includes: preprocessing the first image, wherein the preprocessing includes at least one of the following: noise reduction, cropping, and resolution adjustment.

[0007] In one possible implementation, the first feature information may be textual feature information used to describe the first object.

[0008] For example, the first feature information can be text content expressed in natural language. For instance, the first feature information could be a description such as "the child has unbuckled their seatbelt" or "the puppy is playing."

[0009] For example, the first feature information can also be a text feature vector. A text feature vector is a method of converting text content expressed in natural language into a numerical representation. This vector can capture the semantic information implicit in the text content, enabling the similarity between texts and other features to be quantified in a high-dimensional space.

[0010] In one possible implementation, a first image containing a child in the back seat of a vehicle can be input into a contrastive learning model. The contrastive learning model extracts the visual features of the child in the back seat and outputs first feature information, which is text information: "The child is crying loudly in the back seat".

[0011] In the above technical solution, the second image can visualize the first feature information extracted from the first image, with the aim of highlighting the specific information that the user cares about most.

[0012] In one possible implementation, multiple frames of the second image can be played in chronological order to form a dynamic animation. This dynamic animation can then visually display the relevant state of the first object to the user.

[0013] Based on the above technical solution, the diverse states of the first object can be dynamically monitored, and corresponding second images can be generated for feedback based on different states in the actual situation. This improves the monitoring capability of the first object, as well as the flexibility and accuracy of the feedback, allowing users to quickly understand the situation of the first object and thus make timely responses.

[0014] In conjunction with the first aspect, in some implementations of the first aspect, the first state is the negative state of the first object.

[0015] Negative states can typically be undesirable situations, such as unfavorable developmental states, dangerous behaviors, negative emotions, harmful situations, and so on.

[0016] For example, when the first subject is a child, a negative state can refer to an abnormal or undesirable state that the child exhibits in terms of emotions, behavior, health, or safety over a certain period of time. Examples include emotional fluctuations (such as crying or anxiety), behavioral problems (such as kicking or not wearing a seatbelt), or physical discomfort (such as fever or headache).

[0017] Based on the above technical solution, timely identification of negative states can effectively prevent accidents from occurring and reduce harm to the primary target.

[0018] In conjunction with the first aspect, in some implementations of the first aspect, when the first object is inside the vehicle, before generating the second image, the method further includes: acquiring vehicle state data collected by sensors; acquiring second feature information based on the vehicle state data, the second feature information being used to describe the state of the vehicle; and generating the second image based on the first feature information, including: generating the second image based on the first feature information and the second feature information.

[0019] For example, vehicle status data may include: vehicle speed, temperature, bumpiness, and noise.

[0020] In one possible implementation, the second feature information can be a vehicle state feature vector. The vehicle state feature vector can be a mathematical representation output from the vehicle state data after processing by the vehicle state information module.

[0021] Based on the above technical solution, the in-vehicle environment and the state of the first object can be analyzed by combining visual data and vehicle status data, thereby providing a more comprehensive analysis of the in-vehicle environment and improving the accuracy of monitoring and feedback.

[0022] In conjunction with the first aspect, in some implementations of the first aspect, before generating the second image, the method further includes: acquiring audio data collected by a microphone; acquiring third feature information based on the audio data, the third feature information being used to describe the sound emitted by the first object and / or the sound in the environment; generating the second image based on the first feature information, including: generating the second image based on the first feature information and the third feature information.

[0023] For example, the sound emitted by the first object can be a crying sound, a laughing sound, a breathing sound, a babbling sound, a talking sound, etc.

[0024] For example, ambient sounds inside the vehicle could be sounds played by electronic devices, conversations between other users who are not the primary user, etc.

[0025] In one possible implementation, the third feature information can be an audio feature vector. The audio feature vector can be a series of quantized features extracted from the raw audio data acquired by the microphone, and may include at least one of the following: mel-spectrogram, mel-frequency cepstral coefficients (MFCCs), pitch, loudness, and emotional features.

[0026] Based on the above technical solution, the environment inside the vehicle and the state of the first object can be analyzed by combining audio data, thereby improving the accuracy of monitoring and feedback.

[0027] In one possible implementation, generating a second image based on the first feature information includes: generating a second image based on the first feature information, the second feature information, and the third feature information.

[0028] In one possible implementation, generating a second image based on the first feature information includes: generating a second image based on the first feature information and the third feature information.

[0029] In this implementation, a second image is generated by combining data from multiple modalities. This multi-modal data enhances the vehicle's perception of different situations, improving the reliability of vehicle monitoring and its real-time feedback capabilities. In other words, comprehensively considering visual data, vehicle status data, and audio data facilitates comprehensive monitoring of the first object.

[0030] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: generating care information based on the first feature information, the care information including: the cause of the first state; and the control prompting device prompting the care information.

[0031] Among them, care information can be a comprehensive message that includes analysis, interpretation, and reminders about the current state or behavior of the first object.

[0032] The cause of the first state can be internal to the first object, such as physiological needs or emotional changes, or it can be external, such as environmental changes or intervention from other objects.

[0033] For example, in the case where the first object is a child, the child's first state is "crying". The reasons for the child crying may be: the child is physically uncomfortable, the child is choking on food, the ambient temperature is too cold or too hot, the road is too bumpy, etc.

[0034] In one possible implementation, care information can be generated based on the first feature information and the second feature information.

[0035] In one possible implementation, care information can be generated based on the first feature information and the third feature information.

[0036] In one possible implementation, care information can be generated based on the first feature information, the second feature information, and the third feature information.

[0037] As mentioned above, the technical solution provided in this application embodiment can not only monitor the state of the first object using a camera, but also integrate data from multiple modalities to achieve more comprehensive monitoring and feedback.

[0038] In conjunction with the first aspect, in some implementations of the first aspect, care information also includes: responses to the causes.

[0039] The response measures are designed to guide users on how to help the first object improve its current state.

[0040] For example, in situations where the first object is in danger or an emergency, the response typically requires immediate action from the user. For instance, if a child is choking on food, the vehicle can prompt the driver to pull over and check on the child as soon as possible.

[0041] Based on the above technical solution, users can be provided with comprehensive feedback on the first object. The care information goes beyond simple status notifications; it can include a summary of the first state, analysis of possible causes, and recommended coping strategies. This allows users to quickly understand the situation of the first object, saving them time spent continuously observing it. Simultaneously, the care information helps users gain a deeper understanding of the reasons behind different states of the first object, enabling them to take more accurate and appropriate measures.

[0042] In one possible implementation, the method further includes: inputting the second image output by the state diagram generation module and the care information into a user interface (UI) display module; and controlling the prompting device to prompt the image and text output by the UI display module.

[0043] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: controlling the display device to display a care interface, the care interface including an animation display area, a text prompt area, a multi-function button area, etc.; controlling the display device to display a second image, including: controlling the animation display area to display the second image.

[0044] Based on the above technical solution, an intuitive way to monitor the primary object can be provided. This user-friendly interface can display information of interest to the user in real time, helping them quickly understand the status of the primary object. Especially in scenarios involving in-vehicle monitoring, it helps drivers concentrate on driving, reducing the safety risks caused by frequently focusing on the primary object inside the vehicle.

[0045] In conjunction with the first aspect, in certain implementations of the first aspect, the control prompting device prompts care information, including: controlling the display device to display care information in a text prompting area, and / or controlling the voice broadcasting device to broadcast care information.

[0046] Based on the above technical solutions, different ways of delivering care information can be provided to users, which helps users receive care information according to different scenarios or personal choices, thereby improving their ability to cope with the first state.

[0047] In conjunction with the first aspect, in some implementations of the first aspect, the method further includes: generating speech adjustment parameters based on the first feature information; and controlling the speech broadcasting device to adjust the tone, speed and / or timbre of the speech broadcast based on the speech adjustment parameters.

[0048] Among them, voice adjustment parameters can be a series of variables or parameters used to adjust and control voice broadcasting.

[0049] For example, the vehicle detects that a child in the back seat is becoming restless due to a prolonged ride. Therefore, the vehicle can adjust its voice broadcast system, including adjusting the tone to be gentle and soothing, the speed to be slow, and the timbre to be soft and low. The vehicle then broadcasts to the child via the voice interaction device: "Honey, don't worry, we're almost there!"

[0050] For example, if the vehicle detects that a child in the back seat is exhibiting mild discomfort, such as sneezing or a slight cough, which is not serious but still requires the driver's attention, the vehicle can adjust its voice broadcast system by: adjusting the tone of voice to "smooth and concerned," the speech rate to "medium," and the timbre to "clear and soothing." The vehicle might broadcast to the driver: "The child in the back seat seems to be unwell, possibly with cold symptoms. Please monitor the child's health."

[0051] For example, if a vehicle detects that a child in the back seat has unbuckled their seatbelt, it needs to alert the driver of a potential safety hazard. Therefore, the vehicle can adjust its voice broadcast system, including adjusting the tone to a "serious, warning" manner, the speech rate to a "faster" manner, and the timbre to a "high-pitched, bright" manner. The vehicle then broadcasts the following message to the child via the voice interaction device: "Warning! A child in the back seat has unbuckled their seatbelt. Please check immediately and ensure the child is properly fastened."

[0052] In one possible implementation, speech modulation parameters can be generated based on the first feature information and the second feature information.

[0053] In one possible implementation, speech modulation parameters can be generated based on the first feature information and the third feature information.

[0054] In one possible implementation, speech modulation parameters can be generated based on the first feature information, the second feature information, and the third feature information.

[0055] Based on the above technical solution, the delivered care messages can be more tailored to the needs of the immediate situation and the target audience, while also taking into account the user's response requirements. In a driving scenario, this personalized voice adjustment not only improves the efficiency and accuracy of information transmission but also quickly attracts the driver's attention in emergencies, enabling necessary measures to be taken to ensure driving safety.

[0056] In one possible implementation, the voice broadcasting device is controlled to adjust the tone, speed, and / or timbre of the voice broadcasting based on the voice adjustment parameters, including: inputting the voice adjustment parameters and care information into the text-to-speech (TTS) module; and acquiring the voice audio generated by the TTS module.

[0057] In conjunction with the first aspect, in some implementations of the first aspect, generating a second image based on the first feature information includes: inputting the first feature information into a fusion processing module; obtaining a fusion feature vector output by the fusion processing module; and generating a second image based on the fusion feature vector.

[0058] In some possible implementations, the intervention level of the first state can be calculated based on the voice adjustment parameters. If the intervention level is greater than or equal to the preset level, the tone, speed and / or timbre of the voice broadcast can be adjusted according to the voice adjustment parameters.

[0059] The intervention level can be a level that measures the severity of the first state.

[0060] In conjunction with the first aspect, in some implementations of the first aspect, before obtaining the fused feature vector output by the fusion processing module, the method further includes: inputting second feature information into the fusion processing module; and / or, inputting third feature information into the fusion processing module.

[0061] Among them, the fusion processing module can be a module that abstracts and processes data.

[0062] In one possible implementation, the input to the fusion processing module can be the first feature information, and the fusion feature vector B output by the fusion processing module can be used as a mathematical representation of the first feature information.

[0063] In one possible implementation, the input to the fusion processing module can be first feature information, second feature information, and third feature information. In this case, the fusion processing module can have the ability to fuse multiple data sources, thus it can be a multimodal information fusion module. The fusion processing module can output a fused feature vector C, which can provide a rich information foundation for generating the second image. The second image generated based on the fused feature vector C is a visual representation that integrates multimodal information.

[0064] In one possible implementation, generating a second image based on the fused feature vector includes: inputting the fused feature vector into the state map generation module and obtaining the second image output by the state map generation module.

[0065] The state graph generation module can be an algorithm or a software module. It can process the fused feature vector and use image rendering technology to output a second image, that is, a state graph that reflects the state of the first object.

[0066] In conjunction with the first aspect, in some implementations of the first aspect, after inputting the first feature information into the fusion processing module, the method further includes: obtaining the fused text information output by the fusion processing module; and generating care information based on the fused text information.

[0067] In conjunction with the first aspect, in some implementations of the first aspect, after inputting the first feature information into the fusion processing module, the method further includes: obtaining the voice adjustment parameters output by the fusion processing module; controlling the voice broadcasting device to broadcast care information, including: generating broadcast audio of care information based on the voice adjustment parameters.

[0068] Based on the above technical solution, the fusion processing module can efficiently integrate feature information from different data sources (such as visual, audio, and other sensor data), helping the vehicle understand the state of the first object from multiple dimensions and also contributing to the generation of more accurate second images and care information. Therefore, the fusion processing module can enhance monitoring and feedback capabilities.

[0069] In conjunction with the first aspect, in some implementations of the first aspect, the vehicle status data includes vehicle speed information. Based on the first feature information, care information is generated, including: generating care information based on the first feature information and vehicle speed information; and based on the first feature information, voice adjustment parameters are generated, including: generating voice adjustment parameters based on the first feature information and vehicle speed information.

[0070] In one possible implementation, when the vehicle is stationary, it can alert the driver to information using a normal speaking speed and gentle tone.

[0071] In one possible implementation, when the vehicle is driving at low speed, it can deliver care messages at a slightly faster pace and in a more serious tone.

[0072] In one possible implementation, when the vehicle is driving at high speed, it can broadcast care information through rapid speech and an urgent tone.

[0073] Speed ​​is one of the key factors in measuring driving risk. Based on the above technical solution, driving risk can be measured by combining speed information, thereby providing appropriate reminders and feedback according to the actual driving speed of the vehicle. Different care information and voice adjustment parameters can more accurately match the current driving risk level, taking into account the driver's psychological pressure and the effectiveness of information transmission in different scenarios, thereby improving the driver's experience and emergency response capabilities.

[0074] In conjunction with the first aspect, in some implementations of the first aspect, the vehicle status data includes bumpiness information, and care information is generated based on the first feature information, including: generating care information based on the first feature information and bumpiness information; and voice adjustment parameters are generated based on the first feature information, including: generating voice adjustment parameters based on the first feature information and bumpiness information.

[0075] Based on the above technical solutions, the efficiency of drivers in obtaining important information under bumpy road conditions can be improved, thereby enhancing the transmission of care information instructions.

[0076] In conjunction with the first aspect, in some implementations of the first aspect, the vehicle state data includes noise information, and care information is generated based on the first feature information, including: generating care information based on the first feature information and noise information; and voice adjustment parameters are generated based on the first feature information, including: generating voice adjustment parameters based on the first feature information and noise information.

[0077] Based on the above technical solution, appropriate feedback and adjustments can be provided according to the sound environment inside the vehicle, which helps the driver to clearly obtain key information and take action in a noisy environment, thus providing feedback from the vehicle to the driver.

[0078] In conjunction with the first aspect, in some implementations of the first aspect, the vehicle status data includes temperature information, and care information is generated based on the first feature information, including: generating care information based on the first feature information and temperature information; and voice adjustment parameters are generated based on the first feature information, including: generating voice adjustment parameters based on the first feature information and temperature information.

[0079] Based on the above technical solution, the alert method can be adjusted according to the comfort level of the vehicle interior temperature. When the driver is relaxed and focused, detailed information can be provided; when the driver may be agitated, a brief alert can be given to help the driver quickly understand the situation. This helps to effectively reflect the state of the primary user, thereby improving user satisfaction.

[0080] In conjunction with the first aspect, in some implementations of the first aspect, care information is generated based on the first feature information, including: generating care information based on at least two of vehicle speed information, bumpiness information, noise information, and temperature information and their corresponding weight information, wherein the weight information is dynamically adjusted according to the actual scenario; and voice adjustment parameters are generated based on the first feature information, including: generating voice adjustment parameters based on at least two of vehicle speed information, bumpiness information, noise information, and temperature information and their weight information.

[0081] Based on the above technical solution, information such as vehicle speed, bumpiness, noise, and temperature can be flexibly combined under different environmental conditions to generate care information and voice adjustment parameters. This helps to provide users with real-time feedback and guidance that matches the current driving situation, thereby optimizing the driver's response and decision-making process.

[0082] Secondly, a monitoring device is provided. The device includes: an acquisition unit, configured to: acquire a first image captured by a camera, the first image including a first object; the acquisition unit is further configured to: acquire first feature information based on the first image, the first feature information being used to describe a first state of the first object; a generation unit, configured to: generate a second image based on the first feature information, the second image being used to display the first state; and a control unit, configured to: control a display device to display the second image.

[0083] In conjunction with the second aspect, in some implementations of the second aspect, when the first object is inside the vehicle, the acquisition unit is further configured to: acquire vehicle state data collected by sensors; the acquisition unit is further configured to: acquire second feature information based on the vehicle state data, the second feature information being used to describe the state of the vehicle; the generation unit is further configured to: generate a second image based on the first feature information and the second feature information.

[0084] In conjunction with the second aspect, in some implementations of the second aspect, the acquisition unit is further configured to: acquire audio data collected by the microphone; the acquisition unit is further configured to: acquire third feature information based on the audio data, the third feature information being used to describe the sound emitted by the first object; the generation unit is further configured to: generate a second image based on the first feature information and the third feature information.

[0085] In conjunction with the second aspect, in some implementations of the second aspect, the control unit is also used to: control the display device to display the care interface, the care interface including an animation display area, a text prompt area, a multi-function button area, etc.; the control unit is also used to: control the animation display area to display a second image.

[0086] In conjunction with the second aspect, in some implementations of the second aspect, the generating unit is further configured to: generate care information based on the first feature information, the care information including: the cause of the first state; the control unit is further configured to: control the prompting device to prompt the care information.

[0087] In conjunction with the second aspect, in some implementations of the second aspect, care information also includes: responses to the causes.

[0088] In conjunction with the second aspect, in some implementations of the second aspect, the control unit is also configured to: control the display device to display care information in the text prompt area, and / or control the voice broadcasting device to broadcast care information.

[0089] In conjunction with the second aspect, in some implementations of the second aspect, the generating unit is further configured to: generate voice adjustment parameters based on the first feature information; and the control unit is further configured to: control the voice broadcasting device to adjust the tone, speed and / or timbre of the voice broadcast based on the voice adjustment parameters.

[0090] In conjunction with the second aspect, in some implementations of the second aspect, the device further includes an input unit for: inputting first feature information into the fusion processing module; an acquisition unit for: acquiring the fusion feature vector output by the fusion processing module; and a generation unit for: generating a second image based on the fusion feature vector.

[0091] In conjunction with the second aspect, in some implementations of the second aspect, the input unit is further configured to: input the second feature information into the fusion processing module; and / or, input the third feature information into the fusion processing module.

[0092] In conjunction with the second aspect, in some implementations of the second aspect, the acquisition unit is also used to: acquire the fused text information output by the fusion processing module; the generation unit is also used to: generate care information based on the fused text information.

[0093] In conjunction with the second aspect, in some implementations of the second aspect, the acquisition unit is also used to: acquire the voice adjustment parameters output by the fusion processing module; the generation unit is also used to: generate the broadcast audio of the care information based on the voice adjustment parameters.

[0094] In conjunction with the second aspect, in some implementations of the second aspect, the first state is the negative state of the first object.

[0095] In conjunction with the second aspect, in some implementations of the second aspect, the vehicle status data includes vehicle speed information. The generation unit is further configured to: generate care information based on the first feature information and the vehicle speed information; the generation unit is further configured to: generate voice adjustment parameters based on the first feature information and the vehicle speed information.

[0096] In conjunction with the second aspect, in some implementations of the second aspect, the vehicle state data includes bump information, and the generation unit is further configured to: generate care information based on the first feature information and the bump information; the generation unit is further configured to: generate voice adjustment parameters based on the first feature information and the bump information.

[0097] In conjunction with the second aspect, in some implementations of the second aspect, the vehicle state data includes noise information, and the generation unit is further configured to: generate care information based on the first feature information and the noise information; the generation unit is further configured to: generate voice adjustment parameters based on the first feature information and the noise information.

[0098] In conjunction with the second aspect, in some implementations of the second aspect, the vehicle status data includes temperature information, and the generation unit is further configured to: generate care information based on the first feature information and the temperature information; the generation unit is further configured to: generate voice adjustment parameters based on the first feature information and the temperature information.

[0099] In conjunction with the second aspect, in some implementations of the second aspect, the generation unit is also used to: generate care information based on at least two of the vehicle speed information, bumpiness information, noise information, and temperature information and their corresponding weight information, wherein the weight information is dynamically adjusted according to the actual scenario; the generation unit is also used to: generate voice adjustment parameters based on at least two of the vehicle speed information, bumpiness information, noise information, and temperature information and their weight information.

[0100] Thirdly, a monitoring device is provided, comprising: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, such that the device performs any of the possible methods in the first aspect.

[0101] Fourthly, a vehicle is provided, the vehicle comprising: any one of the possible means of the second or third aspect.

[0102] Fifthly, a computer-readable storage medium is provided that stores instructions which, when executed by a processor, cause the processor to implement any of the possible methods of the first aspect.

[0103] In a sixth aspect, a computer program product is provided, comprising computer program code that, when executed on a computer, causes the computer to implement any of the possible methods of the first aspect.

[0104] In a seventh aspect, a chip is provided, the chip including circuitry for performing any of the possible methods in the first aspect. Attached Figure Description

[0105] Figure 1 This is a functional block diagram of the vehicle 100 provided in the embodiments of this application.

[0106] Figure 2 This is a schematic block diagram of the intelligent driving system 200 provided in the embodiments of this application.

[0107] Figure 3 This is a schematic flowchart of a monitoring method 300 provided in an embodiment of this application.

[0108] Figure 4 This is a top view of the interior of the cabin of a vehicle 100 provided in an embodiment of this application.

[0109] Figure 5 This is a schematic diagram of a first image provided in an embodiment of this application.

[0110] Figure 6 This is a schematic diagram of a graphical user interface (GUI) provided in an embodiment of this application.

[0111] Figure 7 This is a top view of the interior of the cabin of a vehicle 100 provided in an embodiment of this application.

[0112] Figure 8 This is a schematic diagram of a GUI provided in an embodiment of this application.

[0113] Figure 9 This is a schematic diagram of a GUI for a stationary vehicle, provided in an embodiment of this application.

[0114] Figure 10 This is a GUI diagram illustrating a vehicle traveling at low speed, as provided in an embodiment of this application.

[0115] Figure 11 This is a GUI diagram illustrating a vehicle traveling at high speed, as provided in an embodiment of this application.

[0116] Figure 12 This is a schematic flowchart of a monitoring method 1200 provided in an embodiment of this application.

[0117] Figure 13 This is a schematic block diagram of the monitoring device 1300 provided in the embodiments of this application. Detailed Implementation

[0118] The technical solutions in this application will now be described with reference to the accompanying drawings.

[0119] In the description of the embodiments of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in this document is merely a description of the association relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, and B existing alone. "At least one" means one or more. For example, "at least one of A and B" is similar to "A and / or B", describing the association relationship between related objects, indicating that three relationships can exist. For example, at least one of A and B can represent: A existing alone, A and B existing simultaneously, and B existing alone.

[0120] The prefixes such as "first" and "second" used in this application embodiment are merely for distinguishing different descriptive objects and do not limit the position, order, priority, quantity, or content of the described objects. The use of ordinal numbers and other prefixes used to distinguish descriptive objects in this application embodiment does not constitute a limitation on the described objects. The description of the described objects is given in the claims or the context of the embodiments, and should not constitute unnecessary restrictions due to the use of such prefixes. Furthermore, in the description of this embodiment, unless otherwise stated, "multiple" means two or more.

[0121] Figure 1 This is a functional block diagram of a vehicle 100 provided in an embodiment of this application. The vehicle 100 may include a sensing system 110, a computing platform 120, a display device 130, and a microphone 140.

[0122] The perception system 110 may include one or more sensors that sense information about the environment surrounding the vehicle 100.

[0123] Some or all of the functions of vehicle 100 can be controlled by computing platform 120. Computing platform 120 may include one or more processors, such as processors 121 to 12n (n being a positive integer). A processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction read and execute capabilities, such as a central processing unit (CPU), microprocessor, graphics processing unit (GPU) (which can be understood as a type of microprocessor), or digital signal processor (DSP). In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented using an application-specific integrated circuit (ASIC) or a programmable logic device (PLD), such as a field-programmable gate array (FPGA). In reconfigurable hardware circuits, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the process of the processor loading instructions to implement some or all of the functions of the aforementioned units. Furthermore, the processor can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as a neural network processing unit (NPU), tensor processing unit (TPU), deep learning processing unit (DPU), etc. In addition, the computing platform 120 may also include a memory for storing instructions. Some or all of the processors 121 to 12n can call the instructions in the memory to implement the corresponding functions.

[0124] Display devices 130 are mainly divided into two categories: the first is in-vehicle displays; the second is projection displays, such as head-up displays (HUDs). In-vehicle displays are physical displays and an important component of in-vehicle infotainment systems. Multiple displays can be installed in the cabin, such as digital instrument cluster displays, central control screens, displays in front of the front passenger, displays in front of the left and right rear passengers, and even windows can be used as displays. Head-up displays, also known as head-up display systems, are mainly used to display driving information such as speed and navigation on a display device in front of the driver (e.g., the windshield). This reduces driver eye movement time, avoids pupil changes caused by eye movement, and improves driving safety and comfort.

[0125] The above description of the display device 130 uses an in-vehicle display screen and a projection display screen as examples, but the embodiments of this application are not limited thereto. For example, the display device 130 can also be a light display screen or a projection screen.

[0126] Microphone 140 can be a sound input device in vehicle 100, and its main function is to collect sound signals from inside and / or outside the vehicle. Microphone 140 can be used in various scenarios, such as voice recognition or monitoring of the driving environment. Microphone 140 can be a single unit or an array of multiple microphones. The sound signals collected by microphone 140 can be processed by processors 121 to 12n in computing platform 120, such as performing echo cancellation or noise suppression. Microphone 140 can be installed on the top, dashboard, door, or seat of vehicle 100, etc., and this application embodiment does not limit this; its layout and quantity can be designed according to actual needs.

[0127] Optionally, the structure of the vehicle 100 described above is merely illustrative. In actual applications, various components of the vehicle 100 may be added or removed as needed.

[0128] Vehicle 100 may include an intelligent driving system, which may include an advanced driving assistant system (ADAS) and an autonomous driving system (ADS). The intelligent driving system uses various sensors on the vehicle (including but not limited to: lidar, millimeter-wave radar, camera devices, ultrasonic sensors, global positioning system, inertial measurement unit) to acquire information from the vehicle's surroundings, and analyzes and processes the acquired information to achieve functions such as vehicle positioning, route planning, and driver monitoring / alerts, thereby improving the safety, automation, and comfort of driving the vehicle.

[0129] For example, Figure 2 A schematic block diagram of an intelligent driving system 200 provided in an embodiment of this application is shown. The intelligent driving system 200 may include three functional modules: a perception module 210, a decision-making module 220, and a control module 230. The perception module 210 senses the surrounding environment of the vehicle and the status of the occupants through sensors, and outputs corresponding perception data to the decision-making module 220. Based on the information obtained by the perception module 210, the decision-making module 220 determines whether to issue a warning or take emergency measures. The decision-making module 220 may send decision information to the control module 230. After receiving the decision information from the decision-making module 220, the control module 230 may output control signals to control the in-vehicle prompting devices, display devices, and voice broadcasting devices to output corresponding alarm information, or, if necessary, take over control of the vehicle.

[0130] The above-mentioned perception module 210 may include the above-mentioned perception system 110 and microphone 140, and the decision module 220 and control module 230 may be located in the above-mentioned computing platform 120.

[0131] Currently, family driving scenarios are extremely common. For example, parents need to drive with children, who sometimes sit in the back seat. In such situations, parents need to be highly focused while driving, especially on highways, to ensure vehicle stability and safety. However, when children in the back seat cry or become uncomfortable, parents can easily become distracted, even unconsciously turning their heads to check on the children, which increases safety hazards while driving.

[0132] In addition, when the vehicle is traveling at high speed, external wind noise and tire noise may prevent parents from noticing any abnormalities in the back seat, making it difficult to respond to the children's needs in a timely manner and potentially leading to dangerous situations for the children.

[0133] Therefore, how to help parents keep a close eye on the children in the back seat while driving is a key focus for both car manufacturers and consumers.

[0134] One method for implementing rear-seat monitoring can rely on cameras to monitor the safety of rear passengers. Specifically, one or more cameras can be installed inside the vehicle to capture real-time images of rear passengers. These images can be directly displayed on the central control screen in the front of the vehicle, allowing the driver to check the situation in the rear at any time while driving.

[0135] However, in this implementation, the driver needs to spend some time and effort analyzing the images captured by the camera to determine the status of the rear passengers. This process may distract the driver and thus affect driving safety.

[0136] In another implementation of rear-seat monitoring, a preset state map can be used to display the situation of rear passengers. This preset state map can be used to categorize rear passenger behavior; for example, the state map could include behaviors such as: a passenger not wearing a seatbelt, a passenger's hand extending out of the window, or a passenger fainting. Specifically, the vehicle can perform image recognition on images captured by cameras to determine whether the rear passenger's behavior matches the behavior pattern in the vehicle's preset state map. Once a matching behavior is detected, the vehicle can display the corresponding state map to the driver in the front seat to provide timely safety warnings.

[0137] However, this implementation method has a limited capacity to monitor the states of rear passengers. This is because the vehicle's preset state map is fixed and therefore can only cover a limited set of pre-defined behaviors of rear passengers. If the rear passengers' behavior exceeds the preset state, the vehicle may be unable to correctly identify them.

[0138] Therefore, how to more accurately monitor the status of rear passengers while reducing driver distraction, and provide timely and effective feedback to the driver, is an urgent problem to be solved.

[0139] The present application provides a monitoring method, device, and vehicle. The monitoring method provided by the present application can dynamically monitor the diverse states of a first object and generate corresponding second images for feedback based on different states in actual scenarios. This improves the monitoring capability of the first object, as well as the flexibility and accuracy of the feedback. It helps to provide timely and effective feedback on monitoring results, reduces the time when the user's attention is distracted, and allows the user to quickly understand the situation of the first object and take timely countermeasures.

[0140] Figure 3 A schematic flowchart of a monitoring method 300 provided in an embodiment of this application is shown. Figure 3 As shown, method 300 includes:

[0141] S301: Acquire the first image captured by the camera, the first image including the first object.

[0142] The camera can be of different types, such as RGB camera or infrared camera, to capture clear and effective images in different application scenarios.

[0143] The first image can be a frame from a video recorded by the aforementioned camera, or a photograph taken by the aforementioned camera. The first image should include the object of interest, i.e., the first object, so that the state of the first object can be analyzed based on the first image later.

[0144] It should be noted that the embodiments of this application do not limit the first object, which can be a person, an animal or an article.

[0145] For example, the first object can be a group of people of different ages, such as children, youth or the elderly.

[0146] For example, the first object can be a specific person or multiple people. For instance, the first object can be the teachers and students of a school, or the first object can be a specific public figure.

[0147] For example, the first object can be an animal, such as a pet cat or a pet dog.

[0148] For example, the first object can be an item, such as goods transported by a vehicle.

[0149] Optionally, after acquiring the first image captured by the camera module, method 300 further includes: preprocessing the first image, wherein the preprocessing includes at least one of the following: noise reduction, cropping, and resolution adjustment.

[0150] S302: Based on the first image, obtain first feature information, which is used to describe the first state of the first object.

[0151] Optionally, the first state is the negative state of the first object.

[0152] Negative states can typically be undesirable situations, such as unfavorable developmental states, dangerous behaviors, negative emotions, harmful situations, and so on.

[0153] For example, when the first subject is a child, a negative state can refer to an abnormal or undesirable state that a child exhibits in terms of emotions, behavior, health, safety, etc., within a certain period of time.

[0154] In family driving scenarios, parents need to pay special attention to children's negative states to prevent neglecting their needs due to focusing on driving, which could worsen their condition. For example, if a child exhibits emotional fluctuations (such as crying or anxiety), behavioral problems (such as kicking or not wearing a seatbelt), or physical discomfort (such as fever or headache) in the vehicle, parents need to promptly identify these negative states and take appropriate measures.

[0155] For example, in the case where the first subject is an elderly person, the negative state can refer to the physical discomfort experienced by the elderly person over a certain period of time, such as motion sickness or vomiting.

[0156] For example, in the case where the first object is a pet dog, the negative state can be an abnormal state exhibited by the pet dog in terms of behavior and emotions, such as excessive barking, pacing back and forth anxiously, destroying objects, etc.

[0157] For example, in the case where the first object is goods transported by vehicle, a negative condition can refer to any condition that occurs during the transport of the goods that may impair their quality, integrity, or safety. Examples include: goods damaged by impact, goods becoming damp and moldy, and goods spoiling due to excessively high temperatures.

[0158] The first feature information may be textual feature information used to describe the first object. The embodiments of this application do not limit the form of the first feature information.

[0159] For example, the first feature information can be text content expressed in natural language, such as sentences, phrases, or words, which can be used to directly express the scene of the first image or the state of the first object. For example, the first feature information can be a description such as "the child has unbuckled his seatbelt" or "the puppy is playing."

[0160] For example, the first feature information can also be a text feature vector. A text feature vector is a method of converting text content expressed in natural language into a numerical representation. This vector can capture the semantic information implicit in the text content, enabling the similarity between texts and other features to be quantified in a high-dimensional space.

[0161] In some implementations, the first feature information can be obtained based on the first model; that is, the first image is input into the first model, and the first model outputs the first feature information. It should be noted that the embodiments of this application do not limit the form of the first model; the first model can be any model capable of extracting / summarizing text feature information from video or images captured by a camera.

[0162] For example, the first model could be a contrastive learning model. Contrastive learning allows the model to extract meaningful representations from unlabeled data. By leveraging similarity and dissimilarity, contrastive learning enables the model to tightly map similar instances together in the latent space while separating those that are different. Below, using rear-seat child safety monitoring in vehicles as an example, we introduce one method for obtaining first feature information through a contrastive learning model:

[0163] First, a first image containing the child in the back seat of the vehicle is captured by the camera module, and the first image is input into the contrastive learning model A.

[0164] Secondly, the contrastive learning model A extracts the visual features of children in the back seat of the vehicle using a visual feature extractor.

[0165] Next, the contrastive learning model A maps the extracted visual features to a high-dimensional feature space and optimizes the feature representation by comparing positive and negative sample pairs.

[0166] Finally, the contrastive learning model A inputs the optimized feature representation into a text generation module to generate the first feature information describing the state of the child in the back seat of the vehicle, which is "the child is crying loudly in the back seat".

[0167] S303: Generate a second image based on the first feature information. The second image is used to display the first state.

[0168] S304: Control the display device to display the second image.

[0169] It should be noted that, in the embodiments of this application, the first image and the second image are two different images, and the specific differences are as follows:

[0170] The first image, captured by a camera, may contain a wealth of information, reflecting the state of multiple elements or the overall scene within the environment. For example, the first image might show the entire rear seat of a vehicle, including people, animals, objects, and other environmental features. Therefore, the visual data in the first image is more complex.

[0171] The second image is generated based on the first image. Specifically, the second image can be a visual representation of the first feature information extracted from the first image, and its purpose is to highlight the specific information that the user cares about most. In some implementations, multiple frames of the second image can be arranged and played in chronological order to form a dynamic animation. Through this dynamic animation, the relevant status of the first object can be intuitively displayed to the user, such as "child not wearing a seatbelt," "pet dog emotionally unstable," or "damaged goods during transport," reducing interference from other information, improving the user's reaction efficiency, and enabling the user to take appropriate measures in a timely manner.

[0172] As shown above, compared to the first image, the second image simplifies complex visual data and focuses more on the state information of the first object that the user cares about. The generation of the second image provides users with more intuitive and vivid feedback, helping them to identify abnormal states of the first object in a timely manner. Especially in driving scenarios, this can reduce the time drivers spend analyzing complex surveillance videos, improving driving safety, and also enable drivers to take timely countermeasures, thereby effectively preventing accidents.

[0173] The embodiments of this application do not limit the image style of the second image. For example, the image style of the second image can be realistic. Or, for example, the image style of the second image can be cartoonish.

[0174] Below, according to Figures 4-6 An example of generating a second image based on first feature information is given.

[0175] Figure 4 A top view of the interior of a vehicle 100 according to an embodiment of this application is shown. Figure 4 In the scenario shown, the driver is in the driver's seat of vehicle 100, while child A is sitting in the back seat with a seatbelt fastened. Shopping bags 403 and toys 404 are also placed in the back of vehicle 100. A camera 401 is mounted on the back of the rear seat. The gray fan-shaped shaded area represents the field of view 402 of camera 401, within which child A can fall. Camera 401 can capture images containing child A. Vehicle 100 can analyze child A's behavior inside the vehicle using image recognition technology and obtain first feature information describing child A's state.

[0176] For example, camera 401 can capture images such as Figure 5 The first image shown depicts a child, A, pulling on a seatbelt. A shopping bag 403 and a toy 404 are located next to child A's seat. Embodiments of this application can obtain first feature information from the first image based on a first model. This first feature information could be, for example, "a child in the back seat is trying to unbuckle their seatbelt."

[0177] Next, based on this first feature information, the vehicle can generate, for example... Figure 6 The image shown. Figure 6 A schematic diagram of a GUI provided in an embodiment of this application is shown. This interface is a monitoring result feedback interface provided to the driver. The interface displays "Rear Seat Child Monitoring Mode," with multiple control buttons on the left and a second image generated based on first feature information displayed in area 601 on the right. Figure 6 As shown, the second image is a line drawing depicting a child sitting in a seat unfastening their seatbelt: the child is wearing a seatbelt, holding it with both hands, and one end of the seatbelt has been removed from the buckle.

[0178] Users can quickly understand whether there is dangerous behavior by children in the back row by viewing the second image generated in area 601.

[0179] The solution in this application embodiment differs from the solution described above that displays the status of rear passengers through a fixed state diagram. The second image generated in this application embodiment is not a preset image. Therefore, the monitoring solution in this application embodiment has higher flexibility and real-time feedback capability, and can more accurately reflect the behavior and status of the first object, thereby improving the effectiveness and practicality of safety monitoring.

[0180] The monitoring method provided in this application embodiment can not only monitor the state of the first object through a camera, but also provide more comprehensive monitoring and feedback by combining data from other modalities, thereby helping to analyze the in-vehicle environment and identify the behavior of the first object more accurately and improve the accuracy of monitoring.

[0181] This application does not limit the data types of other modalities; for example, it can be vehicle status data, audio data, user interaction data, etc.

[0182] Optionally, if the first object is inside the vehicle, before generating the second image, method 300 further includes: acquiring vehicle state data collected by sensors; acquiring second feature information based on the vehicle state data, the second feature information being used to describe the state of the vehicle; and generating the second image based on the first feature information, including: generating the second image based on the first feature information and the second feature information.

[0183] As can be seen from the above, method 300 can acquire real-time status data provided by vehicle sensors, including but not limited to: vehicle speed acquired by the speed sensor, temperature acquired by the temperature sensor, noise acquired by the microphone 140, and bumpiness acquired by the vibration sensor. This data can indicate the current status of the vehicle, such as whether the vehicle is moving, whether the temperature inside the vehicle is comfortable, whether the noise inside the vehicle is too loud, and whether the vehicle is traveling too bumpily.

[0184] The second feature information can be the vehicle state feature vector.

[0185] Optionally, the second feature information is obtained based on the vehicle status data, including: inputting the vehicle status data into the vehicle status information module and obtaining the vehicle status feature vector output by the vehicle status information module.

[0186] The aforementioned vehicle state feature vector can be a mathematical representation output after the vehicle state data has been processed by the vehicle state information module. The parameters in this mathematical representation can reflect the vehicle's performance, operating status, or in-vehicle environment.

[0187] Below is a specific example of a vehicle state feature vector:

[0188] [v,T,B,N]=[60,90,3,70]

[0189] Where v represents vehicle speed, T represents temperature, B represents bumpiness, and N represents noise level. The meaning of this feature vector is:

[0190] Vehicle speed = 60km / h, meaning the vehicle is currently traveling at a speed of 60 kilometers per hour.

[0191] Temperature = 15℃, meaning the temperature inside the car is 15 degrees Celsius.

[0192] The bumpiness level is 3, meaning the vehicle experiences a level 3 level of bumpiness while driving. It should be noted that in this embodiment, a level 1 bumpiness level represents the smoothest ride, while a level 10 bumpiness level represents the most bumpy ride.

[0193] Noise level = 70dB, meaning the noise level inside the vehicle is 70 decibels.

[0194] In some embodiments, a second image can be generated based on the first feature information and the second feature information, that is, the image data collected by the camera and the vehicle status data collected by the sensor can be combined simultaneously to provide users with more comprehensive monitoring and feedback.

[0195] Optionally, before generating the second image, method 300 further includes: acquiring audio data collected by a microphone; acquiring third feature information based on the audio data, the third feature information being used to describe the sound emitted by the first object and / or the sound in the environment; and generating the second image based on the first feature information, including: generating the second image based on the first feature information and the third feature information.

[0196] As can be seen from the above, method 300 can also acquire audio data captured by the microphone. For example, the audio data may include sounds emitted by the first object, such as crying, laughter, breathing, babbling, and speaking. For example, the audio data may also include ambient sounds inside the vehicle, such as sounds played by electronic devices or conversations between other users who are not the first object. This data can represent the emotions and behaviors of the first object, as well as the activities inside the vehicle.

[0197] The third feature information can be an audio feature vector.

[0198] Optionally, based on the audio data, the third feature information is obtained, including: inputting the audio data into the audio module and obtaining the audio feature vector output by the audio module.

[0199] The aforementioned audio feature vector can be a series of quantized features extracted by the audio module based on the raw audio data acquired by the microphone 140, which can represent the essential attributes of the audio signal.

[0200] For example, an audio feature vector may include at least one of Mel spectrum, MFCCs, pitch, volume, and emotional features.

[0201] Mel spectrum is a commonly used method for representing audio features. It converts audio signals into two-dimensional spectrograms suitable for machine learning models by performing Fourier transform and Mel filtering on the audio signal.

[0202] Among them, MFCCs are features widely used in the fields of audio processing and speech recognition. They extract key information from audio signals by simulating the human ear's perception of sound frequencies and are often used in speech recognition.

[0203] The emotional features can be information extracted from the audio data by the second model that reflects the emotional state of the speaker. It should be noted that the form of the second model is not limited in this application; the second model can be any model capable of extracting emotional states from audio data. For example, the second model can be a deep learning model.

[0204] In addition, pitch can be the fundamental frequency of audio data, and volume can be the loudness of audio data.

[0205] In some embodiments, a second image can be generated based on the first feature information and the third feature information, and the image data collected by the camera and the audio data collected by the microphone 140 can be combined simultaneously to provide the user with more comprehensive monitoring and feedback.

[0206] In some embodiments, a second image may be generated based on the first feature information, the second feature information, and the third feature information.

[0207] Optionally, generating a second image based on the first feature information includes: inputting the first feature information into the fusion processing module; obtaining the fusion feature vector output by the fusion processing module; and generating the second image based on the fusion feature vector.

[0208] Optionally, before obtaining the fused feature vector output by the fusion processing module, the method further includes: inputting second feature information into the fusion processing module; and / or, inputting third feature information into the fusion processing module.

[0209] Among them, the fusion processing module can be a module that abstracts and processes data.

[0210] For example, when the input to the fusion processing module is the first feature information, the fusion feature vector B output by the fusion processing module can be used as a mathematical representation of the first feature information.

[0211] In one implementation, the fusion processing module can be a multi-modal information fusion module, that is, the fusion processing module has the ability to fuse multiple data sources.

[0212] For example, the fusion processing module can integrate the first, second, and third feature information to output a more comprehensive fusion feature vector C. This fusion feature vector C can provide a rich information foundation for generating the second image. The second image generated based on the fusion feature vector C is a visual representation that integrates multimodal information.

[0213] For example, generating a second image based on the fused feature vector includes: inputting the fused feature vector into the state map generation module and obtaining the second image output by the state map generation module.

[0214] The state graph generation module can be an algorithm or a software module. It can process the fused feature vector and use image rendering technology to output a second image, that is, a state graph that reflects the state of the first object.

[0215] Optionally, method 300 further includes: generating care information based on first feature information, the care information including: the cause of the first state; and a control prompting device prompting the care information.

[0216] Optionally, the care information may also include: response measures for the cause.

[0217] Optionally, the method further includes: inputting the second image output by the state diagram generation module and the care information into the UI display module; and controlling the prompting device to prompt the image and text output by the UI display module.

[0218] Among them, care information can be a comprehensive message that includes analysis, interpretation, and reminders about the current state or behavior of the first object.

[0219] The cause of the first state can be internal to the first object, such as physiological needs or emotional changes, or it can be external, such as environmental changes or intervention from other objects.

[0220] For example, in the case where the first object is a child, the child's first state is "crying". The reasons for the child crying may be: the child is physically uncomfortable, the child is choking on food, the ambient temperature is too cold or too hot, the road is too bumpy, etc.

[0221] The response measures are designed to guide users on how to help the first object improve its current situation. For example, in cases where the first object is in a dangerous or emergency situation, the response measures typically require immediate action from the user.

[0222] For example, if a child cries because the interior temperature of the vehicle is too low, the vehicle can prompt the driver to turn up the air conditioning temperature.

[0223] For example, if a child is choking on food, the vehicle can prompt the driver to pull over as soon as possible to check on the child.

[0224] As mentioned above, the technical solution provided in this application embodiment can not only monitor the state of the first object using a camera, but also integrate data from multiple modalities to achieve more comprehensive monitoring and feedback.

[0225] For example, care information can be generated based on the first feature information and the second feature information.

[0226] For example, care information can be generated based on the first feature information and the third feature information.

[0227] For example, care information can be generated based on the first feature information, the second feature information, and the third feature information.

[0228] Optionally, method 300 further includes: controlling the display device to display a care interface, the care interface including an animation display area, a text prompt area, a multi-function button area, etc.; controlling the display device to display a second image, including: controlling the animation display area to display a second image.

[0229] Optionally, the control prompting device prompts care information, including: controlling the display device to display care information in the text prompting area, and / or controlling the voice broadcasting device to broadcast care information.

[0230] Optionally, after inputting the first feature information into the fusion processing module, the method further includes: obtaining the fused text information output by the fusion processing module; and generating care information based on the fused text information.

[0231] Below, in conjunction with Figure 7 and Figure 8 An example of providing a second image and care information is given.

[0232] Figure 7 A top view of the interior of a vehicle 100 according to an embodiment of this application is shown. Figure 7 In the scenario depicted, the driver is in the driver's seat of vehicle 100, while a child is lying on the rear seat. The child's baby bottle has overturned and spilled liquid onto the rear seat, and the baby is crying loudly. The driver can monitor the child's condition and the cause of the situation through the central control screen 701. The content displayed on the central control screen 701 can be as follows: Figure 8 As shown.

[0233] Figure 8 The diagram illustrates a GUI provided by an embodiment of this application. This interface serves as a monitoring result feedback interface provided by the central control screen 701 to the driver. The interface mainly includes a top status information bar, an image display area 801, a care instruction display area 803, and a button area on the left.

[0234] The top status information bar displays relevant information about the vehicle's operation. Specifically, the driving mode is "Child Monitoring Mode", the vehicle speed is "100km / h", the bumpiness level is "Level 5", the temperature is "28℃", and the back noise level is "75db".

[0235] The image display area 801 can display the vehicle's application interface under normal circumstances, and can display, for example, when feedback on the child's status is required. Figure 8 The second image shown depicts a crying baby next to an overturned bottle. In the upper left corner of area 801, there is a back button 802. In response to the user clicking the back button 802, area 801 can return to the previous application interface, such as the application desktop or the navigation interface.

[0236] The care instruction display area 803 can display the care information generated by the vehicle: "Warning! The baby bottle in the back seat has been overturned and milk has spilled. Please pull over to the side of the road as soon as possible to handle the situation!"

[0237] Figure 8 The button area on the left includes a voice communication control 804, a real-time status control 805, a voice broadcast control 806, a sleep-inducing mode control 807, an entertainment mode control 808, a music mode control 809, and a soothing mode control 810.

[0238] For example, when the voice communication control 804 is enabled, the vehicle's voice assistant can communicate with children via voice, such as answering questions asked by children.

[0239] For example, when the real-time status control 805 is enabled, the vehicle can display a second image to the user through area 801.

[0240] For example, when the voice broadcast control 806 is enabled, the vehicle can broadcast the aforementioned care information. For example, when the user does not wish to use the voice broadcast function, the voice broadcast control 806 can be disabled.

[0241] For example, when the sleep-inducing mode control 807 is activated, the vehicle can be adjusted to a suitable sleep environment, including playing soft music, white noise, or reading bedtime stories.

[0242] For example, when the entertainment mode control 808 is enabled, the vehicle can play cartoons or simple interactive games through the display screen on the rear seat.

[0243] For example, when the music mode control 809 is enabled, the vehicle can play appropriate music by analyzing the child's age and mood.

[0244] For example, when the soothing mode control 810 is enabled, the vehicle can provide corresponding soothing measures according to the child's state. For instance, if the child is crying due to fatigue from the journey, the vehicle can play specially designed soothing sounds, such as heartbeat sounds, lullabies, or comfort the child with a gentle voice, such as saying gently, "Don't worry, we'll be at our destination soon."

[0245] Optionally, method 300 further includes: generating speech adjustment parameters based on the first feature information; and controlling the speech broadcasting device to adjust the tone, speed and / or timbre of the speech broadcast based on the speech adjustment parameters.

[0246] Optionally, after inputting the first feature information into the fusion processing module, the method further includes: obtaining the voice adjustment parameters output by the fusion processing module; controlling the voice broadcasting device to broadcast care information, including: generating broadcast audio of care information according to the voice adjustment parameters.

[0247] In one implementation, speech modulation parameters can be generated based on the first feature information and the second feature information.

[0248] In another implementation, speech modulation parameters can be generated based on the first feature information and the third feature information.

[0249] In another implementation, speech modulation parameters can be generated based on the first feature information, the second feature information, and the third feature information.

[0250] Among them, voice adjustment parameters can be a series of variables or parameters used to adjust and control voice broadcasting.

[0251] Table 1

[0252] Voice adjustment parameters tone speech rate tone selection 0.0-0.2 gentle, soothing slow Soft, low 0.2-0.4 Stable, concerned medium Clear, soothing 0.4-0.6 Serious, warning Faster High-pitched and bright 0.6-0.8 Determined, Urgent quick Sharp, bright 0.8-1.0 tense, hurried Very fast Sharp, clear

[0253] Table 1 illustrates the relationship between voice adjustment parameters and the selection of tone, speed, and timbre in voice broadcasting. This is used to deliver personalized care information to the driver or the primary recipient via a voice broadcasting device in different scenarios. In other words, the vehicle can adjust the tone, speed, and timbre of the broadcast voice according to the preset parameters in Table 1 to adapt to different scenario needs, thereby achieving accurate, effective, and emotional information delivery.

[0254] As shown in Table 1, the range of voice adjustment parameters is from 0.0 to 1.0, and different values ​​correspond to different monitoring scenarios and feedback requirements.

[0255] Table 1 shows that different tone of voice can be selected to express care or warning, depending on the different voice adjustment parameters; different speaking speeds can also be selected to express different levels of urgency. For example, "relatively fast," "fast," and "very fast" can all be used to convey an urgent situation.

[0256] Among them, the timbre selection further refines the emotional expression of the voice. The following is an explanation of the different timbre selections:

[0257] 1. A soft, deep tone. Suitable for relaxed settings, providing warm and comforting messages to children to express care or reassurance.

[0258] 2. Clear and soothing tone. Suitable for children in a calm driving condition or when children are experiencing mild discomfort.

[0259] 3. High-pitched, bright tone. Suitable for broadcasting warning messages that need to attract the driver's attention.

[0260] 4. Sharp, bright tone. Suitable for alerting drivers when a quick response or emergency measures are needed.

[0261] 5. Sharp, clear tone. Suitable for scenarios where drivers need to quickly understand and take immediate action.

[0262] In Table 1, the numerical range of each set of voice adjustment parameters is applicable to specific scenarios. Taking child monitoring in the back seat of a vehicle as an example, the following implementation example is given:

[0263] For example, if the vehicle detects that a child in the back seat is becoming restless due to a long ride, the voice adjustment parameter is set to 0.1, within the range of 0.0-0.2. Therefore, the vehicle can adjust the voice broadcast device, including adjusting the tone to be "gentle and soothing," the speech rate to "slow," and the timbre to be "soft and low." The vehicle then broadcasts to the child via the voice interaction device: "Honey, don't worry, we're almost there!"

[0264] For example, if the vehicle detects mild discomfort in a child in the back seat, such as sneezing or a slight cough, the voice adjustment parameter is 0.3, indicating that although the situation is not serious, the driver still needs to pay attention. Therefore, the vehicle can adjust the voice broadcast device, including adjusting the broadcast tone to "smooth and concerned," the speech rate to "medium," and the timbre to "clear and soothing." The vehicle broadcasts to the driver: "The child in the back seat seems to be feeling unwell, possibly with cold symptoms. Please pay attention to the child's health."

[0265] For example, if the vehicle detects that a child in the back seat has unbuckled their seatbelt, the voice adjustment parameter is set to 0.5, indicating a potential safety hazard to the driver. Therefore, the vehicle can adjust the voice broadcast device, including: adjusting the tone to "serious and warning," the speech rate to "faster," and the timbre to "high-pitched and bright." The vehicle then broadcasts the following message to the child via the voice interaction device: "Warning! A child in the back seat has unbuckled their seatbelt. Please check immediately and ensure the child is properly fastened."

[0266] For example, if a vehicle detects that a child in the back seat suddenly leans out of the window while the vehicle is in motion, a highly risky action, the voice adjustment parameter would be 0.8. Therefore, the vehicle can adjust the voice broadcast system, including adjusting the tone to "firm and urgent," the speech rate to "fast," and the timbre to "sharp and bright." The vehicle would then broadcast to the driver: "Emergency! A child in the back seat is leaning out of the window. Please immediately return the child to their seat and fasten their seatbelt!"

[0267] For example, if the vehicle detects that a child in the back seat has suddenly lost consciousness or stopped breathing, the vehicle can adjust the tone of the announcement to "tense and urgent", the speed to "very fast", and the timbre to "sharp and clear" according to the voice adjustment parameter 1.0, and announce to the driver: "Very urgent! The child in the back seat has lost consciousness. Please call emergency rescue and perform first aid immediately!"

[0268] It should be noted that the relationships between the above voice adjustment parameters and tone, speech rate, and timbre are merely illustrative and can be flexibly adjusted according to specific needs in actual applications. Furthermore, this application does not limit the tone, speech rate, and timbre.

[0269] Optionally, based on the voice adjustment parameters, the voice broadcasting device is controlled to adjust the tone, speed, and / or timbre of the voice broadcast, including: inputting the voice adjustment parameters and care information into the TTS module; and acquiring the voice audio generated by the TTS module.

[0270] In some embodiments, the intervention level of the first state can be calculated based on the voice adjustment parameters. If the intervention level is greater than or equal to a preset level, the tone, speed and / or timbre of the voice broadcast can be adjusted according to the voice adjustment parameters.

[0271] The intervention level can be a level that measures the severity of the first state. This application does not limit the method for calculating the intervention level; for example, the intervention level can be determined based on the values ​​of the voice adjustment parameters. For instance, intervention levels can be divided into three levels: Level 1, Level 2, and Level 3. Level 1 represents mild intervention, with corresponding voice adjustment parameters between 0.0 and 0.2; Level 2 represents moderate intervention, with corresponding voice adjustment parameters between 0.2 and 0.6; and Level 3 represents emergency intervention, with corresponding voice adjustment parameters between 0.6 and 1.0. If the preset intervention level is Level 1, then as long as the intervention level is above Level 1, the tone, speed, and / or timbre of the voice broadcast can be adjusted according to the voice adjustment parameters.

[0272] Optionally, the vehicle status data includes vehicle speed information. Based on the first feature information, care information is generated, including: generating care information based on the first feature information and vehicle speed information; and voice adjustment parameters are generated based on the first feature information, including: generating voice adjustment parameters based on the first feature information and vehicle speed information.

[0273] For example, if a child in the back seat of a vehicle chokes on food, the vehicle can recognize the situation and promptly notify the driver. During the notification process, the content of the care message can be adjusted according to the vehicle speed, or the tone, speed, and / or timbre of the voice broadcast can be adjusted.

[0274] When the vehicle is stationary, the voice prompt system can alert the driver in a normal speaking speed and gentle tone. Simultaneously, the vehicle's central control screen can display instructions for the Heimlich maneuver.

[0275] When the vehicle is traveling at low speeds, such as 20 km / h, the voice broadcast system can deliver caring messages at a slightly faster pace and in a more serious tone to attract the driver's attention without causing undue stress. Simultaneously, the vehicle's central control screen can prompt the driver to gradually slow down and find a suitable parking spot.

[0276] When the vehicle is traveling at high speed, such as 160 km / h, the voice broadcast system can deliver care information in a rapid and urgent tone, accompanied by warning sounds, to maximize the driver's awareness of the seriousness of the situation. The central control screen displays an emergency animation prompt, accompanied by flashing text messages, instructing the driver to immediately slow down and find the nearest safe place to stop, while remaining aware of safety.

[0277] Figures 9-11 The diagram shows GUI illustrations at different vehicle speeds. If a child chokes, the vehicle can alert the user via the central control screen 701.

[0278] like Figure 9 As shown, when the instrument panel 901 in the cockpit displays a vehicle speed of 0 km / h, it means the vehicle is stationary. The vehicle control display device displays an animation of a child choking on food and a Heimlich maneuver control 902 in the image display area 801. The user can click the Heimlich maneuver control 902 to view methods for dealing with a choking child. Simultaneously, the vehicle control display device can display the care instruction display area 803 with the care instruction message: "Friendly reminder! Your child seems to be choking. You can view solutions here," and control the voice broadcast device to broadcast the corresponding audio content, with a relatively gentle tone.

[0279] like Figure 10 As shown, when the vehicle speed on instrument panel 901 is 20 km / h, the vehicle is in a low-speed driving state. Image display area 801 can display an animation of a child choking. Care instruction display area 803 displays the care instruction: "Attention! Attention! A child is choking. Please find a place nearby to pull over and check!" and plays the corresponding voice message to the driver. The voice message is delivered quickly and in a serious tone.

[0280] like Figure 11 As shown, when the vehicle speed displayed on the instrument panel 901 is 160 km / h, the vehicle is in a high-speed driving state. The image display area 801 can display an animation of a child choking and a warning icon to attract the driver's attention. The care instruction display area 803 displays the care instruction: "Warning! Warning! The child is choking! The child is choking! Please slow down and stop!" and plays the corresponding voice message to the driver. The voice message is spoken quickly, urgently, and loudly, accompanied by warning sound effects.

[0281] Speed ​​is one of the key factors in measuring driving risk. By combining speed information to generate care information and voice adjustment parameters, appropriate reminders and feedback can be provided according to the actual driving speed of the vehicle. Different care information and voice adjustment parameters can more accurately match the current driving risk level, taking into account the driver's psychological pressure and the effectiveness of information transmission in different scenarios, thereby improving the driver's experience and emergency response capabilities.

[0282] Optionally, the vehicle status data includes bumpiness information. Based on the first feature information, care information is generated, including: generating care information based on the first feature information and bumpiness information; and voice adjustment parameters are generated based on the first feature information, including: generating voice adjustment parameters based on the first feature information and bumpiness information.

[0283] For example, a child in the back seat of a vehicle starts crying while the vehicle is in motion. If the vehicle is traveling on a smooth road with low bumps, the voice announcement system can alert the driver to the child's crying in a normal speaking speed and gentle tone. Simultaneously, the central control screen can display an animation of the child crying along with text messages of concern.

[0284] When the vehicle is traveling on a very bumpy road with a high degree of vibration, the text on the central control screen containing care information can be enlarged to help prevent the driver from not being able to see small text in a bumpy environment.

[0285] By combining bumpiness information to generate care messages and voice adjustment parameters, the efficiency of drivers obtaining important information under bumpy road conditions can be improved, thereby enhancing the transmission of care messages.

[0286] Optionally, the vehicle status data includes noise information. Based on the first feature information, care information is generated, including: generating noise information based on the first feature information and vehicle speed information; and generating voice adjustment parameters based on the first feature information, including: generating voice adjustment parameters based on the first feature information and noise information.

[0287] For example, a child in the back seat of a vehicle sticks their hand out of the window while the vehicle is in motion. In environments with low noise levels inside the vehicle, a voice alert system can use a normal volume and pitch to alert the driver to the dangerous behavior.

[0288] In noisy environments, the voice broadcast system can increase the volume to ensure that the information can be heard by the driver.

[0289] By combining noise information to generate care information and voice adjustment parameters, appropriate feedback and adjustments can be provided based on the in-vehicle sound environment, helping drivers to clearly obtain key information and take action in noisy environments.

[0290] Optionally, the vehicle status data includes temperature information. Based on the first feature information, care information is generated, including: generating care information based on the first feature information and temperature information; and voice adjustment parameters are generated based on the first feature information, including: generating voice adjustment parameters based on the first feature information and temperature information.

[0291] For example, two children are engaging in dangerous roughhousing in the back seat of a vehicle. If the interior temperature is comfortable, the voice announcement system can alert the driver at a slightly faster pace and describe the cause and effect of the roughhousing in detail. If the interior temperature is high, and considering the driver's potentially irritable mood, the voice announcement system can briefly explain the situation in the back seat to the driver at a calmer pace.

[0292] Temperature has a significant impact on people's mood. By combining temperature information with voice adjustment parameters to generate caring messages, the way reminders are given can be adjusted according to the comfort level of the vehicle's interior temperature. Detailed information can be provided when the driver is relaxed and focused, while brief reminders can be given when the driver might be irritable, helping them quickly understand the situation. This facilitates effective feedback on the state of the primary user, thereby improving user satisfaction.

[0293] Optionally, care information is generated based on the first feature information, including: generating care information based on at least two of vehicle speed information, bumpiness information, noise information, and temperature information and their corresponding weight information, wherein the weight information is dynamically adjusted according to the actual scenario; and voice adjustment parameters are generated based on the first feature information, including: generating voice adjustment parameters based on at least two of vehicle speed information, bumpiness information, noise information, and temperature information and their weight information.

[0294] The weighting information can be dynamically adjusted according to the actual scenario, so that the generated care information and voice adjustment parameters can adapt to the current driving environment.

[0295] For example, when a vehicle is traveling at a high speed, the ride is bumpy, the noise level inside the vehicle is high, and the interior temperature is comfortable. Therefore, in this case, the weights corresponding to different vehicle state data could be: speed 0.4; bumpiness 0.25; noise 0.25; temperature 0.1.

[0296] Among these factors, vehicle speed carries the highest weight because drivers should focus more on the road when driving at high speeds, so care messages need to be more concise and voice announcements need to be more urgent.

[0297] Secondly, bumpiness and noise are each weighted at 0.25. This is because in bumpy conditions, the driver may have difficulty seeing the text on the screen, so the information needs to be appropriately enlarged and highlighted. At the same time, in noisy conditions, the vehicle needs to increase the volume of the announcements to help the driver hear them.

[0298] Finally, temperature has the lowest weighting because the interior temperature is comfortable and does not have a negative impact on the driver, so there is no need to consider adjusting the care information and voice control parameters based on the temperature.

[0299] Based on the above scheme, at least two of the vehicle's speed, bumpiness, noise, and temperature information can be combined to generate care information and voice adjustment parameters, which helps to provide users with real-time feedback and guidance that matches the current driving conditions, thereby optimizing the driver's response and decision-making process.

[0300] Figure 12A schematic flowchart of a monitoring method 1200 provided in an embodiment of this application is shown. The method 1200 will now be described using the monitoring of children in the back seat of a vehicle as an example.

[0301] like Figure 12 As shown, the multi-mode information fusion module can receive information output from the camera monitoring module, the audio module, and the vehicle status monitoring module, respectively, where the information output from the audio module and the vehicle status monitoring module is optional.

[0302] Specifically, the camera monitoring module can acquire raw image / video data and process it. The processed image / video data can be input into a contrastive learning model, which can perform advanced semantic analysis on the input data to output first feature information. For example, the first feature information can be a text feature vector, which can be used to describe the child's state in the image / video data.

[0303] The vehicle status monitoring module can acquire real-time vehicle status information such as temperature, bumpiness, back noise, and speed, and output secondary feature information. For example, the secondary feature information can be a vehicle status feature vector.

[0304] The audio module can acquire audio data such as the voice, breathing sounds, crying sounds, and coughing sounds of children in the back seat of the vehicle, and output third feature information. For example, the third feature information can be an audio feature vector.

[0305] The explanations of text feature vectors, vehicle state feature vectors, and audio feature vectors have been elaborated in the previous text, so they will not be repeated here.

[0306] The multimodal information fusion module can fuse the above-mentioned text feature vectors, vehicle status feature vectors, and audio feature vectors to output fused feature vectors, care information, and voice adjustment parameters.

[0307] The fused feature vector can be input into the state graph generation module, which can output the state graph of the rear-seat children, i.e., a real-time animation describing the state of the rear-seat children. The state graph can be input into the UI dynamic display module, which can output images / text for display. The vehicle can control the display device to display images / text, which helps the driver quickly view and understand the children's state.

[0308] Among these features, the care information and voice adjustment parameters can be used to broadcast care commands via voice. Specifically, the intervention level can be calculated, and it can be determined whether the intervention level is greater than a preset level.

[0309] If the intervention level is lower than the preset level, the first TTS module can acquire care information and generate standardized care instructions. The first TTS module can output a standard care instruction broadcast information stream, which the vehicle then uses to broadcast the care instructions via voice.

[0310] If the intervention level is higher than the preset level, the second TTS module can acquire care information and voice adjustment parameters, render standardized care commands, and generate personalized care commands. The second TTS module can output a personalized care command broadcast information stream, which the vehicle then uses to broadcast the personalized care command voice.

[0311] It's important to note that standardized care instructions can be care messages generated using a uniform tone, speed, and timbre, without considering specific contexts or individual differences. These instructions are typically preset and universal, applicable to most routine situations, and designed to provide basic care and prompts. Personalized care instructions, on the other hand, refer to targeted care messages generated based on specific contexts and individual needs, by adjusting parameters such as tone, speed, and timbre. These instructions are better adapted to different scenarios and individual differences, providing more precise and emotionally resonant information delivery, thereby enhancing the effectiveness of care and effectively attracting the driver's attention.

[0312] It should be noted that the embodiments of this application do not limit the selection of multimodal data, that is, they are not limited to data captured by cameras, vehicle status data, and audio data. The monitoring method provided in the embodiments of this application can integrate various types of data, including but not limited to: biometric data, environmental sensor data, etc. In other words, the embodiments of this application are not limited to the aforementioned first feature information, second feature information, and third feature information; multiple feature information can be flexibly integrated and utilized according to actual needs and application scenarios to achieve more accurate and comprehensive monitoring results.

[0313] Figure 13 A schematic block diagram of a monitoring device 1300 provided in an embodiment of this application is shown. The device includes: an acquisition unit 1310, configured to: acquire a first image captured by a camera, the first image including a first object; the acquisition unit 1310 is further configured to: acquire first feature information based on the first image, the first feature information describing a first state of the first object; a generation unit 1320, configured to: generate a second image based on the first feature information, the second image used to display the first state; and a control unit 1330, configured to: control a display device to display the second image.

[0314] Optionally, if the first object is inside the vehicle, the acquisition unit 1310 is further configured to: acquire vehicle status data collected by the sensor; the acquisition unit 1310 is further configured to: acquire second feature information based on the vehicle status data, the second feature information being used to describe the vehicle status; the generation unit 1320 is further configured to: generate a second image based on the first feature information and the second feature information.

[0315] Optionally, the acquisition unit 1310 is further configured to: acquire audio data collected by the microphone; the acquisition unit 1310 is further configured to: acquire third feature information based on the audio data, the third feature information being used to describe the sound emitted by the first object; the generation unit 1320 is further configured to: generate a second image based on the first feature information and the third feature information.

[0316] Optionally, the control unit 1330 is also used to: control the display device to display a care interface, the care interface including an animation display area, a text prompt area, a multi-function button area, etc.; the control unit 1330 is also used to: control the animation display area to display a second image.

[0317] Optionally, the generating unit 1320 is further configured to: generate care information based on the first feature information, the care information including: the cause of the first state; the control unit 1330 is further configured to: control the prompting device to prompt the care information.

[0318] Optionally, the care information may also include: response measures for the cause.

[0319] Optionally, the control unit 1330 is also configured to: control the display device to display care information in the text prompt area, and / or control the voice broadcast device to broadcast care information.

[0320] Optionally, the generation unit 1320 is further configured to: generate voice adjustment parameters based on the first feature information; the control unit 1330 is further configured to: control the voice broadcasting device to adjust the tone, speed and / or timbre of the voice broadcast based on the voice adjustment parameters.

[0321] Optionally, the device further includes an input unit 1340, configured to: input the first feature information into the fusion processing module; an acquisition unit 1310, configured to: acquire the fusion feature vector output by the fusion processing module; and a generation unit 1320, configured to: generate a second image based on the fusion feature vector.

[0322] Optionally, the input unit 1340 is further configured to: input the second feature information into the fusion processing module; and / or, input the third feature information into the fusion processing module.

[0323] Optionally, the acquisition unit 1310 is further configured to: acquire the fused text information output by the fusion processing module; the generation unit 1320 is further configured to: generate care information based on the fused text information.

[0324] Optionally, the acquisition unit 1310 is further configured to: acquire the voice adjustment parameters output by the fusion processing module; the generation unit 1320 is further configured to: generate the broadcast audio of the care information based on the voice adjustment parameters.

[0325] Optionally, the first state is the negative state of the first object.

[0326] Optionally, the vehicle status data includes vehicle speed information. The generation unit 1320 is further configured to: generate care information based on the first feature information and the vehicle speed information; the generation unit 1320 is further configured to: generate voice adjustment parameters based on the first feature information and the vehicle speed information.

[0327] Optionally, the vehicle status data includes bump information. The generation unit 1320 is further configured to: generate care information based on the first feature information and the bump information; the generation unit 1320 is further configured to: generate voice adjustment parameters based on the first feature information and the bump information.

[0328] Optionally, the vehicle status data includes noise information, and the generation unit 1320 is further configured to: generate care information based on the first feature information and the noise information; the generation unit 1320 is further configured to: generate voice adjustment parameters based on the first feature information and the noise information.

[0329] Optionally, the vehicle status data includes temperature information. The generation unit 1320 is further configured to: generate care information based on the first feature information and the temperature information; the generation unit 1320 is further configured to: generate voice adjustment parameters based on the first feature information and the temperature information.

[0330] Optionally, the generation unit 1320 is also used to: generate care information based on at least two of the vehicle speed information, bump information, noise information, and temperature information and their corresponding weight information, wherein the weight information is dynamically adjusted according to the actual scenario; the generation unit 1320 is also used to: generate voice adjustment parameters based on at least two of the vehicle speed information, bump information, noise information, and temperature information and their weight information.

[0331] It should be understood that the division of units in the above device is only a logical functional division. In actual implementation, they can be fully or partially integrated into a single physical entity, or they can be physically separated. Furthermore, the units in the device can be implemented by a processor calling software; for example, the device includes a processor connected to memory, which stores instructions. The processor calls the instructions stored in memory to implement any of the above methods or to implement the functions of each unit in the device. The processor can be, for example, a general-purpose processor, such as a CPU or microprocessor, and the memory can be internal or external to the device. Alternatively, the units in the device can be implemented as hardware circuits. The functions of some or all units can be implemented through the design of the hardware circuits, which can be understood as one or more processors. For example, in one implementation, the hardware circuit is an ASIC, and the functions of some or all units are implemented through the design of the logical relationships between the components within the circuit. In another implementation, the hardware circuit can be implemented using a PLD, such as an FPGA, which can include a large number of logic gates. The connection relationships between the logic gates are configured through configuration files, thereby implementing the functions of some or all units. All units of the above devices can be implemented entirely through processor calling software, or entirely through hardware circuits, or partially through processor calling software with the remaining parts implemented through hardware circuits.

[0332] In this application embodiment, a processor is a circuit with signal processing capabilities. In one implementation, the processor can be a circuit with instruction reading and execution capabilities, such as a CPU, microprocessor, GPU, or DSP. In another implementation, the processor can implement certain functions through the logical relationships of hardware circuits. These logical relationships are fixed or reconfigurable. For example, the processor may be a hardware circuit implemented as an ASIC or PLD, such as an FPGA. In a reconfigurable hardware circuit, the process of the processor loading a configuration document and configuring the hardware circuit can be understood as the processor loading instructions to implement the functions of some or all of the above units. Furthermore, it can also be a hardware circuit designed for artificial intelligence, which can be understood as an ASIC, such as an NPU, TPU, or DPU.

[0333] As can be seen, each unit in the above device can be one or more processors (or processing circuits) configured to implement the above methods, such as: CPU, GPU, NPU, TPU, DPU, microprocessor, DSP, ASIC, FPGA, or a combination of at least two of these processor forms.

[0334] Furthermore, the units in the above devices can be integrated in whole or in part, or they can be implemented independently. In one implementation, these units are integrated together as a System-on-a-Chip (SoC). The SoC may include at least one processor for implementing any of the above methods or implementing the functions of the units in the device. The at least one processor may be of different types, such as CPU and FPGA, CPU and AI processor, CPU and GPU, etc.

[0335] This application also provides a monitoring device, which includes: a memory for storing a computer program; and a processor for executing the computer program stored in the memory, so that the device performs the methods or steps described in the above embodiments.

[0336] Optionally, if the monitoring device is located in a vehicle, the processor may be... Figure 1 The processors shown are 121-12n.

[0337] This application also provides a vehicle that may include the monitoring device 1300 described above.

[0338] This application also provides a computer program product, which includes computer program code that, when run on a computer, causes the computer to perform the methods described in the above embodiments.

[0339] This application also provides a chip, which includes a circuit for performing the methods described in the above embodiments.

[0340] In implementation, each step of the above method can be completed by integrated logic circuits in the processor's hardware or by instructions in software. The method disclosed in the embodiments of this application can be directly implemented by a hardware processor, or by a combination of hardware and software modules within the processor. The software modules can reside in random access memory, flash memory, read-only memory, programmable read-only memory, power-on erasable programmable memory, registers, or other mature storage media in the art. This storage medium is located in memory, and the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, detailed descriptions are omitted here.

[0341] It should be understood that in the embodiments of this application, the memory may include read-only memory and random access memory, and provides instructions and data to the processor.

[0342] It should also be understood that, in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0343] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0344] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0345] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0346] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0347] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0348] If the aforementioned functions are implemented as software functional units and sold or used as independent products, they can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or a portion of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0349] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the technical scope disclosed in this application should be covered. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A monitoring method, characterized in that, include: Acquire a first image captured by a camera, the first image including a first object; Based on the first image, first feature information is obtained, and the first feature information is used to describe the first state of the first object; Based on the first feature information, a second image is generated, and the second image is used to display the first state; The control display device displays the second image.

2. The method according to claim 1, characterized in that, If the first object is inside a vehicle, before generating the second image, the method further includes: Acquire vehicle status data collected by sensors; Based on the vehicle status data, second feature information is obtained, which is used to describe the status of the vehicle; The step of generating the second image based on the first feature information includes: The second image is generated based on the first feature information and the second feature information.

3. The method according to claim 1 or 2, characterized in that, Before generating the second image, the method further includes: Acquire audio data captured by the microphone; Based on the audio data, a third feature information is obtained, which is used to describe the sound emitted by the first object and / or the sound in the environment; The step of generating the second image based on the first feature information includes: The second image is generated based on the first feature information and the third feature information.

4. The method according to any one of claims 1-3, characterized in that, The method further includes: The display device is controlled to display a care interface, which includes an animation display area, a text prompt area, a multi-function button area, etc. The control display device displays the second image, including: controlling the animation display area to display the second image.

5. The method according to any one of claims 1-4, characterized in that, The method further includes: Based on the first feature information, care information is generated, and the care information includes: the reason for the first state; The control prompt device displays the care information.

6. The method according to claim 5, characterized in that, The care information also includes: response measures for the reasons mentioned above.

7. The method according to claim 5 or 6, characterized in that, The control prompting device displays the care information, including: Control the display device to display the care information in the text prompt area, and / or, Control the voice broadcasting device to broadcast care information.

8. The method according to claim 7, characterized in that, The method further includes: Based on the first feature information, voice adjustment parameters are generated; Based on the voice adjustment parameters, the voice broadcasting device is controlled to adjust the tone, speed, and / or timbre of the voice broadcast.

9. The method according to claim 8, characterized in that, The step of generating the second image based on the first feature information includes: The first feature information is input into the fusion processing module; Obtain the fusion feature vector output by the fusion processing module; The second image is generated based on the fused feature vector.

10. The method according to claim 9, characterized in that, Before obtaining the fused feature vector output by the fusion processing module, the method further includes: Input the second feature information into the fusion processing module; and / or, The third feature information is input into the fusion processing module.

11. The method according to claim 9 or 10, characterized in that, After inputting the first feature information into the fusion processing module, the method further includes: Obtain the fused text information output by the fusion processing module; The care information is generated based on the fused text information.

12. The method according to claim 11, characterized in that, After inputting the first feature information into the fusion processing module, the method further includes: Obtain the voice adjustment parameters output by the fusion processing module; The control voice broadcasting device broadcasts care information, including: Based on the voice adjustment parameters, the audio for broadcasting the care information is generated.

13. The method according to any one of claims 1-12, characterized in that, The first state is the negative state of the first object.

14. The method according to any one of claims 8-13, characterized in that, The vehicle status data includes vehicle speed information. The step of generating care information based on the first feature information includes: generating the care information based on the first feature information and the vehicle speed information; The step of generating voice adjustment parameters based on the first feature information includes: generating the voice adjustment parameters based on the first feature information and the vehicle speed information.

15. The method according to any one of claims 8-14, characterized in that, The vehicle status data includes bumpiness information. The step of generating care information based on the first feature information includes: generating the care information based on the first feature information and the bumpiness information; The step of generating voice adjustment parameters based on the first feature information includes: generating the voice adjustment parameters based on the first feature information and the turbulence information.

16. The method according to any one of claims 8-15, characterized in that, The vehicle status data includes noise information. The step of generating care information based on the first feature information includes: generating the care information based on the first feature information and the noise information; The step of generating voice adjustment parameters based on the first feature information includes: generating the voice adjustment parameters based on the first feature information and the noise information.

17. The method according to any one of claims 8-16, characterized in that, The vehicle status data includes temperature information. The step of generating care information based on the first feature information includes: generating the care information based on the first feature information and the temperature information; The step of generating voice adjustment parameters based on the first feature information includes: generating the voice adjustment parameters based on the first feature information and the temperature information.

18. The method according to claim 17, characterized in that, The step of generating care information based on the first feature information includes: generating care information based on at least two of the vehicle speed information, the bumpiness information, the noise information, and the temperature information and their corresponding weight information, wherein the weight information is dynamically adjusted according to the actual scenario; The step of generating voice adjustment parameters based on the first feature information includes: generating the voice adjustment parameters based on at least two of the vehicle speed information, the bumpiness information, the noise information, the temperature information, and the weight information.

19. A monitoring device, characterized in that, include: The acquisition unit is configured to: acquire a first image captured by a camera, wherein the first image includes a first object; The acquisition unit is further configured to: acquire first feature information based on the first image, wherein the first feature information is used to describe the first state of the first object; The generation unit is configured to: generate a second image based on the first feature information, wherein the second image is used to display the first state; The control unit is used to control the display device to display the second image.

20. The apparatus according to claim 19, characterized in that, If the first object is inside the vehicle, The acquisition unit is further configured to: acquire vehicle status data collected by the sensors; The acquisition unit is further configured to: acquire second feature information based on the vehicle status data, wherein the second feature information is used to describe the status of the vehicle; The generation unit is further configured to: generate the second image based on the first feature information and the second feature information.

21. The apparatus according to claim 19 or 20, characterized in that, The acquisition unit is further configured to: acquire audio data collected by the microphone; The acquisition unit is further configured to: acquire third feature information based on the audio data, wherein the third feature information is used to describe the sound emitted by the first object; The generation unit is further configured to: generate the second image based on the first feature information and the third feature information.

22. The apparatus according to any one of claims 19-21, characterized in that, The control unit is further configured to: control the display device to display a care interface, the care interface including an animation display area, a text prompt area, a multi-function button area, etc.; The control unit is further configured to: control the animation display area to display the second image.

23. The apparatus according to any one of claims 19-22, characterized in that, The generating unit is further configured to: generate care information based on the first feature information, wherein the care information includes: the cause of the first state; The control unit is also used to: control the prompting device to display the care information.

24. The apparatus according to claim 23, characterized in that, The care information also includes: response measures for the reasons mentioned above.

25. The apparatus according to claim 23 or 24, characterized in that, The control unit is further configured to: control the display device to display the care information in the text prompt area, and / or, Control the voice broadcasting device to broadcast care information.

26. The apparatus according to claim 25, characterized in that, The generation unit is further configured to: generate voice adjustment parameters based on the first feature information; The control unit is further configured to: control the voice broadcasting device to adjust the tone, speed and / or timbre of the voice broadcast according to the voice adjustment parameters.

27. The apparatus according to claim 26, characterized in that, The device further includes an input unit for: inputting the first feature information into the fusion processing module; The acquisition unit is further configured to: acquire the fusion feature vector output by the fusion processing module; The generation unit is further configured to: generate the second image based on the fused feature vector.

28. The apparatus according to claim 27, characterized in that, The input unit is further configured to: input the second feature information into the fusion processing module; and / or, The third feature information is input into the fusion processing module.

29. The apparatus according to claim 27 or 28, characterized in that, The acquisition unit is further configured to: acquire the fused text information output by the fusion processing module; The generation unit is further configured to: generate the care information based on the fused text information.

30. The apparatus according to claim 29, characterized in that, The acquisition unit is further configured to: acquire the voice adjustment parameters output by the fusion processing module; The generation unit is further configured to: generate the audio for broadcasting the care information based on the voice adjustment parameters.

31. The apparatus according to any one of claims 19-30, characterized in that, The first state is the negative state of the first object.

32. The apparatus according to any one of claims 26-31, characterized in that, The vehicle status data includes vehicle speed information. The generation unit is further configured to: generate the care information based on the first feature information and the vehicle speed information; The generation unit is further configured to: generate the voice adjustment parameters based on the first feature information and the vehicle speed information.

33. The apparatus according to any one of claims 26-32, characterized in that, The vehicle status data includes bumpiness information. The generation unit is further configured to: generate the care information based on the first feature information and the bumpiness information; The generation unit is further configured to: generate the voice adjustment parameters based on the first feature information and the turbulence information.

34. The apparatus according to any one of claims 26-33, characterized in that, The vehicle status data includes noise information. The generation unit is further configured to: generate the care information based on the first feature information and the noise information; The generation unit is further configured to: generate the speech adjustment parameters based on the first feature information and the noise information.

35. The apparatus according to any one of claims 26-34, characterized in that, The vehicle status data includes temperature information. The generation unit is further configured to: generate the care information based on the first feature information and the temperature information; The generation unit is further configured to: generate the voice adjustment parameters based on the first feature information and the temperature information.

36. The apparatus according to claim 35, characterized in that, The generation unit is further configured to: generate the care information based on at least two of the vehicle speed information, the bumpiness information, the noise information, and the temperature information and their corresponding weight information, wherein the weight information is dynamically adjusted according to the actual scenario; The generation unit is further configured to: generate the voice adjustment parameters based on at least two of the vehicle speed information, the bumpiness information, the noise information, the temperature information, and the weight information.

37. A monitoring device, characterized in that, include: Memory, used to store computer programs; A processor for executing a computer program stored in the memory to cause the apparatus to perform the method as described in any one of claims 1 to 18.

38. A vehicle, characterized in that, Includes the apparatus as described in any one of claims 19 to 37.

39. A computer-readable storage medium, characterized in that, It stores instructions that, when executed by a processor, cause the processor to implement the method as described in any one of claims 1 to 18.

40. A computer program product, characterized in that, The computer program product includes computer program code that, when run on a computer, causes the computer to perform the method as described in any one of claims 1 to 18.

41. A chip, characterized in that, The chip includes circuitry for performing the method as described in any one of claims 1 to 18.

Images