Digital human-based cabin interaction method, device and vehicle

Abstract

The disclosed embodiments provide a vehicle cabin interaction method, device and vehicle based on digital humans, which obtain status information of a living body in the vehicle cabin, determine action information matching the status information, and generate an animation of a digital human performing a corresponding action based on the action information and display it on a display device in the vehicle cabin.

Description

Technical Field

[0001] The present disclosure relates to the field of computer vision technology, and in particular to a vehicle cabin interaction method, device and vehicle based on digital human. Background Art

[0002] Currently, many vehicles are equipped with monitoring products to interact with living things in the vehicle. However, the interaction methods of traditional monitoring products are relatively rigid and lack of humanization. Summary of the invention

[0003] The present invention provides a vehicle cabin interaction method, device and vehicle based on digital human.

[0004] According to a first aspect of an embodiment of the present disclosure, a vehicle cabin interaction method based on a digital human is provided, the method comprising: acquiring status information of a living body in the vehicle cabin; determining action information matching the status information; generating an animation of a digital human performing a corresponding action based on the action information and displaying it on a display device in the vehicle cabin.

[0005] In some embodiments, the animation of the digital human performing the corresponding action is generated according to the action information and displayed on the display device in the cabin, including: determining voice information matching the status information; obtaining a corresponding voice according to the voice information, wherein the voice includes a timestamp; while playing the voice, generating and displaying an animation of the digital human performing the action at the moment corresponding to the timestamp according to the action information.

[0006] In some embodiments, the action includes multiple sub-actions, each sub-action matches a phoneme in the speech, and the timestamp includes the timestamp of each phoneme; generating and displaying an animation of the digital human performing the action at the moment corresponding to the timestamp based on the action information includes: determining the execution time of the sub-action matching each phoneme based on the timestamp of each phoneme; generating and displaying an animation of the digital human performing the sub-action matching each phoneme at the timestamp of each phoneme based on the action information.

[0007] In some embodiments, the animation of a digital human performing a corresponding action is generated according to the action information and displayed on a display device in the vehicle cabin, comprising: calling at least one frame of action slices of the digital human corresponding to the action information from an action model library; and displaying each frame of the action slices of the digital human in the at least one frame of action slices of the digital human on the display device in sequence.

[0008] In some embodiments, the status information of the living body includes first status information of the living body, and obtaining the status information of the living body in the cabin includes: collecting surveillance video of the back row in the cabin; performing liveness detection on the surveillance video and performing status analysis on the detected living body to obtain the first status information of the living body.

[0009] In some embodiments, the surveillance video is acquired by a video acquisition device installed on a rearview mirror in the vehicle cabin with a lens facing the back row of the cabin.

[0010] In some embodiments, the first status information includes at least one of the following: category information, identity information, emotion information, expression information, body movement information, seat occupancy information, and seat belt wearing information of the living body; and / or the living body includes at least one of the following: a driver, a co-pilot, a child, an elderly person, a pet, or a rear passenger.

[0011] In some embodiments, the status information of the living body includes first status information and second status information of the living body, and the first status information is obtained based on the surveillance video in the cabin; the obtaining of the status information of the living body in the cabin also includes: obtaining second status information sent by a smart device carried by the living body; the determining of action information matching the status information includes: determining action information matching both the first status information and the second status information.

[0012] In some embodiments, the second status information includes at least any one of the following: health status information and nervous system status information.

[0013] In some embodiments, obtaining the status information of a living being in a vehicle cabin includes: inputting a surveillance video in the vehicle cabin into a pre-trained neural network; and determining the status information of the living being based on an output result of the neural network.

[0014] In some embodiments, the method further includes: generating an image of the digital human before generating an animation of the digital human performing a corresponding action according to the action information and displaying it on a display device in the vehicle cabin.

[0015] In some embodiments, generating the image of the digital human includes: generating the image of the digital human according to the state information of the living body; or generating the image of the digital human according to a predetermined digital human image template.

[0016] In some embodiments, the attribute information of the living body includes at least any one of the following: age, gender, facial features, body shape, clothing, hairstyle and skin color.

[0017] In some embodiments, the method further includes: controlling the operating state of the vehicle-mounted device according to the state information.

[0018] In some embodiments, the determining of the action information that matches the status information includes: acquiring the driving status of the vehicle; and determining the action information that matches the driving status of the vehicle and the status information respectively.

[0019] According to a second aspect of an embodiment of the present disclosure, a cabin interaction device based on a digital human is provided, the device comprising: an acquisition module for acquiring status information of a living body riding in the cabin; a determination module for determining action information matching the status information; and a display module for generating an animation of a digital human performing a corresponding action based on the action information and displaying it on a display device in the cabin.

[0020] In some embodiments, the display module includes: a first determination unit, used to determine voice information matching the status information; a first acquisition unit, used to acquire corresponding voice based on the voice information, wherein the voice includes a timestamp; a first display unit, used to generate and display an animation of the digital human performing the action at the moment corresponding to the timestamp based on the action information while playing the voice.

[0021] In some embodiments, the action includes multiple sub-actions, each sub-action matches a phoneme in the speech, and the timestamp includes the timestamp of each phoneme; the first display unit includes: a determination sub-unit, used to determine the execution time of the sub-action matching each phoneme according to the timestamp of each phoneme; a display sub-unit, used to generate and display an animation of the digital human performing the sub-action matching each phoneme at the timestamp of each phoneme according to the action information.

[0022] In some embodiments, the display module includes: a calling unit, used to call at least one frame of action slices of the digital human corresponding to the action information from the action model library; and a second display unit, used to display each frame of the action slices of the digital human in the at least one frame of action slices of the digital human on the display device in sequence.

[0023] In some embodiments, the state information of the living body includes first state information of the living body, and the acquisition module includes: a collection unit, used to collect surveillance video of the rear row in the vehicle cabin; a detection and analysis unit, used to perform liveness detection on the surveillance video and perform state analysis on the detected living body to obtain the first state information of the living body.

[0024] In some embodiments, the surveillance video is acquired by a video acquisition device installed on a rearview mirror in the vehicle cabin with a lens facing the back row of the cabin.

[0025] In some embodiments, the first status information includes at least one of the following: category information, identity information, emotion information, expression information, body movement information, seat occupancy information, and seat belt wearing information of the living body; and / or the living body includes at least one of the following: a driver, a co-pilot, a child, an elderly person, a pet, or a rear passenger.

[0026] In some embodiments, the status information of the living body includes first status information and second status information of the living body, and the first status information is acquired based on the surveillance video in the cabin; the acquisition module is also used to: acquire second status information sent by the smart device carried by the living body; the determination module is used to: determine action information that matches both the first status information and the second status information.

[0027] In some embodiments, the second status information includes at least any one of the following: health status information and nervous system status information.

[0028] In some embodiments, the acquisition module includes: an input unit for inputting the surveillance video in the cabin into a pre-trained neural network; and a second determination unit for determining the state information of the living body according to the output result of the neural network.

[0029] In some embodiments, the apparatus further comprises: a generating module for generating an image of the digital human before generating an animation of the digital human performing a corresponding action according to the action information and displaying it on a display device in the vehicle cabin.

[0030] In some embodiments, the generating module is used to: generate the image of the digital human according to the state information of the living body; or generate the image of the digital human according to a predetermined digital human image template.

[0031] In some embodiments, the attribute information of the living body includes at least any one of the following: age, gender, facial features, body shape, clothing, hairstyle and skin color.

[0032] In some embodiments, the apparatus further includes: a control module configured to control the operating state of the vehicle-mounted device according to the state information.

[0033] In some embodiments, the determination module is used to: obtain the driving state of the vehicle; and determine action information that matches the driving state of the vehicle and the state information respectively.

[0034] According to a third aspect of an embodiment of the present disclosure, a computer-readable storage medium is provided, on which a computer program is stored. When the program is executed by a processor, the method described in any one of the embodiments is implemented.

[0035] According to a fourth aspect of an embodiment of the present disclosure, a computer device is provided, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any one of the embodiments is implemented.

[0036] According to a fifth aspect of an embodiment of the present disclosure, a vehicle is provided, wherein a display device, a monitoring system, and a digital human-based cabin interaction device as described in any embodiment of the present disclosure or a computer device as described in any embodiment of the present disclosure are arranged in the vehicle cabin.

[0037] The disclosed embodiment obtains status information of living things in the cabin, determines action information matching the status information, generates an animation of a digital human performing a corresponding action based on the action information, and displays the animation on a display device in the cabin. According to the status information of the living things in the cabin, a digital human performing different actions can be displayed, thereby achieving anthropomorphic interaction, making the interaction process more natural, and improving the living things' acceptance of feedback information during the interaction process, thereby improving the safety of the living things during vehicle driving.

[0038] It is to be understood that both the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0039] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present disclosure and are used to illustrate the technical solutions of the present disclosure together with the specification.

[0040] Figure 1 It is a flow chart of the vehicle cabin interaction method based on digital human according to an embodiment of the present disclosure.

[0041] Figure 2A Schematic diagram of a digital human according to an embodiment of the present disclosure.

[0042] Figure 2B Schematic diagrams of digital humans according to other embodiments of the present disclosure.

[0043] Figure 3 It is a schematic diagram of a vehicle cabin interaction method based on a digital human according to an embodiment of the present disclosure.

[0044] Figure 4 It is a block diagram of a vehicle cabin interaction device based on a digital human according to an embodiment of the present disclosure.

[0045] Figure 5 It is a schematic diagram of the structure of a computer device according to an embodiment of the present disclosure.

[0046] Fig. 6A and Figure 6B They are respectively schematic diagrams of vehicles according to embodiments of the present disclosure. DETAILED DESCRIPTION

[0047] Exemplary embodiments will be described in detail herein, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, the same numbers in different drawings represent the same or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with the present disclosure. Instead, they are merely examples of devices and methods consistent with some aspects of the present disclosure as detailed in the appended claims.

[0048] The terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. The singular forms of "a", "said" and "the" used in this disclosure and the appended claims are also intended to include plural forms, unless the context clearly indicates other meanings. It should also be understood that the term "and / or" used herein refers to and includes any or all possible combinations of one or more associated listed items. In addition, the term "at least one" herein means any combination of at least two of any one or more of a plurality of.

[0049] It should be understood that although the terms first, second, third, etc. may be used in the present disclosure to describe various information, such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other. For example, without departing from the scope of the present disclosure, the first information may also be referred to as the second information, and similarly, the second information may also be referred to as the first information. Depending on the context, the word "if" as used herein may be interpreted as "at the time of" or "when" or "in response to determining".

[0050] In order to enable those skilled in the art to better understand the technical solutions in the embodiments of the present disclosure and to make the above-mentioned purposes, features and advantages of the embodiments of the present disclosure more obvious and understandable, the technical solutions in the embodiments of the present disclosure are further described in detail below in conjunction with the accompanying drawings.

[0051] With the popularization of private cars and the rapid development of smart cabins, driving safety has received widespread attention. One way to improve safety is to set up a monitoring system in the cabin to monitor the living things in the cabin, and output interactive information based on the monitoring results, so as to remind the living things in the cabin in time when necessary. The traditional interactive method is generally to output voice prompt information through the audio playback device in the cabin, or to output text prompt information on the vehicle's display device at the same time as the voice prompt information is output. However, this interactive method will inevitably give people a sense of stiffness in interacting with the machine, and sometimes it is easy to cause the living things to have a low degree of acceptance of the interactive information, thereby reducing the safety of the living things during vehicle driving.

[0052] Based on this, the embodiment of the present disclosure provides a vehicle cabin interaction method based on digital human, such as Figure 1 As shown, the method may include:

[0053] Step 101: Acquire status information of a living being in the vehicle cabin;

[0054] Step 102: Determine action information matching the state information;

[0055] Step 103: Generate an animation of the digital human performing the corresponding action according to the action information and display it on a display device in the vehicle cabin.

[0056] In step 101, the vehicle may be a private car, a school bus, a bus, or other types of vehicles. For the convenience of description, a private car is taken as an example to describe the solution of the embodiment of the present disclosure. The living body may include but is not limited to at least one of a driver, a co-driver, a child, an elderly person, a pet, and a rear passenger. According to actual needs, different living bodies can be interacted with to meet the interaction needs of different scenarios.

[0057] For example, during the ride, there are certain safety hazards for living beings sitting in the back row (for example, children and other rear passengers or pets, etc.), for example, children may unfasten their seat belts and pets may leave their seats. However, the driver needs to focus on driving during driving and cannot take into account the situation in the back row of the cabin. In order to improve the safety of the living beings in the back row during the ride, the digital human can interact with the living beings sitting in the back row of the cabin. Therefore, in some embodiments, the living beings are the living beings sitting in the back row of the cabin. By interacting with the living beings sitting in the back row through the digital human, the acceptance of the living beings sitting in the back row to the feedback information during the interaction process can be improved, thereby improving the safety of the living beings sitting, so that the driver can focus on driving during driving without having to devote too much energy to taking into account the situation in the back row.

[0058] In some embodiments, the status information may include first status information of the living body, for example, at least one of category information, identity information, attribute information, emotion information, expression information, body movement information, seat seating information, and seat belt wearing information of the living body.

[0059] The category information is used to indicate the category of the living body, and the category may include humans and / or animals. The identity information may include identification information of the living body, which is used to uniquely identify each living body. The attribute information is used to indicate the characteristic attributes of the living body, and may include but is not limited to at least one of age, gender, facial features, body shape, clothing, hairstyle and skin color. The emotion information is used to indicate the emotion category of the living body, and the emotion category may include but is not limited to at least one category such as happy, sad, angry, shy, surprised, excited, fearful, angry, calm, etc. The expression information is used to characterize the facial expression of the living body, and the facial expression may include but is not limited to at least one of smiling, pouting, crying, squinting, making faces, etc. The body movement information is used to characterize the action performed by the living body, for example, at least one of clapping, stomping, opening the car door, and sticking the head or hand out of the car window. The seat seating information is used to indicate whether the living body has left the seat (for example, a child seat). The seat belt wearing information is used to indicate whether the living body has fastened the seat belt.

[0060] Furthermore, the state information of the living body may also include the second state information of the living body, for example, at least one of health status information and nervous system state information. The health status information is used to characterize the health status of the living body, for example, it may include but is not limited to at least one of heart rate, blood pressure, blood lipids, blood sugar, etc. The nervous system state information is used to characterize the excitement level of the nervous system of the living body, for example, whether the living body is sleepy or asleep. In the case where the first state information and the second state information are obtained at the same time, the action information is the action information that matches both the first state information and the second state information.

[0061] By acquiring different state information, the states of different aspects of the living body can be determined. The more categories of state information acquired, the more comprehensive and accurate the determined state of the living body is, so that the actions performed by the digital human are more consistent with the state of the living body, thereby improving the degree of anthropomorphism of the digital human.

[0062] The status information of the living body can be obtained based on various methods, for example, it can be obtained based on the surveillance video in the cabin, or it can be obtained based on the audio monitoring results in the cabin, or it can be obtained from the smart device carried by the living body. The status information of the living body can also be obtained based on a combination of multiple means. Obtaining status information in different ways can increase the comprehensiveness and flexibility of the acquired status information. According to different application scenarios, different status information acquisition methods can be selected to obtain different types of status information for different application scenarios in a targeted manner. For example, in a dimly lit scene, the status information of the living body can be obtained based on the surveillance video and audio monitoring results in the cabin, thereby improving the accuracy of obtaining the status information.

[0063] In some embodiments, a monitoring system may be installed in the vehicle cabin. According to the needs of the actual application scenario, the monitoring system may monitor living beings in any area of ​​the vehicle cabin. For example, the monitoring system may monitor the driver in the driver's seat; for another example, the monitoring system may also monitor the co-driver in the co-pilot seat; for another example, the monitoring system may also monitor living beings in the back row of the vehicle cabin. Alternatively, the monitoring system may also monitor living beings in multiple areas of the vehicle cabin (for example, the co-pilot seat and the back row).

[0064] The monitoring system may include at least one of a visual monitoring system and an audio monitoring system, and the visual monitoring system is used to obtain visual monitoring results (e.g., monitoring videos or monitoring images) of the living body. In some embodiments, the monitoring video can be obtained by a video acquisition device installed on the rearview mirror in the cabin, with the lens facing the rear row of the cabin. The video acquisition device is installed on the rearview mirror in the cabin, so that the field of view of the video acquisition device is wide and not easily blocked by objects in the cabin, so that a relatively complete monitoring video of the rear row of the cabin can be collected. The audio monitoring system is used to obtain audio monitoring results of the living body, and the audio monitoring results may include the voice emitted by the living body and / or the sound emitted when the living body performs an action (e.g., at least one action such as opening a car door, fastening a seat belt, and knocking on a car window). The first state information of the living body can be determined based on the monitoring results.

[0065] In the case of obtaining the status information of the living body based on the surveillance video in the cabin, optionally, the surveillance video can be input into a pre-trained neural network, and the status information of the living body can be determined according to the output result of the neural network. Optionally, the status information of the living body can be determined by combining a neural network with other algorithms (for example, a face recognition algorithm and / or a skeleton key point detection algorithm, etc.). Optionally, other methods can be used to determine the status information of the living body, which will not be described here. Acquiring the status information of the living body through a neural network can improve the accuracy of obtaining the status information.

[0066] The neural network may include an input layer, at least one intermediate layer and an output layer, wherein the input layer, at least one intermediate layer and the output layer each include one or more neurons. The intermediate layer generally refers to a layer located between the input layer and the output layer, such as a hidden layer. In an optional example, the intermediate layer of the neural network may include but is not limited to at least one of a convolutional layer, a ReLU (Rectified Linear Units) layer, etc. The more intermediate layers the neural network includes, the deeper the network is. The neural network may specifically be a deep neural network or a convolutional neural network.

[0067] Alternatively, a surveillance video of the rear row in the vehicle cabin may be collected; liveness detection may be performed on the surveillance video and the state of the detected liveness may be analyzed to obtain first state information of the liveness. Based on the surveillance video, on the one hand, more comprehensive state information can be obtained; on the other hand, since there is a time correlation between the multiple frames of target images included in the surveillance video, this time correlation may be used to improve the accuracy of obtaining the state information. For example, a surveillance video of the liveness may be obtained, face recognition may be performed on the surveillance video, and the identity information of the liveness may be determined based on the face recognition result.

[0068] For another example, the emotional information of the living body can be identified based on the surveillance video. Specifically, at least one frame of the target image including the living body can be obtained from the surveillance video; and the facial sub-image of the living body can be intercepted from the target image. The action of each of the at least two organs on the face represented by the facial sub-image is identified, for example, frowning, staring, and raising the corners of the mouth. Based on the identified action of each organ, the emotional information on the face represented by the facial sub-image is determined. Before identifying the action of each organ, the facial sub-image can also be subjected to image preprocessing. Specifically, the position information of key points (for example, the corners of the eyes, the corners of the mouth, the eyebrows, the tail of the eyebrows, the nose, etc.) in the facial sub-image can be determined; based on the position information of the key points, the facial sub-image is subjected to affine transformation, and the facial sub-images with different orientations in the facial sub-image are converted into facial sub-images with a frontal orientation; the facial sub-image with a frontal orientation is normalized to obtain the processed facial sub-image.

[0069] For another example, a facial sub-image of a living body can be captured from the target image. Based on the facial sub-image, the left eye opening and closing state information and the right eye opening and closing state information of the living body are determined. In a specific implementation, the facial sub-image is input into a trained neural network, and the left eye opening and closing state information and the right eye opening and closing state information of the living body are determined according to the output result of the neural network. Then, based on the left eye opening and closing state information and the right eye opening and closing state information of the living body, it is determined whether the living body feels sleepy or has fallen asleep. Specifically, based on the left eye opening and closing state information and the right eye opening and closing state information corresponding to multiple consecutive frames of the target image of the living body, the cumulative duration of the eyes being closed of the living body can be determined; when the cumulative duration of the eyes being closed is greater than a preset threshold, it is determined that the living body has fallen asleep; when the cumulative duration of the eyes being closed is less than or equal to the preset threshold, it is determined that the living body is not asleep.

[0070] In the case of obtaining the status information of the living body based on the audio monitoring results in the vehicle cabin, the voice emitted by the living body can be obtained, voice recognition can be performed on the voice, and then the identity information and / or emotional information of the living body can be determined based on the voice recognition result.

[0071] When the state information of the living body is obtained from the smart device carried by the living body, the second state information collected by the pre-associated smart device can be received. The smart device can be a wearable device, such as a smart bracelet, smart glasses, etc., or a handheld terminal, such as a mobile phone, a tablet computer, etc.

[0072] In step 102, action information matching the state information may be determined. In some embodiments, the degree of matching between each type of action information and the state information may be determined, and the action information with the highest degree of matching may be determined as the action information matching the state information. In the case where the state information includes multiple types of information, the degree of matching between the same action information and various types of information in the state information may be determined respectively, and then the degree of matching between the action information and the state information may be determined based on the degree of matching corresponding to the various types of information. For example, a weighted average may be performed on the degree of matching corresponding to the various types of information. In other embodiments, a mapping relationship between different state information and matching action information may be established in advance, and the matching action information may be determined based on the mapping relationship.

[0073] In step 103, after the matching action information is determined, an animation of the digital human performing the corresponding action can be generated according to the matching action information and displayed on a display device in the cabin (e.g., a central control screen or a display device behind a seat). The digital human can be a virtual image generated by software and displayed on a display device in the cabin. The method of generating and displaying the digital human by software is low-cost, has a fast response speed, low subsequent maintenance costs, and is easy to update and upgrade.

[0074] like Figure 2A and 2B As shown, the image of the digital human can be a cartoon image, a 3D image generated based on the real image of the living body, or other types of images. In some embodiments, the image of the digital human can be a fixed image, that is, the image of the digital human displayed on the display device each time is the same, for example, it can be an image of a boy, or an image of a cartoon character (for example, Doraemon). In other embodiments, the image of the digital human can also be dynamically generated according to actual conditions, that is, in different situations, the image of the digital human displayed on the display device may be different.

[0075] Optionally, different digital human images can be displayed according to the state information of the living body (for example, at least one of identity information, emotion information, and attribute information). For example, the first digital human image corresponding to Zhang San can be displayed for Zhang San, and the second digital human image corresponding to Li Si can be displayed for Li Si. For another example, when the emotion of the living body is happy, a digital human image with a smiling expression and / or wearing bright clothes is displayed. For another example, when the living body is a child, a child digital human image is displayed; when the living body is an adult, an adult digital human image is displayed. For another example, when the living body has long hair, a long-haired digital human image is displayed; when the living body has short hair, a short-haired digital human image is displayed.

[0076] In practical applications, the image of the digital human can be generated according to a predetermined digital human image template. The image template can be generated in advance by the user, or it can also be received from a server. For example, when a living person gets on a bus, the image of the living person can be captured by a camera on the bus, or an image of the living person sent by a user terminal can be received, and the image template of the living person can be generated according to the image of the living person. Specifically, the image of the living person can be subjected to attribute detection, the attributes of the living person can be obtained, and the digital human corresponding to the living person can be generated based on the attributes of the living person. The image template that has been generated can also be regenerated (for example, replacing the existing image template with a new image template) or partially edited (for example, changing the hairstyle of the image template). When generating an image template, the degree of cartoonization of the image template can be customized.

[0077] When displaying the animation of the digital human, the corresponding template can be called according to the actual situation to generate the image of the digital human. The template can be an image template of an adult, an image template of a child, or an image template of a pet, etc. In the case where the living body is an adult, the image template of an adult can be called; in the case where the living body is a child, the image template of a child can be called. Further, since the state information of the living body when riding in a car may be inconsistent with the template, after calling the corresponding template, the attribute information of the digital human template can also be adjusted according to the state information of the living body, so that the image of the digital human in the displayed animation is consistent with the state information of the living body. For example, the expression and clothing of the digital human template are adjusted according to the mood of the living body. Further, the display interface of the display device when displaying the digital human can also be adjusted according to the state information of the living body. For example, when the mood of the living body is happy, the background color of the display interface is set to a bright color, and / or a flower-scattering effect is displayed on the display interface.

[0078] By generating and displaying different digital human images in the above manner, the image of the digital human can be made consistent with the living body to a certain extent, so that the living body can feel friendly and warm during the interaction process, and the living body's acceptance of feedback information during the interaction with the digital human can be improved.

[0079] In addition to displaying the animation of the digital human performing the corresponding action on the display device in the cabin, in order to further improve the interactive effect, the voice information matching the state information can also be determined, and the corresponding voice can be played synchronously according to the voice information while displaying the animation. For example, the matching action information is the action information corresponding to the action of unbuckling the seat belt. Then, while displaying the animation of the digital human performing the corresponding action on the display device, a voice can also be played: "Little friend, it is very dangerous to unbuck the seat belt while driving." In actual applications, a neural network can be used to determine the voice information matching the state information, or a mapping relationship between different state information and voice information can be pre-established, and the matching voice information can be determined according to the mapping relationship. Furthermore, while playing the voice, the subtitle information corresponding to the voice can also be displayed on the display interface.

[0080] The played voice is synchronized with the displayed animation, which means that the played voice content matches the action performed by the digital human in the animation, and the action here may include at least one of body movements, mouth movements, eye movements, etc. For example, when the digital human performs a waving action in the animation, and the mouth movement is the action corresponding to "hello", the voice content "hello" is played. Specifically, the voice information matching the state information can be determined; the corresponding voice is obtained according to the voice information, and the voice includes a timestamp; while playing the voice, an animation of the digital human performing the action at the time corresponding to the timestamp is generated and displayed according to the action information. In the above manner, the played voice is played synchronously with the action of the digital human in the animation, which can further improve the degree of anthropomorphism of the digital human and make the interaction between the digital human and the living body more natural.

[0081] The voice can be pulled from the voice database, and the pulled voice carries the timestamp of the voice, which is used to synchronize the time when the digital human in the animation performs the corresponding action with the voice. When pulling the voice, the state information of the living body (for example, at least one of attribute information and emotional information) can also be obtained, and the state information of the living body is sent to the voice database to pull the corresponding voice from the voice database. For example, when the living body is a child, the voice that matches the timbre of the child is pulled.

[0082] A speech often includes multiple phonemes. Phonemes are the smallest speech units divided according to the natural properties of speech. According to the analysis of the pronunciation action in the syllable, one pronunciation action constitutes a phoneme. For example, "hello" includes two phonemes, "you" and "good". In the case where the speech includes multiple phonemes, the timestamp may include the timestamp of each phoneme. An action generally includes multiple sub-actions. For example, the action of waving may include the sub-action of swinging the arm to the left and the sub-action of swinging the arm to the right. In order to make the displayed digital human more vivid, each sub-action can be matched with a phoneme in the speech. Specifically, according to the timestamp of each phoneme, the execution time of the sub-action matching each phoneme can be determined; according to the action information, an animation of the digital human performing the sub-action matching each phoneme at the timestamp of each phoneme is generated and displayed. For example, while playing the phoneme "you", the mouth shape movement that matches "you" is displayed, and the digital human's arm swinging to the left is displayed. While playing the phoneme "good", the mouth shape movement that matches "good" is displayed, and the digital human's arm swinging to the right is displayed. By synchronizing each phoneme with the digital human's movement, the accuracy of the synchronization process is improved, making the digital human's movement and voice playback more realistic, and further improving the digital human's degree of anthropomorphism.

[0083] The above-mentioned action corresponding to the action information can be called from the action model library. Specifically, at least one frame of action slices of the digital human corresponding to the action information can be called from the action model library; each frame of the action slices of the digital human in the at least one frame of the action slices of the digital human is displayed on the display device in sequence. Different action slices correspond to different at least one of the digital human's body movements, facial expression movements, mouth movements, eye movements, etc. By calling and displaying the corresponding action slices in sequence, the animation of the digital human performing the corresponding action can be displayed on the display device. In the above manner, the animation of the digital human can be displayed by simply calling the corresponding slice, with high display efficiency and low cost.

[0084] The above-mentioned speech database and action model library can be updated so that more speech materials and action materials can be added to the speech database and action model library respectively. The update methods of the speech database and the action model library are similar. Here, only the speech database is taken as an example to illustrate the update method of the database. The update method of the action model library can refer to the speech database and will not be repeated here. When updating the speech database, an update package can be downloaded from the server, the data in the update package can be parsed, and the data in the update package can be added to the speech database, or the data in the update package can be used to replace the original data in the speech database.

[0085] As an optional method, the voice database can be updated at a certain time interval; as another optional method, the update prompt information pushed by the server can be received, and the voice database can be updated in response to the update prompt information; as another optional method, an update instruction sent by the living body can be received, an update request is sent to the server in response to the update instruction, and an update package returned by the server in response to the update request is received, and then the voice database is updated based on the update package. The voice database can also be updated in other ways, which will not be repeated here.

[0086] In some embodiments, the driving state of the vehicle can also be obtained, and the action information that matches the driving state of the vehicle and the state information can be determined. For example, when the vehicle is in motion, if the body movement of the living body is detected to be unfastening the seat belt, it is determined that the matching action information includes action information corresponding to the action of indicating to the living body not to open the seat belt. If the body movement of the living body is detected to be unfastening the seat belt when the vehicle is stopped and the engine is turned off, it is determined that the matching action information includes action information corresponding to the action of waving goodbye. In this way, the matching action information can be determined more accurately, reducing the chance of misjudgment.

[0087] In some embodiments, the operating state of the vehicle-mounted device can also be controlled according to the state information. The vehicle-mounted device includes at least one of the lighting equipment, air conditioning, windows, audio playback equipment and seats in the vehicle cabin. For example, when the living body feels sleepy or has fallen asleep, the seat can be laid flat so that the living body can lie as flat as possible on the seat. For another example, when the living body feels stuffy, the window or air conditioning can be opened to adjust the cabin environment in the cabin to provide a more comfortable and safe riding environment for the living body.

[0088] In practical applications, through the embodiments of the present disclosure, it is possible to detect behaviors such as a living body entering or leaving a seat in a vehicle cabin, opening or closing a door, fastening or unfastening a seat belt, and displaying an animation of a digital human performing corresponding actions based on the state information of the living body, and interact with the living body according to the above behaviors.

[0089] Taking the detection of a living body entering or leaving a seat in a vehicle cabin as an example, in some embodiments, a target image in the vehicle cabin can be acquired, the living body in the target image can be identified, and based on the position information of the living body, it is determined whether the living body is located on a seat in the vehicle cabin. The target image can be acquired based on the surveillance video in the vehicle cabin. Specifically, the object information of each object in the target image (including the position information of the center point of the object and the object type information corresponding to the center point of the object) can be determined, and the living body and the seat can be filtered out from each object in the target image based on the object type information, and then it is determined whether the living body is located on the seat based on the position of the center point of the living body and the position of the center point of the seat.

[0090] In some embodiments, the object information of each object in the target image can be determined by the following steps: extracting features from the target image to obtain a first feature map corresponding to the target image. Obtaining the response value of each feature point in the first feature map as the center point of the object from a first preset channel of the first feature map. Dividing the first feature map into multiple sub-regions, and determining the maximum response value and the feature point corresponding to the maximum response value in each sub-region; taking the target feature point with the maximum response value greater than a preset threshold as the center point of the object, and determining the position information of the center point of the object based on the position index of the target feature point on the first feature map.

[0091] In some embodiments, the object type information can be determined by the following steps: extracting features from the target image to obtain a second feature map corresponding to the target image. Determine the position index of the target feature point on the second feature map based on the position index of the target feature point on the first feature map. Obtain the object type information corresponding to the target feature point from the position corresponding to the position index of the target feature point on the second feature map. After determining the center point of the object, the object type information corresponding to the center point of the object can be accurately obtained by using the position index of the center point.

[0092] After determining the object type information corresponding to the center point of each object, the following steps can be used to identify specific groups (for example, children, pets, etc.) in the target image, so that the digital human can interact with the specific group. For the convenience of description, children are taken as an example below. The identification methods for other groups are similar and will not be repeated here. First, based on the position offset information corresponding to the center point of each human body, the predicted position information of the center point of the face matching each human body is determined respectively; wherein the human body and face belonging to the same person match. When determining the above-mentioned position offset information, it can be specifically that the target image is first input into a neural network for image feature extraction to obtain an initial feature map. Thereafter, the initial feature map is input into a neural network for determining the above-mentioned position offset information to obtain a feature map, based on which the position offset information corresponding to the center point of each human body can be determined.

[0093] Then, based on the determined predicted position information and the position information of the center point of each face, the face that matches each human body is determined. Here, specifically, the face corresponding to the position of the center point closest to the position corresponding to the predicted position information is used as the face that matches the human body. For the successfully matched human body and face, the object type information corresponding to the center point of the successfully matched human body and the object type information corresponding to the center point of the face are used to determine whether the person to whom the successfully matched human body and face belong is a child. Here, if the object type information corresponding to the center point of the successfully matched human body indicates that the corresponding human body belongs to a child or the object type information corresponding to the center point of the face indicates that the corresponding face belongs to a child, then the person to whom the successfully matched human body and face belong is determined to be a child. For the human body that is not successfully matched, the object type information corresponding to the center point of the human body is used to determine whether the person to whom the center point of the human body belongs is a child. Specifically, if the object type information corresponding to the center point of the human body indicates a child, then the person to whom the human body belongs is determined to be a child.

[0094] The solution of the embodiment of the present disclosure is described below with reference to specific examples.

[0095] When a person gets on the bus and takes a seat, the monitoring system obtains the monitoring video of the person, detects that the type of the person is human, the identity of the person is Xiao A, the attribute of the person is a child, and detects that the emotion of the person Xiao A is happy. Then, an animation of a digital person with a child image, a smile on his face, and wearing bright clothes raising his hand to say hello is generated and displayed, and a voice greeting is played at the same time: Xiao A, you are in a good mood today. Figure 3As shown, the lip shape and body movements of the digital human in the animation match the voice being played. If the identity of the living person is not recognized, a general term can be used when greeting based on attribute information such as gender and age, for example, children, gentlemen, etc. After the living person Xiao A takes a seat, Xiao A's emotions are relatively calm at this time, so voice interaction is not required. After the vehicle starts driving, it is detected that the living person Xiao A is not wearing a seat belt, then an animation of a digital human with a child's image performing a demonstration action of wearing a seat belt is displayed on the central control screen, and a voice is played at the same time: Xiao A, come and fasten your seat belt with me. The lip shape and body movements of the digital human in the animation match the voice being played.

[0096] After a while, if Xiao A says, "It's a bit hot," the car window will be opened and the air conditioner will be turned on. After another period of time, Xiao A interacts with the digital human in a preset way (for example, clicking on the digital human, looking at the digital human, or calling the digital human by voice, etc.), and the digital human interacts with Xiao A. The interaction method may include chatting, playing games, or controlling music playback. During the interaction, the corresponding digital human animation is displayed according to the interactive content and the voice is played synchronously. When Xiao A got off the car, the monitoring system detected that there was something left on Xiao A's seat, so that the animation of the digital human performing a waving action can be displayed on the central control screen, and the voice can be played synchronously: Xiao A, you forgot something in the car, come back and get it. After Xiao A gets off the car, the windows and music can also be closed.

[0097] The disclosed embodiment can generate and display the animation of the digital human performing corresponding actions on the display device in the cabin according to the state information of the living body in the cabin, thereby realizing an anthropomorphic interaction, making the interaction more natural, making people feel the warmth of human-computer interaction, improving the fun, comfort and sense of companionship of the ride, thereby improving the acceptance of the living body to the feedback information in the interaction process, thereby improving the safety of the living body during vehicle driving. In addition, the disclosed embodiment generates the animation of the digital human through software, which is low in cost, fast in response speed, and easy for later maintenance and update and upgrade.

[0098] Those skilled in the art will appreciate that, in the above method of specific implementation, the order in which the steps are written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of the steps should be determined by their functions and possible internal logic.

[0099] like Figure 4 As shown, the present disclosure also provides a vehicle cabin interaction device based on a digital human, the device comprising:

[0100] An acquisition module 401 is used to acquire status information of a living body in the vehicle cabin;

[0101] A determination module 402, configured to determine action information matching the state information;

[0102] The display module 403 is used to generate an animation of the digital human performing the corresponding action according to the action information and display it on a display device in the cabin.

[0103] In some embodiments, the display module includes: a first determination unit, used to determine voice information matching the status information; a first acquisition unit, used to acquire corresponding voice based on the voice information, wherein the voice includes a timestamp; a first display unit, used to generate and display an animation of the digital human performing the action at the moment corresponding to the timestamp based on the action information while playing the voice.

[0104] In some embodiments, the action includes multiple sub-actions, each sub-action matches a phoneme in the speech, and the timestamp includes the timestamp of each phoneme; the first display unit includes: a determination sub-unit, used to determine the execution time of the sub-action matching each phoneme according to the timestamp of each phoneme; a display sub-unit, used to generate and display an animation of the digital human performing the sub-action matching each phoneme at the timestamp of each phoneme according to the action information.

[0105] In some embodiments, the display module includes: a calling unit, used to call at least one frame of action slices of the digital human corresponding to the action information from the action model library; and a second display unit, used to display each frame of the action slices of the digital human in the at least one frame of action slices of the digital human on the display device in sequence.

[0106] In some embodiments, the state information of the living body includes first state information of the living body, and the acquisition module includes: a collection unit, used to collect surveillance video of the rear row in the vehicle cabin; a detection and analysis unit, used to perform liveness detection on the surveillance video and perform state analysis on the detected living body to obtain the first state information of the living body.

[0107] In some embodiments, the surveillance video is acquired by a video acquisition device installed on a rearview mirror in the vehicle cabin with a lens facing the back row of the cabin.

[0108] In some embodiments, the first status information includes at least one of the following: category information, identity information, emotion information, expression information, body movement information, seat occupancy information, and seat belt wearing information of the living body; and / or the living body includes at least one of the following: a driver, a co-pilot, a child, an elderly person, a pet, or a rear passenger.

[0109] In some embodiments, the status information of the living body includes first status information and second status information of the living body, and the first status information is acquired based on the surveillance video in the cabin; the acquisition module is also used to: acquire second status information sent by the smart device carried by the living body; the determination module is used to: determine action information that matches both the first status information and the second status information.

[0110] In some embodiments, the second status information includes at least any one of the following: health status information and nervous system status information.

[0111] In some embodiments, the acquisition module includes: an input unit for inputting the surveillance video in the cabin into a pre-trained neural network; and a second determination unit for determining the state information of the living body according to the output result of the neural network.

[0112] In some embodiments, the apparatus further comprises: a generating module for generating an image of the digital human before generating an animation of the digital human performing a corresponding action according to the action information and displaying it on a display device in the vehicle cabin.

[0113] In some embodiments, the generating module is used to: generate the image of the digital human according to the state information of the living body; or generate the image of the digital human according to a predetermined digital human image template.

[0114] In some embodiments, the attribute information of the living body includes at least any one of the following: age, gender, facial features, body shape, clothing, hairstyle and skin color.

[0115] In some embodiments, the apparatus further includes: a control module configured to control the operating state of the vehicle-mounted device according to the state information.

[0116] In some embodiments, the determination module is used to: obtain the driving state of the vehicle; and determine action information that matches the driving state of the vehicle and the state information respectively.

[0117] In some embodiments, the functions or modules included in the device provided by the embodiments of the present disclosure can be used to execute the method described in the above method embodiments. The specific implementation can refer to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0118] An embodiment of the present specification also provides a computer device, which at least includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the method described in any of the above embodiments is implemented.

[0119] Figure 5A more specific hardware structure diagram of a computing device provided in an embodiment of this specification is shown, and the device may include: a processor 501, a memory 502, an input / output interface 503, a communication interface 504, and a bus 505. The processor 501, the memory 502, the input / output interface 503, and the communication interface 504 are connected to each other in communication within the device through the bus 505.

[0120] The processor 501 can be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this specification.

[0121] The memory 502 can be implemented in the form of ROM (Read Only Memory), RAM (Random Access Memory), static storage device, dynamic storage device, etc. The memory 502 can store an operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented by software or firmware, the relevant program codes are stored in the memory 502 and called and executed by the processor 501.

[0122] The input / output interface 503 is used to connect the input / output module to realize information input and output. The input / output module can be configured in the device as a component (not shown in the figure), or it can be externally connected to the device to provide corresponding functions. The input device may include a keyboard, a mouse, a touch screen, a microphone, various sensors, etc., and the output device may include a display, a speaker, a vibrator, an indicator light, etc.

[0123] The communication interface 504 is used to connect a communication module (not shown) to realize communication interaction between the device and other devices. The communication module can realize communication through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WIFI, Bluetooth, etc.).

[0124] The bus 505 includes a path for transmitting information between various components of the device (eg, the processor 501 , the memory 502 , the input / output interface 503 , and the communication interface 504 ).

[0125] It should be noted that, although the above device only shows a processor 501, a memory 502, an input / output interface 503, a communication interface 504, and a bus 505, in a specific implementation process, the device may also include other components necessary for normal operation. In addition, it can be understood by those skilled in the art that the above device may also only include components necessary for implementing the embodiments of the present specification, and does not necessarily include all the components shown in the figure.

[0126] like Fig. 6A and Figure 6B As shown, the embodiment of the present disclosure further provides a vehicle, wherein a display device 601, a monitoring system 602, and a digital human-based cabin interaction device 603 or a computer device 604 are provided in the cabin of the vehicle.

[0127] The display device 601 is used to display the animation of the digital human performing corresponding actions. The display device 601 may include at least one of a central control screen of the vehicle and a display screen installed on the back of a vehicle seat.

[0128] The monitoring system 602 may include at least one of a visual monitoring system and a voice monitoring system. The visual monitoring system may include at least one camera, which may be installed above the area to be monitored to collect video or images of the area to be monitored. For example, the camera may be installed on the front windshield of the vehicle, or installed above a seat or other location. For another example, the camera may be installed on a rearview mirror in the cabin, with the lens facing the back row of the cabin. This installation method allows the camera to have a wider field of view, making it easier to obtain monitoring video of the back row of the cabin. The voice monitoring system may include at least one microphone for collecting audio signals from the area to be monitored.

[0129] The digital human-based cabin interaction device 603 of the embodiment of the present disclosure may adopt the cabin interaction device of any of the above embodiments, and the computer device 604 of the embodiment of the present disclosure may adopt the computer device of any of the above embodiments. The digital human-based cabin interaction device 603 or computer device 604 may be integrated into the central control system of the vehicle. The monitoring system 602 may communicate with the cabin interaction device 603 or computer device 604 via an in-vehicle communication bus, for example, the in-vehicle communication bus may be a controller area network (CAN) bus.

[0130] In some embodiments, the digital human-based cabin interaction device 603 or computer device 604 can also control vehicle-mounted equipment based on the status information, such as at least one of the lighting equipment, air conditioning, windows, audio playback equipment and seats in the cabin.

[0131] In some embodiments, the digital human-based cabin interaction device 603 or computer device 604 can also be connected to the user's smart device or voice database, motion model database, etc. through a network to interact with the smart device or voice database, motion model database.

[0132] The present disclosure also provides a computer-readable storage medium on which a computer program is stored. When the program is executed by a processor, the method described in any of the above embodiments is implemented.

[0133] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0134] It can be known from the above description of the implementation mode that the technicians in this field can clearly understand that the embodiments of this specification can be implemented by means of software plus the necessary general hardware platform. Based on such an understanding, the technical solution of the embodiments of this specification can be essentially or the part that contributes to the prior art can be embodied in the form of a software product, which can be stored in a storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments of this specification.

[0135] The systems, devices, modules or units described in the above embodiments may be implemented by computer chips or entities, or by products with certain functions. A typical implementation device is a computer, which may be in the form of a personal computer, a laptop computer, a cellular phone, a camera phone, a smart phone, a personal digital assistant, a media player, a navigation device, an email transceiver, a game console, a tablet computer, a wearable device or a combination of any of these devices.

[0136] Each embodiment in this specification is described in a progressive manner, and the same and similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment. The device embodiment described above is merely schematic, wherein the modules described as separate components may or may not be physically separated, and the functions of each module can be implemented in the same one or more software and / or hardware when implementing the embodiment scheme of this specification. It is also possible to select some or all of the modules according to actual needs to achieve the purpose of the embodiment scheme. A person of ordinary skill in the art can understand and implement it without paying creative labor.

[0137] The above is only a specific implementation of the embodiments of this specification. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the embodiments of this specification. These improvements and modifications should also be regarded as the protection scope of the embodiments of this specification.

Claims

1. A vehicle cabin interaction method based on digital human, characterized in that: The method comprises: Acquiring status information of a living body in the vehicle cabin; specifically comprising: determining object information of each object in a target image including the living body, including position information of a center point of the object and object type information corresponding to the center point of the object; determining action information matching the state information; generating an animation of a digital human performing a corresponding action according to the action information and displaying it on a display device in the vehicle cabin; Based on the position information and object type information of each identified object, a specific group in the target image is identified, so that the digital human interacts with the specific group; The position information of the center point of each object in the target image is determined based on the following method: Performing feature extraction on the target image to obtain a first feature map corresponding to the target image; Obtaining, from a first preset channel of the first feature map, a response value of each feature point in the first feature map as a center point of an object; Divide the first feature map into multiple sub-regions, and determine the maximum response value and the feature point corresponding to the maximum response value in each sub-region; The target feature point whose maximum response value is greater than a preset threshold is taken as the center point of the object, and the position information of the center point of the object is determined based on the position index of the target feature point on the first feature map.

2. The method according to claim 1, characterized in that The animation of the digital human performing the corresponding action is generated according to the action information and displayed on the display device in the vehicle cabin, including: Determining voice information matching the status information; Acquire corresponding voice according to the voice information, wherein the voice includes a timestamp; While playing the voice, an animation of the digital human performing the action at the moment corresponding to the timestamp is generated and displayed according to the action information.

3. The method according to claim 2, characterized in that The action includes a plurality of sub-actions, each sub-action matches a phoneme in the speech, and the timestamp includes the timestamp of each phoneme; and the step of generating and displaying an animation of the digital human performing the action at the time corresponding to the timestamp according to the action information includes: Determining, according to the timestamp of each phoneme, the execution time of the sub-action matching each phoneme; Based on the action information, an animation is generated and displayed in which the digital human performs a sub-action matching each phoneme at the timestamp of each phoneme.

4. The method according to claim 1, characterized in that The animation of the digital human performing the corresponding action is generated according to the action information and displayed on the display device in the vehicle cabin, including: Calling at least one frame of action slice of the digital human corresponding to the action information from the action model library; Each frame of the digital human action slice in the at least one frame of the digital human action slice is sequentially displayed on the display device.

5. The method according to claim 1, characterized in that The state information of the living body includes first state information of the living body, and the acquiring state information of the living body in the vehicle cabin includes: Collect surveillance video of the rear seats in the cabin; Performing live body detection on the monitoring video and performing status analysis on the detected live body to obtain first status information of the live body.

6. The method according to claim 5, characterized in that The monitoring video is acquired by a video acquisition device installed on a rearview mirror in the vehicle cabin with a lens facing the rear row of the vehicle cabin.

7. The method according to claim 5, characterized in that The first state information includes at least one of the following: category information, identity information, attribute information, emotion information, expression information, body movement information, seat seating information, and seat belt wearing information of the living body; and / or, The living body includes at least one of the following: a driver, a co-driver, a child, an elderly person, a pet, and a rear passenger.

8. The method according to claim 1, characterized in that The state information of the living body includes first state information and second state information of the living body, wherein the first state information is obtained based on a surveillance video in the vehicle cabin; The acquiring of the state information of the living body in the vehicle cabin further includes: acquiring second state information sent by a smart device carried by the living body; The determining of action information matching the state information includes: Determine action information that matches both the first state information and the second state information.

9. The method according to claim 1, characterized in that: The obtaining of status information of a living body in the vehicle cabin includes: Inputting the surveillance video in the cabin into a pre-trained neural network; The state information of the living body is determined according to the output result of the neural network.

10. The method according to claim 1, characterized in that The method further comprises: The operating state of the vehicle-mounted device is controlled according to the state information.

11. The method according to any one of claims 1 to 10, characterized in that: The determining of action information matching the state information includes: Get the driving status of the vehicle; Action information that matches the driving state of the vehicle and the state information respectively is determined.

12. A cabin interaction device based on digital human, characterized in that: The device comprises: The acquisition module is used to acquire the state information of the living body in the vehicle cabin; specifically, it includes: determining the object information of each object in the target image including the living body, including the position information of the center point of the object and the object type information corresponding to the center point of the object; A determination module, used to determine action information matching the state information; A display module, used for generating an animation of a digital human performing a corresponding action according to the action information and displaying it on a display device in the vehicle cabin; A recognition module, used to recognize a specific group in the target image based on the position information and object type information of each object recognized, so that the digital human can interact with the specific group; The acquisition module determines the position information of the center point of each object in the target image based on the following method: Performing feature extraction on the target image to obtain a first feature map corresponding to the target image; Obtaining, from a first preset channel of the first feature map, a response value of each feature point in the first feature map as a center point of an object; Divide the first feature map into multiple sub-regions, and determine the maximum response value and the feature point corresponding to the maximum response value in each sub-region; The target feature point whose maximum response value is greater than a preset threshold is taken as the center point of the object, and the position information of the center point of the object is determined based on the position index of the target feature point on the first feature map.

13. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 11 is implemented.

14. A computer device comprising a memory, a processor and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method according to any one of claims 1 to 11 is implemented.

15. A vehicle, characterized in that: The vehicle cabin is provided with a display device, a monitoring system, and a digital human-based cabin interaction device as claimed in claim 12 or a computer device as claimed in claim 14.

Images