Vehicle-mounted mechanical arm control method and device, vehicle-mounted mechanical arm and vehicle

Abstract

The embodiment of the invention provides a vehicle-mounted mechanical arm control method and device, a vehicle-mounted mechanical arm and a vehicle, and is applied to the technical field of intelligent cabins. In the process that the vehicle-mounted mechanical arm executes the nursing operation, the image data of the to-be-nursed object is analyzed, the state information of the to-be-nursed object is determined, and accordingly the corresponding nursing operation prompt information is generated according to the state information. And under the condition that the user carries out operation confirmation, the mechanical arm is controlled to complete the corresponding nursing operation, and therefore man-machine interaction between the vehicle-mounted mechanical arm and the user is achieved. And the vehicle-mounted mechanical arm can actively analyze the image data of the to-be-nursed object in the process of executing the nursing operation, and determine the corresponding nursing operation according to the state information of the to-be-nursed object, so that the vehicle-mounted mechanical arm realizes more intelligent nursing operation, the intelligent degree of the vehicle-mounted mechanical arm is improved, and the nursing efficiency of the vehicle-mounted mechanical arm is improved. Various requirements of a user can be met, and then the user experience is improved.

Description

Technical Field

[0001] This application relates to the technical field of intelligent cockpits, and particularly to a method and device for controlling an in-vehicle robotic arm, an in-vehicle robotic arm, and a vehicle. Background Art

[0002] The emergence of automobiles has provided great convenience for people's travel. It can not only meet people's daily commuting needs but also easily handle long-distance trips, significantly saving time and improving the quality of life, making people's travel more free and flexible.

[0003] In order to improve the user's driving experience, in related technologies, a robotic arm is set in the vehicle to meet the user's needs during driving. However, this robotic arm can only passively execute relevant operations according to the user's instructions, with a low degree of intelligence and unable to meet the user's intelligent needs, thus affecting the user experience. Summary of the Invention

[0004] The purpose of the embodiments of this application is to provide a method and device for controlling an in-vehicle robotic arm, an in-vehicle robotic arm, and a vehicle to improve the user experience. The specific technical solutions are as follows:

[0005] In the first aspect of the embodiments of this application, a method for controlling an in-vehicle robotic arm is provided. The method includes:

[0006] During the process of using the in-vehicle robotic arm to perform a care operation on a care target, obtaining image data of the care target;

[0007] Analyzing the image data to obtain status information of the care target;

[0008] Generating care operation prompt information corresponding to the status information;

[0009] When a confirmation operation for the care operation prompt information is obtained, controlling the in-vehicle robotic arm to perform the care operation corresponding to the care operation prompt information.

[0010] In a possible implementation manner, the care target is an infant. The generating of the care operation prompt information corresponding to the status information includes:

[0011] When the status information indicates that the infant is awake, generating a first care operation prompt information for prompting whether to perform a doll show;

[0012] Using the vehicle control system to control the vehicle display screen to display the first care operation prompt information, and / or controlling the vehicle audio to play the voice of the first care operation prompt information;

[0013] When a confirmation operation for the care operation prompt information is obtained, controlling the on-vehicle robotic arm to perform the care operation corresponding to the care operation prompt information includes:

[0014] When a confirmation operation for the first care operation prompt information is obtained, controlling the on-vehicle robotic arm to move to a first designated position in front of the object to be cared for and perform a preset doll performance action.

[0015] In a possible implementation manner, when the object to be cared for is an infant, generating the care operation prompt information corresponding to the status information includes:

[0016] When the status information indicates that the infant is sleepy, generating a second care operation prompt information for prompting whether to perform infant falling asleep care;

[0017] Using the vehicle control system to control the vehicle display screen to display the second care operation prompt information, and / or controlling the vehicle audio to play the voice of the second care operation prompt information;

[0018] When a confirmation operation for the care operation prompt information is obtained, controlling the on-vehicle robotic arm to perform the care operation corresponding to the care operation prompt information includes:

[0019] When a confirmation operation for the second care operation prompt information is obtained, controlling the on-vehicle robotic arm to perform the action of covering the quilt for the object to be cared for;

[0020] The method further includes:

[0021] Using the control system to adjust the vehicle lights to the sleep light intensity, and / or adjusting the temperature of the rear row air conditioner of the vehicle to the sleep temperature, and / or controlling the vehicle audio to play a preset lullaby audio.

[0022] In a possible implementation manner, when the object to be cared for is an infant, generating the care operation prompt information corresponding to the status information includes:

[0023] When the status information indicates that the infant is fast asleep, generating a third care operation prompt information for prompting whether to perform infant sleep care;

[0024] Using the vehicle control system to control the vehicle display screen to display the third care operation prompt information, and / or controlling the vehicle audio to play the voice of the third care operation prompt information;

[0025] When a confirmation operation for the care operation prompt information is obtained, controlling the on-vehicle robotic arm to perform the care operation corresponding to the care operation prompt information includes:

[0026] When a confirmation operation for the third care operation prompt information is obtained, control the vehicle-mounted robotic arm to move to a second designated position on the side of the object to be cared for. When the object to be cared for has an action of slipping out of the baby chair, control the vehicle-mounted robotic arm to perform a supporting operation on the object to be cared for;

[0027] The method further includes:

[0028] Use the control system to adjust the volume of the vehicle audio to off,

[0029] and / or,

[0030] Obtain the motion condition of the vehicle, calculate a target damping coefficient according to the motion condition, and use the control system to set the damping coefficient of the seat damper of the target vehicle seat to the target damping coefficient, where the baby chair is installed on the target vehicle seat.

[0031] In a possible implementation manner, the object to be cared for is an infant or toddler. Generating the care operation prompt information corresponding to the status information includes:

[0032] When the status information indicates that the infant or toddler is struggling to get out of the seat, control the vehicle-mounted robotic arm to perform a supporting operation on the object to be cared for, and generate a fourth care operation prompt information for prompting to pull over to the side of the road.

[0033] Use the control system of the vehicle to control the vehicle display screen to display the fourth care operation prompt information, and / or control the vehicle audio to play the voice of the fourth care operation prompt information.

[0034] In a possible implementation manner, before obtaining the image data of the object to be cared for during the process of using the vehicle-mounted robotic arm to perform a care operation on the object to be cared for, the method further includes:

[0035] In response to obtaining the first voice control instruction indicating entering the care mode, enter the care mode so that the vehicle-mounted robotic arm performs a care operation on the object to be cared for;

[0036] Or,

[0037] In response to a first operation control instruction indicating entering the care mode input by the user through the human-machine interface on the vehicle display screen, enter the care mode so that the vehicle-mounted robotic arm performs a care operation on the object to be cared for.

[0038] In a possible implementation manner, the method further includes:

[0039] In response to receiving a second voice control instruction indicating to pick up a first object at a first position, control the on-vehicle robotic arm to move to the first position information;

[0040] Obtain first image data at the first position, perform target recognition on the first image data at the first position to obtain a first image position of the first object; convert the first image position into a vehicle coordinate system to obtain a first three-dimensional position; control the on-vehicle robotic arm to perform a grasping action on the first three-dimensional position to grasp the first object.

[0041] In a possible implementation manner, the method further includes:

[0042] In response to receiving a third voice control instruction indicating to pick up a second object, obtain an in-vehicle image;

[0043] Perform target recognition on the in-vehicle image to determine whether the in-vehicle image includes the second object;

[0044] If so, determine a second image position of the second object according to the in-vehicle image; convert the second image position into a vehicle coordinate system to obtain a second three-dimensional position; control the on-vehicle robotic arm to perform a grasping action on the second three-dimensional position to grasp the second object;

[0045] If not, generate position inquiry information for the second object;

[0046] Use the vehicle control system to control the vehicle display screen to display the position inquiry information, and / or control the vehicle audio to play the position inquiry information;

[0047] Obtain a second position where the second object is located input by the user, obtain second image data at the second position, perform target recognition on the second image data to obtain a second image position of the second object; convert the second image position into a vehicle coordinate system to obtain a second three-dimensional position; control the on-vehicle robotic arm to perform a grasping action on the second three-dimensional position to grasp the second object.

[0048] In a second aspect of the embodiments of the present application, there is provided an on-vehicle robotic arm control device, and the device includes:

[0049] An image data acquisition module, configured to acquire image data of a to-be-cared-for object during the process of using the on-vehicle robotic arm to perform a care operation on the to-be-cared-for object;

[0050] A status information determination module, configured to analyze the image data to obtain status information of the to-be-cared-for object;

[0051] A prompt information generation module, configured to generate a care operation prompt information corresponding to the status information;

[0052] A nursing operation control module, configured to control the on-vehicle robotic arm to perform a nursing operation corresponding to the nursing operation prompt information when a confirmation operation for the nursing operation prompt information is obtained.

[0053] In a possible implementation manner, the object to be nursed is an infant, and the prompt information generation module includes:

[0054] A first prompt information generation sub-module, specifically configured to generate a first nursing operation prompt information for prompting whether to perform a doll performance when the status information indicates that the infant is awake;

[0055] A first prompt control sub-module, specifically configured to control the vehicle display screen to display the first nursing operation prompt information by using the vehicle control system, and / or control the vehicle audio to play the voice of the first nursing operation prompt information;

[0056] The nursing operation control module includes:

[0057] A first nursing operation control sub-module, specifically configured to control the on-vehicle robotic arm to move to a first designated position in front of the object to be nursed and perform a preset doll performance action when a confirmation operation for the first nursing operation prompt information is obtained.

[0058] In a possible implementation manner, the object to be nursed is an infant, and the prompt information generation module includes:

[0059] A second prompt information generation sub-module, specifically configured to generate a second nursing operation prompt information for prompting whether to perform infant sleep nursing when the status information indicates that the infant is sleepy;

[0060] A second prompt control sub-module, specifically configured to control the vehicle display screen to display the second nursing operation prompt information by using the vehicle control system, and / or control the vehicle audio to play the voice of the second nursing operation prompt information;

[0061] The nursing operation control module includes:

[0062] A second nursing operation control sub-module, specifically configured to control the on-vehicle robotic arm to perform a quilt covering action for the object to be nursed when a confirmation operation for the second nursing operation prompt information is obtained.

[0063] The device further includes:

[0064] A sleep environment adjustment module, which is used to adjust the vehicle's lights to the sleep light intensity by using the control system, and / or adjust the temperature of the rear-row air conditioner of the vehicle to the sleep temperature, and / or control the vehicle audio to play a preset lullaby audio.

[0065] In a possible implementation manner, the object to be cared for is an infant, and the prompt information generation module includes:

[0066] A third prompt information generation sub-module, which is specifically used to generate a third care operation prompt information for prompting whether to perform infant sleep care when the status information indicates that the infant is in a deep sleep;

[0067] A third prompt control sub-module, which is specifically used to control the vehicle display screen to display the third care operation prompt information by using the vehicle's control system, and / or control the vehicle audio to play the voice of the third care operation prompt information;

[0068] The care operation control module includes:

[0069] A third care operation control sub-module, which is specifically used to control the on-vehicle robotic arm to move to a second specified position on the side of the object to be cared for when an acknowledgement operation for the third care operation prompt information is obtained, and control the on-vehicle robotic arm to perform a support operation for the object to be cared for when the object to be cared for has an action of slipping out of the baby chair;

[0070] The device further includes:

[0071] A vehicle parameter adjustment module, which is used to adjust the volume of the vehicle audio to be turned off by using the control system, and / or obtain the movement condition of the vehicle, calculate a target damping coefficient according to the movement condition, and use the control system to set the damping coefficient of the seat damper of the target vehicle seat to the target damping coefficient, where the baby chair is installed on the target vehicle seat.

[0072] In a possible implementation manner, the object to be cared for is an infant, and the prompt information generation module includes:

[0073] A fourth prompt information generation sub-module, which is specifically used to control the on-vehicle robotic arm to perform a support operation for the cared-for object when the status information indicates that the infant is struggling to get out of the seat, and generate a fourth care operation prompt information for prompting to pull over;

[0074] A fourth prompt control sub-module, which is specifically used to control the vehicle display screen to display the fourth care operation prompt information by using the vehicle's control system, and / or control the vehicle audio to play the voice of the fourth care operation prompt information.

[0075] In a possible implementation, the device further includes:

[0076] A first determination module for the care mode, configured to enter the care mode in response to obtaining a first voice control instruction indicating entering the care mode, so that the on-vehicle robotic arm performs a care operation on the object to be cared for;

[0077] Or,

[0078] A second determination module for the care mode, configured to enter the care mode in response to a first operation control instruction indicating entering the care mode input by the user through the human-machine interface on the vehicle display screen, so that the on-vehicle robotic arm performs a care operation on the object to be cared for.

[0079] In a possible implementation, the device further includes:

[0080] A first movement module for the on-vehicle robotic arm, configured to control the on-vehicle robotic arm to move to the first position in response to receiving a second voice control instruction indicating picking up a first object at the first position;

[0081] A first execution module for the grasping action, which acquires first image data at the first position, performs target recognition on the first image data at the first position to obtain a first image position of the first object; converts the first image position into the vehicle coordinate system to obtain a first three-dimensional position; and controls the on-vehicle robotic arm to perform a grasping action on the first three-dimensional position to grasp the first object.

[0082] In a possible implementation, the device further includes:

[0083] An in-vehicle image acquisition module, configured to acquire an in-vehicle image in response to receiving a third voice control instruction indicating picking up a second object;

[0084] A target object determination module, configured to perform target recognition on the in-vehicle image to determine whether the second object is included in the in-vehicle image;

[0085] A second execution module for the grasping action, configured to, if so, determine a second image position of the second object according to the in-vehicle image, convert the second image position into the vehicle coordinate system to obtain a second three-dimensional position; and control the on-vehicle robotic arm to perform a grasping action on the second three-dimensional position to grasp the second object;

[0086] A position inquiry information generation module, configured to, if not, generate position inquiry information for the second object;

[0087] A position information inquiry module, configured to control the vehicle display screen to display the position inquiry information by using the vehicle control system, and / or control the vehicle audio to play the position inquiry information;

[0088] The third execution module for the grasping action is configured to obtain the second position where the second object is located as input by the user, obtain the second image data at the second position, perform target recognition on the second image data to obtain the second image position of the second object; convert the second image position into the vehicle coordinate system to obtain the second three-dimensional position; and control the on-vehicle robotic arm to perform a grasping action on the second three-dimensional position to grasp the second object.

[0089] In the third aspect of the embodiments of the present application, an on-vehicle robotic arm is provided, and the on-vehicle robotic arm is configured to execute any one of the on-vehicle robotic arm control methods in the first aspect of the embodiments of the present application.

[0090] In the fourth aspect of the embodiments of the present application, a vehicle is provided, and the vehicle includes the on-vehicle robotic arm described in the third aspect of the embodiments of the present application, and the on-vehicle robotic arm is configured to execute any one of the on-vehicle robotic arm control methods in the third aspect of the embodiments of the present application.

[0091] An on-vehicle robotic arm control method, device, on-vehicle robotic arm and vehicle provided by the embodiments of the present application can, during the process of the on-vehicle robotic arm performing a care operation, analyze the image data of the object to be cared for to determine the status information of the object to be cared for, and thus generate corresponding care operation prompt information according to the status information. And in the case where the user performs an operation confirmation, control the robotic arm to complete the corresponding care operation, thereby realizing the human-machine interaction between the on-vehicle robotic arm and the user. Moreover, the on-vehicle robotic arm can actively analyze the image data of the object to be cared for during the process of performing the care operation, and determine the corresponding care operation based on the status information of the object to be cared for, so that the on-vehicle robotic arm realizes a more intelligent care operation, improves the intelligent level of the on-vehicle robotic arm, can meet various needs of the user, and further improves the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0092] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings required for use in the description of the embodiments or the prior art.

[0093] Figure 1 The first flowchart of the on-vehicle robotic arm control method provided by the embodiments of the present application;

[0094] Figure 2 The second flowchart of the on-vehicle robotic arm control method provided by the embodiments of the present application;

[0095] Figure 3 The third flowchart of the on-vehicle robotic arm control method provided by the embodiments of the present application;

[0096] Figure 4The fourth flowchart of the on-vehicle robotic arm control method provided by the embodiment of the present application;

[0097] Figure 5 A flowchart for refining step S103 provided by the embodiment of the present application;

[0098] Figure 6 A flowchart for the on-vehicle robotic arm to achieve the grasping function provided by the embodiment of the present application;

[0099] Figure 7 Another flowchart for the on-vehicle robotic arm to achieve the grasping function provided by the embodiment of the present application;

[0100] Figure 8 A structural schematic diagram of the on-vehicle robotic arm control device provided by the embodiment of the present application. Detailed implementation manners

[0101] Next, the technical solutions in the embodiments of the present application will be described with reference to the accompanying drawings in the embodiments of the present application.

[0102] The emergence of automobiles has provided great convenience for people's travel. It can not only meet people's daily commuting needs, but also easily handle long-distance trips, significantly saving time and improving the quality of life, making people's travel more free and flexible.

[0103] In order to improve the driving experience of users, in the related art, a robotic arm is added to the vehicle to meet the needs of users during driving. However, this robotic arm can only passively execute relevant operations according to the instructions of users, with a low degree of intelligence and unable to meet the intelligent needs of users, thus affecting the user experience.

[0104] For example, in one case, if a driver drives a vehicle alone with an infant or toddler, it is inevitable that the infant or toddler needs to be taken care of during driving, then the driver cannot spare the energy to take care of the infant or toddler in time, resulting in inconvenience during driving.

[0105] To solve at least one of the above problems, in the first aspect of the embodiments of the present application, an on-vehicle robotic arm control method is provided, and the method includes the following Figure 1 shown steps:

[0106] Step S101: During the process of using the on-vehicle robotic arm to perform a care operation on the object to be cared for, obtain the image data of the object to be cared for.

[0107] Among them, the in-vehicle robotic arm is a robotic arm installed inside the vehicle and can have the characteristics of a flexible robotic arm, so as to be able to complete fine movements. For example, the in-vehicle robotic arm can perform the action of extracting tissues, so as to facilitate delivering tissues to the user when needed. In practical applications, the installation position of the in-vehicle robotic arm can also be set according to the user's needs. For example, the in-vehicle robotic arm can determine the installation position according to the position of the child safety seat, so that the in-vehicle robotic arm can be set at a position convenient for taking care of infants and young children. In another example, the in-vehicle robotic arm can be foldable, and when the in-vehicle robotic arm is not needed, it can be stored in the vehicle body.

[0108] In practical applications, the object to be cared for is usually an infant or young child, but the specific object to be cared for can also be determined according to actual needs. For example, the object to be cared for can also be an elderly person, a patient, or even a pet, etc., who needs care or may affect driving safety. When the object to be cared for is an infant or young child, the outside of the in-vehicle robotic arm can be wrapped with materials that can be chewed by the baby, so as to prevent other materials from affecting the health of the infant or young child.

[0109] The image data of the object to be cared for can be obtained through a camera installed in the vehicle, or through a camera installed on the in-vehicle robotic arm. Among them, the obtained image data can be a visible light image or a grayscale image.

[0110] Step S102: Analyze the image data to obtain the status information of the object to be cared for.

[0111] Among them, the status information of the object to be cared for can be one or more status information indicating the physiological activities or psychological activities of the object to be cared for, and this status information can be determined according to the functions to be realized by the in-vehicle robotic arm. For example, if the in-vehicle robotic arm can realize the care of infants and young children in multiple different states such as awake, sleepy, sound asleep, crying, laughing, breaking free from the seat, etc. under the care function, these different states can be set as status information respectively.

[0112] Use computer vision technology to analyze the image data, so as to obtain the status information of the object to be cared for. In one example, the object to be cared for can be obtained by first using a target recognition network to perform target recognition on the image data. Then, use a long short-term memory neural network to extract the image features of the object to be cared for in multiple frames of image data before and after, and obtain the image features containing temporal information. Finally, use a classification network to classify the image features containing temporal information, so as to obtain the status information (classification result) of the object to be cared for. It can be understood that the process of determining the status information based on computer vision technology (for example, the awake, sleepy, sound asleep, crying, laughing, breaking free from the seat, etc. of infants and young children) is a prior art, and this application does not specifically limit this process.

[0113] Step S103: Generate a nursing operation prompt message corresponding to the status information.

[0114] Among them, the nursing operation prompt message can be pre-set or generated instantaneously. In one example, for each status information, several corresponding prompt messages can be pre-set. After determining the status information, one of them can be randomly selected or selected in sequence from the several corresponding prompt messages according to the label of the status information as the nursing operation prompt message. For example, for the status information of "drowsy", prompt messages such as "The baby is dozing off" and "The baby is about to fall asleep" can be pre-set. When analyzing the image data of the object to be nursed and obtaining that the status information of the object to be nursed indicates "drowsy", one of the above two prompt messages can be randomly selected as the nursing operation prompt message.

[0115] In another example, a large language model can be embedded in the intelligent cockpit system of the vehicle. After obtaining the status information, the large language model can instantaneously generate a nursing operation prompt message for prompting the driver for this status information.

[0116] Step S104: When a confirmation operation for the nursing operation prompt message is obtained, control the vehicle-mounted robotic arm to perform the nursing operation corresponding to the nursing operation prompt message.

[0117] In practical applications, after generating the nursing operation prompt message, the user can be reminded by means of voice broadcast or pop-up prompt for this prompt message, so that the user can perform corresponding operations, and this operation is used to indicate whether the vehicle-mounted robotic arm needs to further perform the nursing operation.

[0118] There can also be various ways for the user to confirm the nursing operation prompt message. For example, it can be confirmed through the shortcut key settings in the vehicle, or through voice commands, or through relevant operations on the vehicle's display screen. In another example, after the nursing operation prompt, it can also be judged whether the user has a cancellation operation within a preset time range (such as 5s, 8s or 10s). If not, it is default that the user has performed the confirmation operation.

[0119] The nursing operations performed by the vehicle-mounted robotic arm for different nursing operation prompt messages can be pre-debugged. In one example, for the nursing operation prompt message corresponding to the status information of "crying loudly", when a confirmation operation is received, the vehicle-mounted robotic arm can be controlled to perform the action of hugging the infant and gently shaking it to achieve the nursing purpose of soothing the infant. In another example, for the nursing operation prompt message corresponding to the status information of "drowsy", when a confirmation operation is received, the vehicle-mounted robotic arm can be controlled to perform the action of covering the quilt to achieve the nursing purpose of soothing the infant to sleep.

[0120] By applying the method of the embodiments of the present application, during the care operation of the on-vehicle robotic arm, the status information of the object to be cared for can be determined by analyzing the image data of the object to be cared for, and then the corresponding care operation prompt information can be generated according to the status information. And in the case of user operation confirmation, the robotic arm is controlled to complete the corresponding care operation, thus realizing the human-machine interaction between the on-vehicle robotic arm and the user. Moreover, during the care operation, the on-vehicle robotic arm can actively analyze the image data of the object to be cared for and determine the corresponding care operation based on the status information of the object to be cared for, so that the on-vehicle robotic arm realizes a more intelligent care operation, improves the intelligence level of the on-vehicle robotic arm, can meet various needs of users, and further improves the user experience.

[0121] In a possible implementation manner, the object to be cared for is an infant, such as Figure 2 shown, step S103 can be implemented through the following steps:

[0122] Step S201: When the status information indicates that the infant is awake, generate the first care operation prompt information for prompting whether to perform a doll show.

[0123] In practical applications, in the waking state of the infant, early education can be used to soothe its emotions and prevent it from having intense emotions or other actions during vehicle driving that may affect the driver. Among them, the early education content can be realized through a doll show. Similarly, the first care operation prompt information can be pre-set prompt content or prompt content generated in real time by a large language model.

[0124] Step S202: Use the vehicle control system to control the vehicle display screen to display the first care operation prompt information, and / or control the vehicle audio to play the voice of the first care operation prompt information.

[0125] Among them, the system for controlling the on-vehicle robotic arm can be integrated into the vehicle control system or independent of the vehicle control system, and realizes the care function by interacting with the vehicle control system. In one example, if the system for controlling the on-vehicle robotic arm is integrated into the vehicle control system, after generating the first care operation prompt information, the vehicle control system can directly control the vehicle display screen to display the first care operation prompt information, and / or control the vehicle audio to play the voice of the first care operation prompt information.

[0126] In another example, if the system for controlling the on-vehicle robotic arm and the control system for controlling the vehicle are two independent systems, then after the on-vehicle robotic arm control system generates the first care operation prompt information, it sends this information to the vehicle control system. After receiving the first care operation prompt information, the vehicle control system controls the vehicle display screen to display the first care operation prompt information, and / or controls the vehicle audio to play the voice of the first care operation prompt information.

[0127] When performing care operation prompts through the vehicle display screen, it can be displayed on the vehicle display screen in the form of a pop-up window. The vehicle display screen can display the specific content of the first care operation prompt information, or can also display the image information of the object to be cared for collected by the camera.

[0128] Step S104 can be implemented through the following steps:

[0129] Step S203: In the case of obtaining a confirmation operation for the first care operation prompt information, control the on-vehicle robotic arm to move to the first specified position in front of the object to be cared for, and perform a preset doll performance action.

[0130] Similar to the above embodiment, the user can confirm the first care operation prompt information by operating the screen, shortcut keys, or voice commands. Among them, the first specified position can be a certain fixed position set in advance, or a position calculated in real time according to the position of the object to be cared for. In one example, it can be set that the position 30 cm in front of the child safety seat and 50 cm from the roof of the vehicle is the first specified position. In another example, image processing can be performed on the image of the object to be cared for, calculate the distance from the object to be cared for to the camera, and determine the first specified position according to this distance.

[0131] The doll performance action can be pre-adjusted according to early education content. The user only needs to place the doll at the specified position of the on-vehicle robotic arm or the vehicle in advance. In actual applications, the user can also replace the doll according to their own needs. When the doll is not placed at the specified position of the on-vehicle robotic arm, before the on-vehicle robotic arm performs the doll performance, it can first reach the doll position to perform a grasping action, and then reach the first specified position to perform the preset doll performance action.

[0132] In one example, since the on-vehicle robotic arm has the characteristics of a flexible robotic arm and can complete fine movements, when demonstrating early education content, it can directly rely on the finger part of the on-vehicle robotic arm to conduct early education through gesture performances or shadow puppetry without the need for doll grasping. Among them, the content of the gesture performance can be composed of multiple pre-adjusted gesture actions.

[0133] In another example, when performing a shadow puppetry show, a light-emitting device can also be arranged on the vehicle-mounted robotic arm in cooperation, and a projection device such as a curtain can also be arranged on the vehicle. When a confirmation operation for the first care operation prompt information is obtained, the vehicle-mounted robotic arm is controlled to move to the first designated position, and the vehicle control system is used to control the curtain to appear in front of the infant by rising or falling. At this time, the first designated position needs to ensure that the finger part of the vehicle-mounted robotic arm is located between the light-emitting device and the curtain, so as to ensure that the hand projection of the vehicle-mounted robotic arm can be projected on the curtain for the infant to watch.

[0134] When the vehicle-mounted robotic arm performs a doll performance action, the control system of the vehicle can also control the vehicle audio or the audio on the vehicle-mounted robotic arm to play early education-related content or music, so as to cooperate with the doll performance to achieve the purpose of early education.

[0135] In practical applications, the vehicle-mounted robotic arm can also be equipped with a display screen, and when conducting early education for infants, the early education content can be directly played through the display screen. For example, when a confirmation operation for the first care operation prompt information is obtained, the vehicle-mounted robotic arm is controlled to move so that the display screen is at the first designated position in front of the infant, and a preset early education video is played.

[0136] By applying the method of the embodiment of the present application, when the vehicle-mounted robotic arm recognizes that the infant is in a waking state, it can generate the first care operation prompt information to prompt the user, and when the user confirms, it can perform the doll performance function for the infant to realize the early education of the infant, so that the vehicle-mounted robotic arm can help the user achieve a more intelligent care operation and improve the user experience.

[0137] In a possible implementation manner, the object to be cared for is an infant, such as Figure 3 shown, step S103 can be implemented through the following steps:

[0138] Step S301: When the status information indicates that the infant is sleepy, generate a second care operation prompt information for prompting whether to perform infant sleep care.

[0139] In the state of the infant being sleepy, the infant sleep care can be to change parameters such as the temperature inside the vehicle and the volume inside the vehicle to achieve the purpose of facilitating the infant to fall asleep. Similarly, the second care operation prompt information can be pre-set prompt content or prompt content generated in real time through a large language model.

[0140] Step S302: Use the control system of the vehicle to control the vehicle display screen to display the second care operation prompt information, and / or control the vehicle audio to play the voice of the second care operation prompt information.

[0141] Among them, the control method of controlling the vehicle display screen to display the second care operation prompt information and / or controlling the vehicle audio to play the voice of the second care operation prompt information is the same as that in the above embodiments, and can be directly implemented through the integrated vehicle control system, or can be implemented through the interaction between the independent on-vehicle robotic arm control system and the vehicle control system, which will not be elaborated here.

[0142] Step S104 can be implemented through the following steps:

[0143] Step S303: When an acknowledgment operation for the second care operation prompt information is obtained, control the on-vehicle robotic arm to perform the action of covering the quilt for the object to be cared for.

[0144] Similar to the above embodiments, the user can confirm the second care operation prompt information by operating the screen, shortcut keys, or voice commands. The action of covering the quilt can be pre-debugged. When the on-vehicle robotic arm performs the action of covering the quilt, it directly completes the quilt covering instruction according to the pre-debugged action. Similarly, when performing the quilt covering operation, the on-vehicle robotic arm can first obtain the position of the quilt, move to the position of the quilt to perform the grasping action to obtain the quilt, and then move to the object to be cared for to perform the quilt covering operation.

[0145] The method of the embodiment of the present application may further include:

[0146] Step S304: Use the control system to adjust the vehicle lights to the sleep light intensity, and / or adjust the temperature of the rear row air conditioner of the vehicle to the sleep temperature, and / or control the vehicle audio to play the preset lullaby audio.

[0147] Among them, the sleep light intensity and sleep temperature can be preset default values, or values set by the user according to their own needs during use. In one example, when it is determined that the infant is in a drowsy state, the vehicle reminds the user in the form of voice broadcast, and the user confirms through voice commands so that the on-vehicle robotic arm continues to perform relevant care operations. After the user confirms, the on-vehicle robotic arm covers the quilt on the infant according to the pre-debugged quilt covering action. At the same time, it controls the lights in the rear row of the vehicle to be turned off, and the audio plays soothing lullabies at a low volume; and controls the temperature of the rear row air conditioner of the vehicle to be adjusted within the range of 20-25 degrees Celsius.

[0148] Applying the method of the embodiment of the present application, the on-vehicle robotic arm can generate the second care operation prompt information to prompt the user when it recognizes that the infant is in a drowsy state, and when the user confirms, perform the quilt covering operation for the infant and adjust the vehicle interior environment to help the user take care of the infant to fall asleep, so that the on-vehicle robotic arm can help the user achieve more intelligent care operations and improve the user experience.

[0149] In a possible implementation, the object to be cared for is an infant. For example, Figure 4 as shown, step S103 can be implemented through the following steps:

[0150] Step S401: When the status information indicates that the infant is in a deep sleep, generate a third care operation prompt message for prompting whether to enter the infant sleep care.

[0151] Similarly, the third care operation prompt message can be pre-set prompt content or prompt content generated in real time by a large language model. In practical applications, in addition to prompting whether to enter the infant sleep care, the third care operation prompt message can also be used to prompt changes in other environmental parameters or driving parameters in the vehicle to ensure that the vehicle's driving and the in-vehicle environment do not affect the infant's sleep. In one example, the third care operation prompt message can also be used to prompt whether to adjust the in-vehicle volume, the seat damping coefficient, the vehicle driving speed or acceleration limit, etc.

[0152] Step S402: Use the vehicle control system to control the vehicle display screen to display the third care operation prompt message, and / or control the vehicle audio to play the voice of the third care operation prompt message.

[0153] Among them, the control method of controlling the vehicle display screen to display the third care operation prompt message, and / or controlling the vehicle audio to play the voice of the third care operation prompt message is the same as that in the above embodiment. It can be directly implemented through an integrated vehicle control system, or through the interaction between an independent on-vehicle robotic arm control system and the vehicle control system. Details are not described here again.

[0154] Step S104 can be implemented through the following steps:

[0155] Step S403: In the case of obtaining a confirmation operation for the third care operation prompt message, control the on-vehicle robotic arm to move to the second designated position on the side of the object to be cared for. When the object to be cared for has an action of slipping out of the baby chair, control the on-vehicle robotic arm to perform a supporting operation on the object to be cared for.

[0156] In practical applications, the second designated position can be determined according to the length of the on-vehicle robotic arm during the supporting operation to ensure that the infant is within the supporting distance range of the on-vehicle robotic arm; it can also be artificially specified according to the actual application scenario; it can also be determined according to the position of the camera on the on-vehicle robotic arm to ensure that the on-vehicle robotic arm can observe the status of the infant through the camera at the second designated position.

[0157] Among them, the baby chair in the vehicle can ensure the safe riding of infants and young children in most cases, but it does not rule out the situation where infants and young children slide off or break free from the baby chair. In some actual cases, due to the restraint of the safety belt on infants and young children, they may have a resistant psychology towards the baby chair, so they will make corresponding struggling and escaping movements. During this process, infants and young children may accidentally touch the safety buckle and loosen it, enabling them to leave the baby chair. In other actual cases, improper operations by guardians may also cause the baby chair to be unable to perfectly restrain infants and young children during the ride. In addition, when infants and young children are wearing a lot of clothes, for example, when an infant is wearing a down jacket, due to its fluffy characteristics, there will be a gap between the infant and the baby chair, and they cannot fit perfectly with the baby chair. In this case, when the vehicle brakes or turns, due to inertia, there is a risk that the infant will slide off the baby chair from the gap. In this situation, since the in-vehicle robotic arm has the characteristics of a flexible robotic arm and can complete fine movements, the assisting operation of the in-vehicle robotic arm can be completed through the finger part of the in-vehicle robotic arm to achieve specific assisting steps; it can also be completed only relying on the position of the arm to support the infant and prevent it from sliding down from the seat continuously.

[0158] The method of the embodiment of the present application may further include:

[0159] Step S404: Use the control system to adjust the volume of the vehicle audio to off, and / or, obtain the motion situation of the vehicle, calculate the target damping coefficient according to the motion situation, and use the control system to set the damping coefficient of the seat damper of the target vehicle seat to the target damping coefficient.

[0160] Among them, the baby chair is installed on the target vehicle seat. In practical applications, there is no sequential order between the actions of the robotic arm performing the assisting operation in step S404 and step S403, and they can be carried out simultaneously or sequentially. When the control system adjusts the volume of the vehicle audio to off, it can turn off the volume of one or more vehicle audio according to the actual application scenario. In one example, during the vehicle driving process, the navigation sound is played through the audio at the driver's seat headrest of the vehicle. Then, when the infant is sleeping soundly, the vehicle control system can adjust the volume of other audio except the headrest audio to off. For the headrest audio, the vehicle control system can adjust the volume of the headrest audio to a volume that can only be heard by the driver.

[0161] During the vehicle driving process, the vibrations generated by the engine operation, road surface bumps, or the imbalance of the drive shaft, etc., will be transmitted to the vehicle seat through structures such as the vehicle frame, body metal structure, and suspension, causing the vehicle passengers to feel bumps according to the vehicle driving conditions. In order to reduce the bumps felt by the passengers, in one implementation, the vibration amplitude and frequency can be reduced through the seat damper set in the vehicle to improve the riding stability and comfort of users.

[0162] Among them, the seat damper can generate a damping force through an internal mechanical structure or medium (such as liquid, gas, viscoelastic material), absorb and consume the kinetic energy generated by the seat due to vibration or impact, and convert it into heat energy or other non-mechanical energy, thereby improving the vibration damping performance of the seat. In one example, for a fluid damper, frictional heat can be generated by the liquid passing through the throttle hole on the piston, thereby consuming the kinetic energy generated by the seat vibration. In addition, through the resistance generated by the seat damper, a force opposite to the movement direction of the seat can be applied to the seat, thereby delaying the transmission speed of the seat vibration.

[0163] The seat in the vehicle of the present application is equipped with a seat damper with adjustable damping coefficient. When taking care of an infant during sleep, the damping coefficient of the seat damper of the target vehicle seat can be adjusted according to the real-time movement condition of the vehicle. Among them, the target vehicle seat is the seat on which the baby chair is installed, and the infant is located on the baby chair. Specifically, the weight of the infant can be obtained through a pressure sensor and a vibration sensor in the target vehicle seat, etc., combined with the movement of the vehicle, parameter processing and calculation are carried out to obtain the target damping coefficient, and by adjusting the damping coefficient of the seat damper to the target damping coefficient, the bumpy feeling of the infant can be reduced. The specific calculation process of the target damping coefficient can refer to the prior art and is not specifically limited in the present application.

[0164] In one example, the seat damper can be a liquid damper. In this case, when a CDC (Continuous Damping Control) system is installed in the vehicle, the system can perform parameter processing and calculation based on the data of body sensors (such as acceleration, height sensors), and dynamically adjust the opening of the solenoid valve in the seat damper to control the amount of liquid used by the seat damper for shock absorption, so as to achieve continuous adjustment of the damping coefficient. In another example, the seat damper can be a magnetorheological damper. In this case, by performing parameter processing and calculation based on the data of body sensors (such as acceleration, height sensors), the current intensity can be dynamically adjusted to change the viscosity change of the magnetorheological fluid in the magnetic field, thereby realizing the adjustment of the damping coefficient.

[0165] After setting the damping coefficient of the seat damper to the target damping coefficient, the vibration of the seat can be greatly reduced; and since the baby chair is installed on the seat of the target vehicle, and the baby chair includes a large amount of soft materials such as sponge and cotton, the vibration of the baby chair can be further reduced, so that the infant on the baby chair can be in a relatively stable sleep environment.

[0166] When the method of the embodiment of the present application is applied, the on-vehicle robotic arm can generate a third care operation prompt message to prompt the user when it recognizes that the infant is in a deep sleep state. With the user's confirmation, the damping coefficient of the seat damper of the target vehicle seat can be set to the target damping coefficient, and a supporting operation for the infant can be performed when the infant gets out of the seat, ensuring that the infant's sleep is relatively stable. Thus, the on-vehicle robotic arm can help the user achieve a more intelligent care operation and improve the user experience.

[0167] In a possible implementation manner, the object to be cared for is an infant. For example Figure 5 As shown, step S103 can be implemented through the following steps:

[0168] Step S501: When the status information indicates that the infant is struggling to get out of the seat, control the on-vehicle robotic arm to perform a supporting operation on the care object and generate a fourth care operation prompt message for prompting to pull over.

[0169] Step S502: Use the vehicle control system to control the vehicle display screen to display the fourth care operation prompt message, and / or control the vehicle audio to play the voice of the fourth care operation prompt message.

[0170] Similar to the above embodiment, the fourth care operation prompt message can be pre-set prompt content or prompt content generated in real time by a large language model. The way for the on-vehicle robotic arm to implement the supporting operation can also be the same as that in the above embodiment. In practical applications, when the infant is in a state of struggling to get out of the seat, the on-vehicle robotic arm can directly perform the supporting operation without waiting for the user's confirmation operation to ensure the safety of the infant.

[0171] In one example, when it is recognized that the infant is struggling to get out of the seat, the on-vehicle robotic arm directly performs a supporting operation on the infant, and at the same time generates a fourth care operation prompt message to prompt the user to pull over. After receiving the fourth care operation prompt message, the user can choose whether to pull over according to the actual driving environment and comfort the infant to sit back on the child safety seat.

[0172] When the method of the embodiment of the present application is applied, the on-vehicle robotic arm can perform a supporting operation when it recognizes that the infant is struggling to get out of the seat to ensure the riding safety of the infant, and generate a fourth care operation prompt message to prompt the user. Thus, the on-vehicle robotic arm can help the user achieve a more intelligent care operation and improve the user experience.

[0173] In a possible implementation, before acquiring the image data of the object to be cared for during the process of using the vehicle-mounted robotic arm to perform care operations on the object to be cared for, the method of the embodiments of the present application may further include the following steps: in response to acquiring a first voice control instruction indicating entering the care mode, enter the care mode, so that the vehicle-mounted robotic arm performs care operations on the object to be cared for.

[0174] Among them, the first voice control instruction can be obtained through voice recognition processing by a first neural network model installed in the vehicle. Among them, the first neural network model can be a network model used for voice recognition, natural language processing, etc. in the related art. In one example, the first neural network model can perform multi-modal fusion of multiple network models to achieve voice recognition and voice interaction functions. For example, the first neural network model can include a voice recognition module, a natural language processing module, and a voice synthesis module. Among them, the voice recognition module can include at least one of voice recognition models such as Whisper (a voice recognition model developed by OpenAI), Google Cloud Speech-to-Text (a voice recognition model that converts audio into text), and Microsoft Azure Speech-to-Text (a speech-to-text tool).

[0175] The natural language processing module can include at least one of language processing models such as ChatGPT (Chat Generative Pre-trained Transformer), DeepSeek, and Doubao. The voice synthesis module can include at least one of voice generation models such as WaveNet (an audio waveform generation network) and Voicebox (a generative voice model).

[0176] The voice recognition model in the first neural network model acquires the user's voice, extracts features from the acquired voice, and obtains the voice text corresponding to the user's voice; the natural language processing module performs feature processing and segmentation processing on the voice text, obtains multiple word sequences, performs context modeling through the self-attention mechanism in the natural language processing module, and generates the dialogue text corresponding to the voice text; finally, the voice synthesis module generates the audio waveform of the dialogue text to obtain the dialogue audio wave, so as to play the dialogue audio wave through the vehicle's speaker to achieve interaction with the user.

[0177] In one example, relevant wake-up words can be set in advance for the vehicle-mounted robotic arm. When the wake-up words are recognized, the vehicle-mounted robotic arm acquires the voice information of the user and performs recognition processing on the voice information through the built-in first neural network model, so as to determine the first voice control instruction.

[0178] In related technologies, the technologies for performing speech control command recognition have been relatively mature. The specific network structure of the first neural network model can refer to the network model structure in related technologies, and will not be elaborated here.

[0179] In addition, since the first neural network model is built into the vehicle control system, the on-vehicle robotic arm can also achieve natural conversations with users through the first neural network model to provide users with a more intelligent interaction experience.

[0180] In addition to voice control, in the on-vehicle robotic arm control method of the embodiments of the present application, the robotic arm can also be made to enter the care mode through the vehicle display screen. The on-vehicle robotic arm control method in the embodiments of the present application further includes: in response to a first operation control instruction indicating entering the care mode input by the user through the human-machine interaction interface on the vehicle display screen, entering the care mode so that the on-vehicle robotic arm performs care operations on the object to be cared for.

[0181] In practical applications, before performing driving operations, users can first input relevant control operations through the vehicle display screen via the human-machine interaction interface. In one example, the vehicle display screen is located on the vehicle console, and this display screen can integrate multiple control functions for the vehicle. Users can operate on the vehicle display screen in the same human-machine interaction manner as operating a mobile phone or a tablet computer to control the vehicle to implement various functions. In one example, the display interface of the vehicle display screen includes relevant function buttons for entering the care mode, and users can directly touch the button to make the on-vehicle robotic arm enter the care mode.

[0182] By applying the method of the embodiments of the present application, users can operate the vehicle display screen by voice control commands or in the way of input through the human-machine interaction interface to control the on-vehicle robotic arm to implement various functions. Through a relatively simple control method, the user's operation can be made more convenient, improving the user experience.

[0183] In a possible implementation manner, the on-vehicle robotic arm can also implement a grasping function, and then the method of the embodiments of the present application can further include as Figure 6 shown in the following steps:

[0184] Step S601: In response to receiving a second voice control command indicating going to the first position to pick up the first object, control the on-vehicle robotic arm to move to the first position.

[0185] Step S602: Obtain the first image data at the first position, perform target recognition on the first image data at the first position to obtain the first image position of the first object; convert the first image position into the vehicle coordinate system to obtain the first three-dimensional position; control the on-vehicle robotic arm to perform a grasping action on the first three-dimensional position to grasp the first object.

[0186] Among them, the second voice control instruction may include the target object and the position information of the target object, that is, it includes the first object (target object) and the first position of the first object (position information of the target object). When the user is performing a driving operation, if the user needs to pick up an item, the user can directly control the in-vehicle robotic arm through voice interaction.

[0187] In a possible implementation manner, the user can directly tell the in-vehicle robotic arm the first object to be grabbed and the first position. For example, the user can directly say "Help me get the water cup at the rear seat of the vehicle". Then, the in-vehicle robotic arm recognizes and processes the voice information through the built-in first neural network model, obtains that the first object is the water cup and the first position is at the rear seat, so as to control the in-vehicle robotic arm to reach the rear seat.

[0188] Collect image information (first image data) of the rear seat of the vehicle through the camera on the in-vehicle robotic arm or the in-vehicle camera. Perform object recognition on the first image data of the rear seat of the vehicle through a pre-trained second neural network model for image recognition processing, determine the specific position of the water cup, and control the in-vehicle robotic arm to reach that position to perform a grabbing action to get the water cup.

[0189] Among them, the second neural network model may be a model for image recognition processing in related technologies. For example, the second neural network model may be at least one of image recognition models such as VGG (Visual Geometry Group), ResNet (Residual Network), and ConvNeXt (convolutional neural network model).

[0190] The second neural network model can perform object recognition on various objects inside the vehicle to obtain the types of objects. The training process of the second neural network model can be: input the sample images into the second neural network model for training. Among them, the sample image data is large and diverse, and the sample images include the first sample object. The second neural network model performs feature extraction, downsampling, feature fusion, feature enhancement, position regression, classification prediction, etc. on the sample images to perform the recognition of the first sample object, and obtain the first sample image position of the first sample object in the sample images. Calculate the accuracy rate of the first sample image position of the first sample object output by the second neural network model in the sample images. If the accuracy rate is greater than or equal to the specified threshold, it is considered that the second neural network model has completed training; otherwise, adjust the parameters of the second neural network model and continue with the model training.

[0191] After obtaining the first image position of the first object through the second neural network model, since the first image position is two-dimensional information, it is necessary to convert it into the corresponding coordinate position in the actual scenario. In one example, through the camera internal parameter matrix, the first image position is converted from the image coordinate system to the camera coordinate system to obtain the three-dimensional position information corresponding to the first object in the first image, and then the three-dimensional position information is converted from the camera coordinate system to the vehicle coordinate system to obtain the first three-dimensional position of the first object in the vehicle, so as to control the on-vehicle robotic arm to perform a grasping action on the first object at the first three-dimensional position.

[0192] After the on-vehicle robotic arm grasps the first object, the first object can be placed at the third position for the user to pick up, realizing the pick-up and delivery service. Among them, the third position can be pre-set, or can be specified by the user through a voice command according to actual needs. In practical applications, since it is inconvenient for the driver to unscrew the water cup while driving, the on-vehicle robotic arm can also perform the action of unscrewing the bottle cap to open the water cup and place the water cup at the third position for the driver to directly pick up and drink the water cup.

[0193] Applying the method of the embodiment of the present application, the on-vehicle robotic arm can realize the grasping function when obtaining the first object and the first position, help the user obtain the target object, provide a more intelligent service for the user, and thus improve the user experience.

[0194] In a possible implementation manner, the grasping function of the on-vehicle robotic arm can also be realized through Figure 7 the steps shown below:

[0195] Step S701: In response to receiving the third voice control instruction indicating to pick up the second object, obtain the in-vehicle image.

[0196] Step S702: Perform target recognition on the in-vehicle image to determine whether the in-vehicle image includes the second object.

[0197] If so, execute Step S703: Determine the second image position of the second object according to the in-vehicle image; convert the second image position to the vehicle coordinate system to obtain the second three-dimensional position; control the on-vehicle robotic arm to perform a grasping action on the second three-dimensional position to grasp the second object.

[0198] If not, execute Step S704: Generate position inquiry information for the second object.

[0199] Step S705: Use the vehicle control system to control the vehicle display screen to display the position inquiry information, and / or control the vehicle audio to play the position inquiry information.

[0200] Step S706: Obtain the second position where the second object is located as input by the user, obtain the second image data at the second position, perform target recognition on the second image data to obtain the second image position of the second object; convert the second image position into the vehicle coordinate system to obtain the second three-dimensional position; control the on-vehicle robotic arm to perform a grasping action on the second three-dimensional position to grasp the second object.

[0201] Among them, the third voice control instruction includes the target object (the second object). In practical applications, the user may only tell the information of the target object. For example, in the case of suddenly encountering a road condition with dazzling light, the driver may directly say "Please help me get my glasses". In such a case, the on-vehicle robotic arm needs to meet more intelligent requirements and obtain the position information of the glasses by itself. Then the on-vehicle robotic arm needs to obtain the image information inside the vehicle and perform image processing on the image information to determine the position information of the glasses.

[0202] In one example, the in-vehicle images corresponding to each in-vehicle camera can be obtained, and the second neural network model trained in advance is used to perform target recognition on each in-vehicle image, so as to judge whether the second object exists in the in-vehicle image. In another example, control the on-vehicle robotic arm to traverse each position inside the vehicle, use the camera on the on-vehicle robotic arm to obtain the in-vehicle images at each position, perform target recognition on each in-vehicle image, and judge whether the second object exists in the in-vehicle image. Among them, when performing target recognition on the in-vehicle images at various positions of the vehicle, the same method as in the above embodiment can be adopted, and image processing is performed through the pre-trained second neural network model to achieve the target recognition of the second object.

[0203] If the second object is found, that is, it is judged that the second object exists in the in-vehicle image, then the second image position information of the second object can be determined by the same method as in the above embodiment, and the second image position is converted into the vehicle coordinate system to obtain the second three-dimensional position, and the on-vehicle robotic arm is controlled to move to the second three-dimensional position to perform the grasping action on the second object.

[0204] If the second object is not found, it is necessary to interact with the user to ask for the position information of the second object. In one example, the position inquiry information can be generated in the same way as in the above embodiment. For example, when the glasses are not found through the in-vehicle image, the position inquiry information "Excuse me, where are the glasses" is generated and displayed through the vehicle display screen, and / or the vehicle audio plays this information to make an inquiry.

[0205] After receiving the location query information, the user tells the vehicle-mounted robotic arm the location information of the second object through voice. In one example, the user can directly state the specific location, such as "at the co-pilot seat", or directly tell the direction, such as "about 30 cm eastward". In practical applications, when generating the location query information, the vehicle control system can also be used to display the in-vehicle image on the vehicle display screen, so as to facilitate the user to observe the specific location of the second object.

[0206] After obtaining the second location of the second object, collect the in-vehicle image at the second location to obtain the second image data. Through the pre-trained second neural network model, determine the second image location information of the second object in the second image data, and convert this second image location into the vehicle coordinate system to obtain the second three-dimensional location, and control the vehicle-mounted robotic arm to move to the second three-dimensional location to perform the grasping action for the second object.

[0207] Applying the method of the embodiment of the present application, in the grasping function, the vehicle-mounted robotic arm can ask the user for location information when the location information of the target object has not been obtained, making the human-machine interaction between the vehicle-mounted robotic arm and the user more intelligent; and when the location information is obtained, it can help the user obtain the target object and provide more intelligent services to the user, thereby improving the user experience.

[0208] In addition, the vehicle-mounted robotic arm can have a certain load-bearing capacity to realize the function of helping the user carry luggage. In one example, when it is recognized that the vehicle trunk is opened, the vehicle control system can be used to control the vehicle-mounted robotic arm to reach the designated position of the trunk, and identify the specific position of the luggage through the camera set on the vehicle-mounted robotic arm or in the vehicle. After obtaining the specific position of the luggage, control the vehicle-mounted robotic arm to reach this specific position and perform the grasping action for the luggage, so as to realize the function of helping the user carry luggage.

[0209] In practical applications, the vehicle-mounted robotic arm can also help the user realize the wake-up function. For example, when the vehicle-mounted robotic arm recognizes that the status information of the infant indicates that the infant is in a deep sleep, it can start timing. When the timing time arrives, generate a prompt message for reminding the user whether to wake up the infant. After the user confirms, it can control the vehicle-mounted robotic arm to move to one side of the infant and perform the wake-up action for the infant. In one example, this wake-up action can be realized by shaking the infant left and right, or by gently pushing the infant. While the vehicle-mounted robotic arm performs the wake-up action for the infant, the vehicle control system can also control the change of the in-vehicle environment to wake up the infant. For example, the light intensity and the in-vehicle audio volume can be increased, and the infant can be woken up by playing a ringtone or a preset audio through the in-vehicle audio.

[0210] In the second aspect of the embodiments of the present application, a vehicle-mounted robotic arm control device is provided, and the device includes as shown in Figure 8 the following structure:

[0211] An image data acquisition module 801, configured to acquire image data of a target object to be cared for during the process of using the vehicle-mounted robotic arm to perform a care operation on the target object to be cared for.

[0212] A status information determination module 802, configured to analyze the image data to obtain the status information of the target object to be cared for.

[0213] A prompt information generation module 803, configured to generate a care operation prompt information corresponding to the status information.

[0214] A care operation control module 804, configured to control the vehicle-mounted robotic arm to perform the care operation corresponding to the care operation prompt information when an acknowledgement operation for the care operation prompt information is obtained.

[0215] By applying the device of the embodiments of the present application, during the process of the vehicle-mounted robotic arm performing a care operation, the status information of the target object to be cared for can be determined by analyzing the image data of the target object to be cared for, so as to generate a corresponding care operation prompt information according to the status information. And when the user performs an operation confirmation, the robotic arm is controlled to complete the corresponding care operation, thereby realizing the human-machine interaction between the vehicle-mounted robotic arm and the user. Moreover, the vehicle-mounted robotic arm can actively analyze the image data of the target object to be cared for during the process of performing the care operation, and determine the corresponding care operation according to the status information of the target object to be cared for, so that the vehicle-mounted robotic arm realizes a more intelligent care operation, improves the intelligent level of the vehicle-mounted robotic arm, can meet various needs of the user, and further improves the user experience.

[0216] In a possible implementation manner, the target object to be cared for is an infant, and the prompt information generation module includes:

[0217] A first prompt information generation sub-module, specifically configured to generate a first care operation prompt information for prompting whether to perform a doll performance when the status information indicates that the infant is awake.

[0218] A first prompt control sub-module, specifically configured to control the vehicle display screen to display the first care operation prompt information by using the control system of the vehicle, and / or control the vehicle audio to play the voice of the first care operation prompt information.

[0219] The care operation control module includes:

[0220] The first nursing operation control sub-module is specifically configured to control the vehicle-mounted robotic arm to move to the first designated position in front of the object to be nursed and perform a preset doll performance action when a confirmation operation for the first nursing operation prompt information is obtained.

[0221] By applying the device according to the embodiment of the present application, when the vehicle-mounted robotic arm recognizes that the infant is in a waking state, it can generate the first nursing operation prompt information to prompt the user, and perform the doll performance function for the infant when the user confirms, so as to realize the early education of the infant, so that the vehicle-mounted robotic arm can help the user achieve a more intelligent nursing operation and improve the user experience.

[0222] In a possible implementation manner, the object to be nursed is an infant, and the prompt information generation module includes:

[0223] The second prompt information generation sub-module is specifically configured to generate the second nursing operation prompt information for prompting whether to perform the nursing operation for the infant to fall asleep when the status information indicates that the infant is sleepy.

[0224] The second prompt control sub-module is specifically configured to control the vehicle display screen to display the second nursing operation prompt information and / or control the vehicle audio to play the voice of the second nursing operation prompt information by using the vehicle control system.

[0225] The nursing operation control module includes:

[0226] The second nursing operation control sub-module is specifically configured to control the vehicle-mounted robotic arm to perform the action of covering the quilt for the object to be nursed when a confirmation operation for the second nursing operation prompt information is obtained.

[0227] The device according to the embodiment of the present application may further include:

[0228] The sleep environment adjustment module is configured to adjust the vehicle lights to the sleep light intensity and / or adjust the temperature of the rear row air conditioner of the vehicle to the sleep temperature and / or control the vehicle audio to play the preset lullaby audio by using the control system.

[0229] By applying the device according to the embodiment of the present application, when the vehicle-mounted robotic arm recognizes that the infant is in a sleepy state, it can generate the second nursing operation prompt information to prompt the user, and perform the operation of covering the quilt for the infant when the user confirms, and adjust the vehicle interior environment to help the user take care of the infant to fall asleep, so that the vehicle-mounted robotic arm can help the user achieve a more intelligent nursing operation and improve the user experience.

[0230] In a possible implementation manner, the object to be nursed is an infant, and the prompt information generation module includes:

[0231] The third prompt information generation sub-module is specifically configured to generate a third care operation prompt information for prompting whether to perform infant sleep care when the status information indicates that the infant is in a deep sleep.

[0232] The third prompt control sub-module is specifically configured to use the vehicle control system to control the vehicle display screen to display the third care operation prompt information, and / or control the vehicle audio to play the voice of the third care operation prompt information.

[0233] The care operation control module includes:

[0234] The third care operation control sub-module is specifically configured to control the on-vehicle robotic arm to move to the second specified position on the side of the object to be cared for when an acknowledgment operation for the third care operation prompt information is obtained, and when the object to be cared for has an action of slipping out of the baby chair, control the on-vehicle robotic arm to perform a supporting operation on the object to be cared for.

[0235] The device according to the embodiment of the present application may further include:

[0236] The vehicle parameter adjustment module is used to use the control system to adjust the volume of the vehicle audio to mute, and / or obtain the movement condition of the vehicle, calculate the target damping coefficient according to the movement condition, and use the control system to set the damping coefficient of the seat damper of the target vehicle seat to the target damping coefficient, where the baby chair is installed on the target vehicle seat.

[0237] When applying the device according to the embodiment of the present application, the on-vehicle robotic arm can generate the third care operation prompt information to prompt the user when it recognizes that the infant is in a deep sleep state, and when the user confirms, set the damping coefficient of the seat damper of the target vehicle seat to the target damping coefficient, and perform a supporting operation on the infant when the infant gets out of the seat, ensuring that the infant's sleep is relatively stable, so that the on-vehicle robotic arm can help the user achieve a more intelligent care operation and improve the user experience.

[0238] In a possible implementation manner, the object to be cared for is an infant, and the prompt information generation module includes:

[0239] The fourth prompt information generation sub-module is specifically configured to control the on-vehicle robotic arm to perform a supporting operation on the care object and generate a fourth care operation prompt information for prompting to pull over when the status information indicates that the infant is struggling to get out of the seat.

[0240] The fourth prompt control sub-module is specifically configured to use the vehicle control system to control the vehicle display screen to display the fourth care operation prompt information, and / or control the vehicle audio to play the voice of the fourth care operation prompt information.

[0241] When the vehicle-mounted robotic arm of the device according to the embodiments of the present application recognizes that an infant breaks free from the seat, it can perform a supporting operation to ensure the riding safety of the infant and generate a fourth care operation prompt message to prompt the user, so that the vehicle-mounted robotic arm can help the user achieve a more intelligent care operation and improve the user experience.

[0242] In a possible implementation manner, the device according to the embodiments of the present application may further include:

[0243] A first care mode determination module, configured to enter the care mode in response to obtaining a first voice control instruction indicating entering the care mode, so that the vehicle-mounted robotic arm performs a care operation on the object to be cared for.

[0244] And / or,

[0245] A second care mode determination module, configured to enter the care mode in response to a first operation control instruction indicating entering the care mode input by the user through a human-computer interaction interface on the vehicle display screen, so that the vehicle-mounted robotic arm performs a care operation on the object to be cared for.

[0246] When using the device according to the embodiments of the present application, the user can control the vehicle-mounted robotic arm to implement various functions by means of voice commands or operating the vehicle display screen. Through a relatively simple control method, the user's operation can be made more convenient, improving the user experience.

[0247] In a possible implementation manner, the device according to the embodiments of the present application may further include:

[0248] A first vehicle-mounted robotic arm movement module, configured to control the vehicle-mounted robotic arm to move to the first position in response to receiving a second voice control instruction indicating going to the first position to pick up the first object.

[0249] A first grasping action execution module, which acquires the first image data at the first position, performs target recognition on the first image data at the first position to obtain the first image position of the first object; converts the first image position into the vehicle coordinate system to obtain the first three-dimensional position; and controls the vehicle-mounted robotic arm to perform a grasping action on the first three-dimensional position to grasp the first object.

[0250] When using the device according to the embodiments of the present application, the vehicle-mounted robotic arm can implement the grasping function when obtaining the target object and the target object position information, help the user obtain the target object, provide a more intelligent service to the user, and thus improve the user experience.

[0251] In a possible implementation manner, the device according to the embodiments of the present application may further include:

[0252] An in-vehicle image acquisition module, configured to acquire an in-vehicle image in response to receiving a third voice control instruction indicating the taking of a second object.

[0253] A target object determination module, configured to perform target recognition on the in-vehicle image to determine whether the in-vehicle image includes a second object.

[0254] A second grasping action execution module, configured to, if so, determine a second image position of the second object based on the in-vehicle image, convert the second image position into a vehicle coordinate system to obtain a second three-dimensional position; control the on-vehicle robotic arm to perform a grasping action on the second three-dimensional position to grasp the second object.

[0255] A position inquiry information generation module, configured to, if not, generate position inquiry information for the second object.

[0256] A position information inquiry module, configured to control a vehicle display screen to display the position inquiry information by using a control system of the vehicle, and / or control a vehicle audio system to play the position inquiry information.

[0257] A third grasping action execution module, configured to obtain a second position where the second object is located input by a user, obtain second image data at the second position, perform target recognition on the second image data to obtain a second image position of the second object; convert the second image position into a vehicle coordinate system to obtain a second three-dimensional position; control the on-vehicle robotic arm to perform a grasping action on the second three-dimensional position to grasp the second object.

[0258] When applying the device according to the embodiments of the present application, in the grasping function, the on-vehicle robotic arm can inquire the user about the position information of the target object without obtaining the position information of the target object, making the human-machine interaction between the on-vehicle robotic arm and the user more intelligent; and when obtaining the position information of the target object, it can help the user obtain the target object and provide more intelligent services to the user, thereby improving the user experience.

[0259] In a third aspect of the embodiments of the present application, an on-vehicle robotic arm is provided, and the on-vehicle robotic arm is configured to execute any one of the on-vehicle robotic arm control methods in the first aspect of the embodiments of the present application.

[0260] In a fourth aspect of the embodiments of the present application, a vehicle is provided, and the vehicle includes the on-vehicle robotic arm according to the third aspect of the embodiments of the present application, and the on-vehicle robotic arm is configured to execute any one of the on-vehicle robotic arm control methods in the third aspect of the embodiments of the present application.

[0261] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the processes or functions described in the embodiments of the present application are generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions can be transmitted from a website, computer, server, or data center to another website, computer, server, or data center via wired (such as coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (such as infrared, wireless, microwave, etc.) means. The computer-readable storage medium can be any available medium that a computer can access, or a data storage device such as a server or data center that includes one or more integrated available media. The available medium can be a magnetic medium (such as a floppy disk, hard disk, magnetic tape), an optical medium (such as a DVD), or a semiconductor medium (such as a solid state disk (SSD)).

[0262] It should be noted that in this document, relational terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising", "including", or any other variant thereof is intended to cover non-exclusive inclusion, such that a process, method, article, or device that includes a series of elements includes not only those elements but also other elements that are not explicitly listed, or also includes elements that are inherent to such process, method, article, or device. Without further limitation, an element defined by the statement "including a..." does not exclude the existence of additional identical elements in the process, method, article, or device that includes the element.

[0263] Each embodiment in this specification is described in a related manner. For the same or similar parts among the embodiments, reference can be made to each other. Each embodiment focuses on the differences from other embodiments. In particular, for the device and vehicle-mounted robotic arm and vehicle embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and reference can be made to the corresponding parts of the method embodiments for the relevant content.

[0264] The above are only the preferred embodiments of the present application and are not intended to limit the protection scope of the present application. Any modifications, equivalent replacements, improvements, etc. made within the spirit and principle of the present application are all included in the protection scope of the present application.

Claims

1. A vehicle-mounted robotic arm control method, characterized in that: The method comprises: In the process of performing a care operation on the object to be cared for by using the vehicle-mounted mechanical arm, acquiring image data of the object to be cared for; Analyzing the image data to obtain status information of the object to be cared for; generating nursing operation prompt information corresponding to the status information; When a confirmation operation for the nursing operation prompt information is obtained, the vehicle-mounted mechanical arm is controlled to perform the nursing operation corresponding to the nursing operation prompt information.

2. The method according to claim 1, characterized in that: The object to be cared for is an infant, and the generating of the care operation prompt information corresponding to the status information includes: When the state information indicates that the infant is awake, generating first care operation prompt information for prompting whether to perform a puppet show; Using a vehicle control system to control a vehicle display screen to display the first nursing operation prompt information, and / or controlling a vehicle audio system to play a voice message of the first nursing operation prompt information; When a confirmation operation for the nursing operation prompt information is obtained, controlling the vehicle-mounted mechanical arm to perform the nursing operation corresponding to the nursing operation prompt information includes: When a confirmation operation for the first care operation prompt information is obtained, the vehicle-mounted mechanical arm is controlled to move to a first designated position in front of the object to be cared for, and a preset puppet performance action is performed.

3. The method according to claim 1, characterized in that The object to be cared for is an infant, and the generating of the care operation prompt information corresponding to the status information includes: When the state information indicates that the infant is sleepy, generating second care operation prompt information for prompting whether to perform care for the infant to fall asleep; Using a vehicle control system to control a vehicle display screen to display the second nursing operation prompt information, and / or controlling a vehicle audio system to play a voice message of the second nursing operation prompt information; When a confirmation operation for the nursing operation prompt information is obtained, controlling the vehicle-mounted mechanical arm to perform the nursing operation corresponding to the nursing operation prompt information includes: In the case of obtaining a confirmation operation for the second care operation prompt information, controlling the vehicle-mounted mechanical arm to perform a quilt-covering action for the object to be cared for; The method further comprises: The control system is used to adjust the vehicle's lights to sleeping light intensity, and / or adjust the temperature of the vehicle's rear air conditioner to sleeping temperature, and / or control the vehicle's audio system to play preset lullaby audio.

4. The method according to claim 1, characterized in that: The object to be cared for is an infant, and the generating of the care operation prompt information corresponding to the status information includes: When the state information indicates that the infant is soundly asleep, generating third care operation prompt information for prompting whether to perform sleep care for the infant; Using the vehicle control system to control the vehicle display screen to display the third nursing operation prompt information, and / or controlling the vehicle audio to play the voice of the third nursing operation prompt information; When a confirmation operation for the nursing operation prompt information is obtained, controlling the vehicle-mounted mechanical arm to perform the nursing operation corresponding to the nursing operation prompt information includes: In the case of obtaining a confirmation operation for the third care operation prompt information, controlling the vehicle-mounted mechanical arm to move to a second designated position on the side of the object to be cared for, and when the object to be cared for has a movement of sliding off the baby chair, controlling the vehicle-mounted mechanical arm to perform a supporting operation for the object to be cared for; The method further comprises: Using the control system to adjust the volume of the vehicle audio to off; and / or, The movement condition of the vehicle is obtained, a target damping coefficient is calculated according to the movement condition, and the damping coefficient of the seat damper of the target vehicle seat is set to the target damping coefficient using the control system, wherein the baby chair is installed on the target vehicle seat.

5. The method according to claim 1, characterized in that The object to be cared for is an infant, and the generating of the care operation prompt information corresponding to the status information includes: When the status information indicates that the infant is trying to break free from the seat, controlling the vehicle-mounted mechanical arm to perform a supporting operation on the care object, and generating fourth care operation prompt information for prompting the infant to pull over; The vehicle control system is used to control the vehicle display screen to display the fourth care operation prompt information, and / or to control the vehicle audio to play the voice of the fourth care operation prompt information.

6. The method according to claim 1, characterized in that In the process of performing a care operation on the object to be cared for by using the vehicle-mounted mechanical arm, before acquiring image data of the object to be cared for, the method further includes: In response to obtaining a first voice control instruction indicating entering a care mode, entering the care mode so that the vehicle-mounted mechanical arm performs a care operation on the object to be cared for; or, In response to a first operation control instruction entered by a user through a human-computer interaction interface on a vehicle display screen, indicating entering a care mode, the care mode is entered so that the vehicle-mounted mechanical arm performs a care operation on the object to be cared for.

7. The method according to claim 1, characterized in that The method further comprises: In response to receiving a second voice control instruction indicating to go to a first position to pick up a first object, controlling the vehicle-mounted mechanical arm to move to the first position; Acquire first image data at the first position, perform target recognition on the first image data at the first position to obtain a first image position of a first object; convert the first image position into a vehicle coordinate system to obtain a first three-dimensional position; and control the vehicle-mounted mechanical arm to perform a grasping action on the first three-dimensional position to grasp the first object.

8. The method according to claim 1, characterized in that The method further comprises: In response to receiving a third voice control instruction indicating to pick up a second object, acquiring an image inside the vehicle; Performing target recognition on the in-vehicle image to determine whether the in-vehicle image includes the second object; If yes, determining a second image position of the second object according to the in-vehicle image; converting the second image position into a vehicle coordinate system to obtain a second three-dimensional position; and controlling the vehicle-mounted mechanical arm to perform a grasping action on the second three-dimensional position to grasp the second object; If not, generate position query information for the second object; use the vehicle control system to control the vehicle display screen to display the position query information, and / or control the vehicle audio to play the position query information; obtain the second position of the second object input by the user, obtain second image data at the second position, perform target recognition on the second image data to obtain the second image position of the second object; convert the second image position into the vehicle coordinate system to obtain a second three-dimensional position; control the vehicle-mounted mechanical arm to perform a grasping action on the second three-dimensional position to grasp the second object.

9. A vehicle-mounted mechanical arm control device, characterized in that: The device comprises: An image data acquisition module, used to acquire image data of the object to be cared for during the process of performing a care operation on the object to be cared for by using the vehicle-mounted mechanical arm; A status information determination module, used for analyzing the image data to obtain status information of the object to be cared for; A prompt information generating module, used for generating nursing operation prompt information corresponding to the status information; The nursing operation control module is used to control the vehicle-mounted mechanical arm to perform the nursing operation corresponding to the nursing operation prompt information when a confirmation operation for the nursing operation prompt information is obtained.

10. A vehicle-mounted robotic arm, characterized in that: The vehicle-mounted robotic arm is used to execute the vehicle-mounted robotic arm control method described in any one of claims 1-8.

11. A vehicle, characterized in that: The vehicle comprises the vehicle-mounted robotic arm as claimed in claim 10.

Images