An object-taking method, device, vehicle and medium in a vehicle cabin

Abstract

The application relates to a method and device for taking objects in a vehicle cabin, a vehicle and a medium, and belongs to the technical field of vehicle cabins; the method comprises the following steps: firstly detecting whether a target user in the vehicle cabin has an object-taking intention; when it is detected that the target user has the object-taking intention, acquiring target image data of the vehicle cabin, and determining a user position of the target user and an object position of a target object corresponding to the object-taking intention according to the target image data; and according to the user position and the object position, controlling a mechanical arm to deliver the target object to the target user. Through the application, the target object can be automatically taken for the user in the vehicle cabin during vehicle driving, so that the user driving the vehicle is prevented from being distracted, and the safety of the user in the vehicle is guaranteed.

Description

Technical Field

[0001] This invention relates to the technical field of vehicle cabins, and more specifically to a method, device, vehicle, and medium for retrieving items from a vehicle cabin. Background Technology

[0002] Driver inattention is a major contributing factor to traffic accidents. One reason for this is that while driving, drivers may reach for items in the passenger seat or back seat, or retrieve dropped items. This can cause drivers to lose focus on their surroundings for a short period, leading to traffic accidents. Summary of the Invention

[0003] In view of the above problems, a method, apparatus, vehicle, and medium for retrieving items from a vehicle cabin are proposed to overcome or at least partially solve the above problems, including:

[0004] A method for retrieving an object from a vehicle cabin, the cabin being equipped with a robotic arm, the method comprising:

[0005] Detect whether the target user inside the cabin has the intention to retrieve an item;

[0006] When the target user is detected to have the intention to retrieve an object, target image data of the cabin is acquired, and the user's location and the location of the target object corresponding to the intention to retrieve the object are determined based on the target image data.

[0007] Based on the user's location and the item's location, the robotic arm is controlled to deliver the target item to the target user.

[0008] Optionally, the robotic arm includes multiple grippers, and the method further includes:

[0009] The target object is identified to determine its type.

[0010] Based on the type of item, the target gripper is determined from multiple grippers;

[0011] The process of controlling the robotic arm to transfer the target object to the target user includes:

[0012] The robotic arm is controlled to use the target gripper to pick up the target object and then pass it to the target user.

[0013] Optionally, controlling the robotic arm to deliver the target object to the target user based on the user's location and the object's location includes:

[0014] Based on the user's location and the item's location, a movement trajectory for the robotic arm is generated;

[0015] The robotic arm is controlled according to the movement trajectory.

[0016] Optionally, detecting whether the target user inside the cabin intends to retrieve an item includes:

[0017] Receive voice data input by the target user and determine whether the voice data includes a target trigger word;

[0018] When the voice data includes the target trigger word, it is determined that the target user has the intention to retrieve the object;

[0019] Determining the user location of the target user and the item location corresponding to the retrieval intention based on the target image data includes:

[0020] The voice data is then recognized to identify the target object;

[0021] The target image data is identified to determine the user location of the target user who input the voice data, and to determine the object location of the target object.

[0022] Optionally, detecting whether the target user inside the cabin intends to retrieve an item includes:

[0023] When the target object is detected to have fallen, it is checked whether the target user has been received to perform an operation on a preset button.

[0024] When the target user's operation on the preset button is detected, it is determined that the target user has the intention to retrieve the object.

[0025] Optionally, detecting whether the target user inside the cabin intends to retrieve an item includes:

[0026] When the target object is detected to have fallen, the frequency of the target user's viewpoint being in the falling area within a preset time period is detected.

[0027] When the frequency exceeds a preset number of times, it is determined that the target user has the intention to retrieve the object.

[0028] Optionally, detecting whether the target user inside the cabin intends to retrieve an item includes:

[0029] When it is detected that the frequency of the target user's viewpoint being located in an area outside the target area exceeds a preset number of times within a preset time period, it is determined that the target user has the intention to retrieve the object.

[0030] Determining the user location of the target user and the item location corresponding to the retrieval intention based on the target image data includes:

[0031] The target image data is identified to determine the user's location and at least one item within the area;

[0032] Based on the user information of the target user, a target object is determined from the at least one item, and the location of the target object is determined.

[0033] The present invention also provides a retrieval device for a vehicle cabin, the cabin being equipped with a robotic arm, the device comprising:

[0034] The intent detection module is used to detect whether the target user in the cabin has the intent to retrieve an object;

[0035] The location determination module is used to acquire target image data of the cabin when the target user is detected to have the intention to retrieve an object, and to determine the user's location and the location of the target object corresponding to the intention to retrieve the object based on the target image data.

[0036] The control module is used to control the robotic arm to transfer the target object to the target user based on the user's location and the object's location.

[0037] Optionally, the robotic arm includes multiple grippers, and the device further includes:

[0038] The gripper determination module is used to identify the target object, determine the type of the target object, and determine the target gripper from multiple grippers based on the type of the target object.

[0039] The control module is used to control the robotic arm to use the target gripper to pick up the target object and pass the target object to the target user.

[0040] Optionally, the control module is configured to generate a movement trajectory for the robotic arm based on the user's position and the object's position; and to control the robotic arm based on the movement trajectory.

[0041] Optionally, the intent detection module is configured to receive voice data input by the target user and determine whether the voice data includes a target trigger word; when the voice data includes a target trigger word, it is determined that the target user has the intent to retrieve an object.

[0042] The location determination module is used to identify the voice data to determine the target object; to identify the target image data to determine the user location of the target user who input the voice data; and to determine the object location.

[0043] Optionally, the intent detection module is used to detect whether the target user has been received to operate on a preset button when the target object is detected to have fallen; when the target user is detected to operate on the preset button, it is determined that the target user has the intent to retrieve the object.

[0044] Optionally, the intent detection module is used to detect, when the target object is detected to have fallen, the frequency of the target user's viewpoint being in the falling area within a preset time period; when the frequency exceeds a preset number of times, it is determined that the target user has the intent to retrieve the object.

[0045] Optionally, the intent detection module is used to determine that the target user has the intent to retrieve an object when it detects that the frequency of the target user's viewpoint being located in an area outside the target area exceeds a preset number of times within a preset time period.

[0046] The location determination module is used to identify the target image data to determine the user location of the target user and at least one item in the area; based on the user information of the target user, to determine the target object from the at least one item and to determine the item location of the target object.

[0047] The present invention also provides a vehicle, characterized in that it includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the above-described method for retrieving items from the vehicle cabin.

[0048] The present invention also provides a computer-readable storage medium, characterized in that a computer program is stored on the computer-readable storage medium, and the computer program, when executed by a processor, implements the above-described method for retrieving items from a vehicle cabin.

[0049] The beneficial effects of this invention are:

[0050] In this invention, the system first detects whether the target user in the cabin intends to retrieve an object. When this intention is detected, target image data of the cabin is acquired, and the user's position and the object's position corresponding to the intention are determined based on the target image data. Then, a robotic arm is controlled to deliver the object to the user based on the user's position and the object's position. This invention enables automatic retrieval of objects for the user in the cabin while the vehicle is in motion, thus avoiding distraction for the driver and ensuring the safety of the occupants. Attached Figure Description

[0051] Figure 1 This is a flowchart illustrating the steps of a method for retrieving items from a vehicle cabin according to an embodiment of the present invention.

[0052] Figure 2 This is a flowchart of another method for retrieving items from a vehicle cabin according to an embodiment of the present invention;

[0053] Figure 3a This is a flowchart of another method for retrieving items from a vehicle cabin according to an embodiment of the present invention;

[0054] Figure 3b This is a schematic diagram of the structure of a vehicle according to an embodiment of the present invention;

[0055] Figure 3c This is an embodiment of the present invention based on Figure 3b A flowchart outlining the steps for retrieving items from a vehicle;

[0056] Figure 4 This is a flowchart of another method for retrieving items from a vehicle cabin according to an embodiment of the present invention;

[0057] Figure 5 This is a flowchart of another method for retrieving items from a vehicle cabin according to an embodiment of the present invention;

[0058] Figure 6 This is a flowchart of another method for retrieving items from a vehicle cabin according to an embodiment of the present invention;

[0059] Figure 7 This is a schematic diagram of the structure of a vehicle cabin retrieval device according to an embodiment of the present invention. Detailed Implementation

[0060] To make the above-mentioned objects, features, and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without inventive effort are within the scope of protection of the present invention.

[0061] To prevent drivers from being distracted while retrieving items from the vehicle cabin, this invention provides a method for retrieving items from the vehicle cabin. The cabin includes a movable robotic arm that can be folded and stored in a storage compartment. When the system detects that a user in the cabin intends to retrieve an item, it collects image data of the cabin and determines the positions of the user and the item based on the image data. Then, based on the positions of the user and the item, it controls the robotic arm to deliver the item to the user, thus allowing the user to retrieve the item without being distracted and ensuring driving safety.

[0062] Reference Figure 1The diagram illustrates a step-by-step flowchart of a method for retrieving items from a vehicle cabin according to an embodiment of the present invention, which may include the following steps:

[0063] Step 101: Detect whether the target user in the cabin has the intention to retrieve the object.

[0064] During vehicle operation, the intent to retrieve an object can be detected for target users in the cabin. Target users can refer to the driver, co-driver, or passenger currently in the cabin. The intent to retrieve an object refers to the user's intention to obtain an item currently in the cabin.

[0065] As an example, it could be to monitor whether the target user intends to take the object in real time, or it could be to detect whether the target user intends to take the object at preset time intervals. This embodiment of the invention does not limit this.

[0066] If no intention of the target user to retrieve the item is detected, step 101 can be executed; otherwise, if an intention of the target user to retrieve the item is detected, steps 102-103 can be executed to complete the automatic item retrieval process.

[0067] Step 102: When the target user is detected to have the intention to retrieve an object, acquire the target image data of the cockpit, and determine the user's location and the location of the target object corresponding to the user's intention to retrieve the object based on the target image data.

[0068] Specifically, when the intention of the target user to retrieve the object is detected, target image data collected from the vehicle's cabin can be obtained. This target image data can be collected by a camera device installed in the cabin, or by a camera of another device bound to the vehicle. For example, the target image data can be image data collected from the cabin by the camera of the user's mobile phone, which is bound to the vehicle. This embodiment of the invention does not limit this.

[0069] It should be noted that when the target image data is captured by the camera of another device, the camera usage permission needs to be authorized on the other device in advance.

[0070] After acquiring the target image data, the target image data can be identified to determine the user's location in the cockpit and the location of the target object corresponding to the intention to retrieve the object in the cockpit. The target object can refer to the item that the target user wants to retrieve from the cockpit, such as an item that has fallen into the cockpit or an item placed somewhere in the cockpit. This embodiment of the invention does not limit this.

[0071] Step 103: Based on the user's location and the object's location, control the robotic arm to deliver the target object to the target user.

[0072] After determining the user's location and the object's location, the robotic arm can be controlled to move to the area corresponding to the object's location to grab the object, and then move to the area corresponding to the user's location to hand the object to the user.

[0073] As an example, when the target object is moved to the user's location, the user can be reminded to remove the target object held by the robotic arm; for example, a voice prompt can be used to remind the user.

[0074] As another example, once the target user is detected taking the target item, the robotic arm can return to the storage compartment until the user's intention to take the item is detected again.

[0075] In this embodiment of the invention, it is possible to first detect whether the target user in the cabin intends to retrieve an object; when this intention is detected, target image data of the cabin is acquired, and the user's position and the object's position corresponding to the intention are determined based on the target image data; based on the user's position and the object's position, a robotic arm is controlled to deliver the object to the user. This invention enables automatic retrieval of objects for the user in the cabin while the vehicle is in motion, thereby avoiding distraction for the driver and ensuring the safety of the occupants.

[0076] Reference Figure 2 The diagram illustrates a flowchart of another method for retrieving items from a vehicle cabin according to an embodiment of the present invention, which may include the following steps:

[0077] Step 201: Detect whether the target user in the cabin has the intention to retrieve the object.

[0078] While the vehicle is in motion, the system can detect the intention of target users inside the cabin to retrieve items.

[0079] Step 202: When the target user is detected to have the intention to retrieve an object, acquire target image data of the cockpit, and determine the user's location and the location of the target object corresponding to the user's intention to retrieve the object based on the target image data.

[0080] When the system detects that the target user intends to retrieve an object, it can acquire target image data collected from the vehicle's cabin. After acquiring the target image data, it can identify the target image data to determine the user's position in the cabin and the position of the target object corresponding to the intention to retrieve the object in the cabin.

[0081] Step 203: The robotic arm includes multiple grippers; it identifies the target object and determines the type of the target object.

[0082] In practical applications, different items can withstand different forces, and different items also have different shapes. In order to avoid damaging the items, multiple grippers can be set on the robotic arm. Different grippers can be used for different types of items. The relationship between the grippers and the type of items can be preset, and this embodiment of the invention does not limit this.

[0083] After identifying the target object, its type can be determined to identify the corresponding item type.

[0084] Step 204: Based on the type of item, determine the target gripper from multiple grippers.

[0085] Then, based on the pre-defined relationship between the grippers and the type of object, the target gripper suitable for gripping the target object can be determined from multiple grippers.

[0086] Step 205: Based on the user's location and the object's location, control the robotic arm to use the target gripper to pick up the target object and pass it to the target user.

[0087] After determining the user's location and the object's location, the robotic arm can be controlled to move to the area corresponding to the object's location to grab the object, and then move to the area corresponding to the user's location to hand the object to the user.

[0088] Specifically, the robotic arm can first be switched to the target gripper state. For example, multiple grippers can be detached from the robotic arm. When a target object needs to be gripped, the robotic arm can first be controlled to connect to the target grippers; then, the robotic arm can be controlled to use the target grippers to grip the target object and pass it to the target user. Alternatively, multiple grippers can be connected to the robotic arm simultaneously; when a target object needs to be gripped, the robotic arm can control the target grippers to face the target object and grip it. This embodiment of the invention does not limit this.

[0089] In this embodiment of the invention, the system first detects whether the target user inside the cabin intends to retrieve an object. When this intention is detected, target image data of the cabin is acquired, and the user's position and the object's position corresponding to the intention are determined based on the target image data. Furthermore, the object is identified to determine its type, and based on this type, a target gripper is selected from multiple grippers on the robotic arm. Then, based on the user's position and the object's position, the robotic arm is controlled to use the target gripper to grasp the object and deliver it to the user. This invention allows different grippers to be used to grasp different objects, thus preventing damage to the user's items during the robotic arm's grasping process.

[0090] The following section will further explain the method for retrieving items from the vehicle cabin provided by this invention, focusing on how to identify the user's intention to retrieve the item. (Refer to...) Figure 3a The diagram illustrates a flowchart of another method for retrieving items from a vehicle cabin according to an embodiment of the present invention, which may include the following steps:

[0091] Step 301: Receive the voice data input by the target user and determine whether the voice data includes the target trigger word.

[0092] In this embodiment of the invention, the target user can trigger the cockpit's item retrieval function by inputting voice data; specifically, when the target user needs to retrieve an item, they can input voice data into the voice input terminal in the cockpit; wherein, the voice input terminal can be a microphone or the like installed in the cockpit.

[0093] After receiving voice data input by the target user, the vehicle system can first recognize the voice data and determine whether the voice data contains a target trigger word. The target trigger word can refer to a word used to trigger the object retrieval function, such as words like "take" or "get". This embodiment of the invention does not limit this.

[0094] If the voice data does not contain the target trigger word, it can be determined that the target user does not have the intention to retrieve the object. In this case, step 301 can be continued. Conversely, if the voice data contains the target trigger word, steps 302 to 307 can be executed to provide the target user with the object retrieval function.

[0095] Step 302: When the voice data includes the target trigger word, it is determined that the target user has the intention to retrieve the object.

[0096] When the voice data contains the target trigger word, it can be determined that the target user has the intention to retrieve items from the cabin.

[0097] Step 303: When the target user is detected to have the intention to retrieve an object, acquire target image data for the cockpit.

[0098] When the system detects that a target user intends to retrieve items from the cabin, it can further acquire target image data collected from the cabin.

[0099] Step 304: Recognize the voice data to determine the target object.

[0100] When the system detects that the target user intends to take an object, it can further identify the voice data to determine the target object that the user wants to take.

[0101] For example, if the voice data is "Please help me get my fallen glasses", by recognizing the voice data and determining that it contains the target trigger word "get", it can be determined that the target user has the intention to retrieve the object. At this point, the voice data can be further recognized to determine that the target object is "the fallen glasses".

[0102] Step 305: Recognize the target image data to determine the user location of the target user of the input voice data and the object location of the target object.

[0103] After acquiring the target image data, the target image data can be identified to determine the user location of the target user of the input voice data.

[0104] For example, target image data can include image data when the target user inputs voice data. By identifying the lip region of each user in the target image data, the target user can be identified; and then the user's position in the cockpit can be determined.

[0105] As another example, user location can also be determined based on voice data. For example, the location of the target user who inputs the voice data relative to the voice input terminal can be determined based on the voice data; then, based on this relative location and the location of the voice input terminal in the cockpit, the user location of the target user can be determined.

[0106] In this embodiment of the invention, when identifying the user's location, the target image data can also be identified to determine the item location of the target object; specifically, each item in the target image data can be identified, and the location of the item that matches the target object can be taken as the item location of the target object.

[0107] Step 306: Generate a movement trajectory for the robotic arm based on the user's location and the object's location.

[0108] After determining the user's location and the object's location, the robotic arm can be controlled to deliver the object to the user. Specifically, to avoid the robotic arm's movement affecting the user's driving or riding in a vehicle, a movement trajectory can be generated based on the user's location and the object's location. This trajectory does not overlap with the user's activity area.

[0109] Step 307: Control the robotic arm according to the movement trajectory.

[0110] After generating the movement trajectory, the robotic arm can be controlled according to the movement trajectory to first move to the location of the target object and then grab the target object; then, the robotic arm can be controlled to continue moving along the movement trajectory to the area where the target user is located and hand the target object to the target user.

[0111] As an example, when the system detects that a target user has taken the target item, the vehicle's infotainment system can control the robotic arm to return to the storage compartment.

[0112] In this embodiment of the invention, voice data input by a target user can be received first, and it can be determined whether the voice data includes a target trigger word. When the voice data includes the target trigger word, it is determined that the target user has the intention to retrieve an object. When the target user's intention to retrieve an object is detected, target image data for the cockpit is acquired. The voice data is recognized to identify the target object. The target image data is recognized to determine the user's position and the object's position. Based on the user's position and the object's position, a movement trajectory for the robotic arm is generated. The robotic arm is controlled according to the movement trajectory. Through this embodiment of the invention, the user can automatically retrieve items in the cockpit through voice input.

[0113] The following detailed description, based on a more specific embodiment of the present invention, further illustrates this invention:

[0114] like Figure 3b As shown, the vehicle may include a vehicle infotainment system 1 (or an in-vehicle entertainment terminal), a microphone 2, a speaker 3, an in-vehicle camera 4, and a robotic arm 5.

[0115] Microphone 2 is connected to the vehicle's infotainment system 1 to receive user-inputted voice and generate corresponding voice data, which is then sent to the system. The system is used to process and recognize the voice data, as well as perform logical judgments and handle interactions between various modules.

[0116] The in-vehicle camera 4 is connected to the vehicle's infotainment system 1 to provide images of the interior of the cabin to the system, enabling the system to locate objects and pinpoint the speaker's location based on the image data.

[0117] Speaker 3 is used to provide feedback on the processing results of vehicle infotainment system 1 and is connected to vehicle infotainment system 1.

[0118] The robotic arm 5 is used to pick up and transfer items; when not in use, the robotic arm 5 is folded and stored in the hibernation chamber, and the robotic arm 5 is connected to the vehicle's machine 1.

[0119] based on Figure 3b The vehicles shown, such as Figure 3c As shown, automatic item retrieval can be achieved through the following steps:

[0120] Step S01: The user issues a voice command, such as: "Hand me the phone on the passenger seat." Microphone 2 converts the voice command into voice data and transmits it to the vehicle's infotainment system 1.

[0121] Step S02: After parsing the voice data, the vehicle system 1 determines whether it is a retrieval command. If it is not a retrieval command, proceed to step S11: end the current process (the retrieval process ends, and the vehicle system 1 can continue to determine other voice requests). If it is a retrieval command, proceed to step S03.

[0122] Step S03: The in-vehicle camera 4 collects image data from the cabin.

[0123] Step S04: The in-vehicle camera 4 sends the image data to the vehicle system 1.

[0124] Step S05: The vehicle system 1 determines whether the item requested by the user exists based on the image data. If it does not exist, proceed to S07: Inform the user via speaker 3 that the item was not found, and proceed to step S11: End the current process. If the item requested by the user has been found, proceed to step S06.

[0125] Step S06: The vehicle-mounted unit 1 sends the trajectory generated based on the object's location and the user's location to the robotic arm 5.

[0126] Step S08: The robotic arm 5 emerges from the storage compartment, picks up the item according to the trajectory transmitted by the vehicle-mounted unit 1, delivers the item to the vicinity of the user's location, and informs the user through the speaker 3.

[0127] Step S09: Determine if the user has accepted the item. This step can be determined by the weight supported by the robotic arm 5. If the item has not been accepted, return to step S08. At this time, the robotic arm 5 will remain waiting for the user to accept the item. If the item has been accepted, proceed to step S10: The robotic arm 5 returns to the storage compartment. After the robotic arm 5 returns, proceed to step S11: End the current process.

[0128] Reference Figure 4 The diagram illustrates a flowchart of another method for retrieving items from a vehicle cabin according to an embodiment of the present invention, which may include the following steps:

[0129] Step 401: When the target object is detected to have fallen, check whether the target user has been received to perform an operation on the preset button.

[0130] In this embodiment of the invention, it is also possible to first detect whether an event of a target object falling has occurred; specifically, it is possible to detect whether an event of a target object falling has occurred by recognizing the image data collected from the target user; the falling of the target object can refer to the target object falling off the target user or the target object falling off a position in the cockpit.

[0131] When an event of a target object falling is detected, it is possible to further detect whether the target user has interacted with a preset button. Specifically, it is possible to detect whether an interaction with the preset button by the target user has been received. This interaction could be a click, touch, or swipe operation on the preset button, and this embodiment of the invention does not impose any limitations on this.

[0132] As an example, the preset button can be a button set on the central control screen or a physical button set in the cockpit; this embodiment of the invention does not limit this.

[0133] Step 402: When the target user's operation on the preset button is detected, it is determined that the target user has the intention to pick up the object.

[0134] When a user's action on the preset button is detected, it can be determined that the target user intends to retrieve the object; conversely, if no user action on the preset button is detected, it can be determined that the target user does not intend to retrieve the object, and in this case, step 401 can be continued.

[0135] Step 403: When the target user is detected to have the intention to retrieve an object, acquire the target image data of the cockpit, and determine the user's location and the location of the target object corresponding to the user's intention to retrieve the object based on the target image data.

[0136] When the system detects that the target user intends to retrieve an object, it can acquire target image data collected from the vehicle's cabin. After acquiring the target image data, it can identify the target image data to determine the user's position in the cabin and the position of the target object corresponding to the intention to retrieve the object in the cabin.

[0137] Step 404: Generate a movement trajectory for the robotic arm based on the user's location and the object's location.

[0138] After determining the user's location and the object's location, the robotic arm can be controlled to deliver the object to the user. Specifically, to avoid the robotic arm's movement affecting the user's driving or riding in a vehicle, a movement trajectory can be generated based on the user's location and the object's location. This trajectory does not overlap with the user's activity area.

[0139] Step 405: Control the robotic arm according to the movement trajectory.

[0140] After generating the movement trajectory, the robotic arm can be controlled according to the movement trajectory to first move to the location of the target object and then grab the target object; then, the robotic arm can be controlled to continue moving along the movement trajectory to the area where the target user is located and hand the target object to the target user.

[0141] As an example, when controlling a robotic arm to pick up an object, the robotic arm can be controlled to use a target gripper to pick up the target object.

[0142] As another example, once the vehicle's infotainment system detects that a target user has taken the target item, it can control the robotic arm to return to the storage compartment.

[0143] In this embodiment of the invention, when a target object is detected to have fallen, it checks whether an operation by the target user on a preset button has been received. If the operation is detected, it determines that the target user intends to retrieve the object. When this intention is confirmed, target image data of the cockpit is acquired, and the user's position and the object's position corresponding to the user's intention are determined based on the target image data. A movement trajectory for the robotic arm is generated based on the user's position and the object's position. The robotic arm is then controlled according to the movement trajectory. This invention enables the automatic retrieval of a fallen object based on the user's operation on a preset button, preventing the user from being distracted while driving.

[0144] Reference Figure 5 The diagram illustrates a flowchart of another method for retrieving items from a vehicle cabin according to an embodiment of the present invention, which may include the following steps:

[0145] Step 501: When a target object is detected falling, detect the frequency of the target user's viewpoint being in the falling area within a preset time period.

[0146] In this embodiment of the invention, it is also possible to first detect whether an event of a target object falling has occurred; specifically, it is possible to detect whether an event of a target object falling has occurred by recognizing the image data collected from the target user; the falling of the target object can refer to the target object falling off the target user or the target object falling off a position in the cockpit.

[0147] When an event of a target object falling is detected, it is possible to further detect whether the target user has the intention to retrieve the target object; specifically, it is possible to detect the frequency with which the target user's viewpoint is in the area where the target object fell within a preset time period.

[0148] The falling area can refer to the area where the target object falls after it has fallen. This area can be determined based on the area where the target object was located before it fell, or it can be determined based on the image data collected during the falling process of the target object. This embodiment of the invention does not limit this.

[0149] Step 502: When the frequency exceeds the preset number of times, it is determined that the target user has the intention to take the item.

[0150] When the detected frequency does not exceed the preset number of times, it can be indicated that the target user does not have the intention to pick up the target object, that is, the target user does not have the intention to take the object; at this time, step 501 can continue to be executed.

[0151] Conversely, if the detected frequency exceeds the preset number, it indicates that the target user has the intention to pick up the target object; at this time, step 503 can be continued.

[0152] The preset number of times can be set according to the actual situation, such as 3 times, 5 times, etc., and the embodiments of the present invention do not limit this.

[0153] Step 503: When the target user is detected to have the intention to retrieve an object, acquire target image data of the cockpit, and determine the user's location and the location of the target object corresponding to the user's intention to retrieve the object based on the target image data.

[0154] When the system detects that the target user intends to retrieve an object, it can acquire target image data collected from the vehicle's cabin. After acquiring the target image data, it can identify the target image data to determine the user's position in the cabin and the position of the target object corresponding to the intention to retrieve the object in the cabin.

[0155] Step 504: Generate a movement trajectory for the robotic arm based on the user's location and the object's location.

[0156] After determining the user's location and the object's location, the robotic arm can be controlled to deliver the object to the user. Specifically, to avoid the robotic arm's movement affecting the user's driving or riding in a vehicle, a movement trajectory can be generated based on the user's location and the object's location. This trajectory does not overlap with the user's activity area.

[0157] Step 505: Control the robotic arm according to the movement trajectory.

[0158] After generating the movement trajectory, the robotic arm can be controlled according to the movement trajectory to first move to the location of the target object and then grab the target object; then, the robotic arm can be controlled to continue moving along the movement trajectory to the area where the target user is located and hand the target object to the target user.

[0159] As an example, when controlling a robotic arm to pick up an object, the robotic arm can be controlled to use a target gripper to pick up the target object.

[0160] As another example, once the vehicle's infotainment system detects that a target user has taken the target item, it can control the robotic arm to return to the storage compartment.

[0161] In this embodiment of the invention, when a target object is detected to have fallen, the frequency with which the target user's viewpoint is located in the falling area within a preset time period is detected. When the frequency exceeds a preset number of times, it is determined that the target user intends to retrieve the object. When the target user's intention to retrieve the object is detected, target image data for the cockpit is acquired, and the user's position and the object's position corresponding to the user's intention to retrieve the object are determined based on the target image data. Based on the user's position and the object's position, a movement trajectory for the robotic arm is generated. The robotic arm is controlled according to the movement trajectory. Through this embodiment of the invention, when an object falls, it is possible to automatically pick up the object and return it to the user based on the shift in the user's attention, avoiding the user being distracted while driving.

[0162] Reference Figure 6 The diagram illustrates a flowchart of another method for retrieving items from a vehicle cabin according to an embodiment of the present invention, which may include the following steps:

[0163] Step 601: When it is detected that the frequency of the target user's viewpoint being located in an area outside the target area exceeds a preset number of times within a preset time period, it is determined that the target user has the intention to take the object.

[0164] In this embodiment of the invention, the target object that the target user wants to obtain may not be a dropped item, but rather an item stored in a location within the cabin, such as an item in the passenger seat or a rear seat. When the target user needs to retrieve the target object, they may repeatedly look at the area where the target object is located; therefore, the frequency with which the target user's viewpoint is outside the target area that the driver needs to look at can be detected within a preset time period.

[0165] Based on this frequency, it can be determined whether the target user has the intention to take the object; specifically, the relationship between this frequency and a preset number of times can be compared; when the frequency exceeds the preset number of times, it can be determined that the target user has the intention to take the object; otherwise, it can be determined that the target user does not have the intention to take the object.

[0166] Step 602: When the target user is detected to have the intention to retrieve an object, acquire target image data for the cockpit.

[0167] When the system detects that the target user intends to retrieve an object, it can acquire target image data collected from the vehicle's cabin.

[0168] Step 603: Identify the target image data to determine the user's location and at least one item within a region.

[0169] After acquiring the target image data, the target image data can be identified to determine the user's position in the cockpit and to identify at least one item in an area where the viewpoint is located.

[0170] Step 604: Based on the target user's user information, determine the target item from at least one item and determine the item location of the target item.

[0171] Then, based on the target user's user information, the target item that the target user might need can be determined from at least one item and delivered to the target user; specifically, the location of the target item can be determined first based on the target image data.

[0172] As an example, after identifying the target item from at least one item, it can be displayed to the target user. Once the target user confirms the target item, its location is then determined to avoid picking up the wrong item. The target item can be displayed by playing its name via voice or by showing its image on the central control screen. The confirmation operation can be performed via voice, by pressing a button on the central control screen, or by pressing a physical button in the cockpit; this embodiment of the invention does not limit the scope of the method.

[0173] Step 605: Generate a movement trajectory for the robotic arm based on the user's location and the object's location.

[0174] After determining the user's location and the object's location, the robotic arm can be controlled to deliver the object to the user. Specifically, to avoid the robotic arm's movement affecting the user's driving or riding in a vehicle, a movement trajectory can be generated based on the user's location and the object's location. This trajectory does not overlap with the user's activity area.

[0175] Step 606: Control the robotic arm according to the movement trajectory.

[0176] After generating the movement trajectory, the robotic arm can be controlled according to the movement trajectory to first move to the location of the target object and then grab the target object; then, the robotic arm can be controlled to continue moving along the movement trajectory to the area where the target user is located and hand the target object to the target user.

[0177] As an example, when controlling a robotic arm to pick up an object, the robotic arm can be controlled to use a target gripper to pick up the target object.

[0178] As another example, once the vehicle's infotainment system detects that a target user has taken the target item, it can control the robotic arm to return to the storage compartment.

[0179] In this embodiment of the invention, when it is detected that the frequency of the target user's viewpoint being outside the target area exceeds a preset number within a preset time period, it is determined that the target user intends to retrieve an object. When the target user's intention to retrieve an object is detected, target image data of the cockpit is acquired. The target image data is then identified to determine the target user's position and at least one item within the area. Based on the target user's information, the target object is identified from the at least one item, and its position is determined. Based on the user's position and the item's position, a movement trajectory for the robotic arm is generated. The robotic arm is then controlled based on the movement trajectory. Through this embodiment of the invention, based on the user's viewpoint position, items that the user wants to retrieve within the cockpit are automatically handed to the user, without the user needing to be distracted to retrieve them.

[0180] It should be noted that the above methods for determining whether the target user has the intention to take the object can be selected individually or multiple methods can be used together. This embodiment of the invention does not limit this.

[0181] It should be noted that, for the sake of simplicity, the method embodiments are all described as a series of actions. However, those skilled in the art should understand that the embodiments of the present invention are not limited to the described order of actions, because according to the embodiments of the present invention, some steps can be performed in other orders or simultaneously. Furthermore, those skilled in the art should also understand that the embodiments described in the specification are preferred embodiments, and the actions involved are not necessarily essential to the embodiments of the present invention.

[0182] Reference Figure 7 The diagram illustrates a structural schematic of a retrieval device in a vehicle cabin according to an embodiment of the present invention. The cabin is equipped with a robotic arm and may include the following modules:

[0183] The intent detection module 701 is used to detect whether the target user in the cockpit has the intent to retrieve an object.

[0184] The location determination module 702 is used to acquire target image data of the cockpit when it detects that the target user has the intention to retrieve an object, and determine the user's location and the location of the target object corresponding to the retrieval intention based on the target image data.

[0185] The control module 703 is used to control the robotic arm to deliver the target object to the target user based on the user's position and the object's position.

[0186] In one embodiment of the present invention, the robotic arm includes multiple grippers, and the device further includes:

[0187] The gripper identification module is used to identify the target object and determine its type; based on the type, it identifies the target gripper from multiple grippers.

[0188] The control module 703 is used to control the robotic arm to use the target gripper to pick up the target object and pass the target object to the target user.

[0189] In one embodiment of the present invention, the control module 703 is used to generate a movement trajectory for the robotic arm based on the user's position and the object's position; and to control the robotic arm based on the movement trajectory.

[0190] In one embodiment of the present invention, the intent detection module 701 is used to receive voice data input by the target user and determine whether the voice data includes a target trigger word; when the voice data includes a target trigger word, it is determined that the target user has the intent to retrieve an object.

[0191] The location determination module 702 is used to recognize voice data to determine the target object; to recognize target image data to determine the user location of the target user inputting the voice data; and to determine the object location.

[0192] In one embodiment of the present invention, the intent detection module 701 is used to detect whether the target user has received an operation on a preset button when the target object is detected to have fallen; when the target user's operation on the preset button is detected, it is determined that the target user has the intent to retrieve the object.

[0193] In one embodiment of the present invention, the intent detection module 701 is used to detect the frequency of the target user's viewpoint being in the falling area within a preset time period when the target object is detected to have fallen; when the frequency exceeds a preset number of times, it is determined that the target user has the intent to retrieve the object.

[0194] In one embodiment of the present invention, the intent detection module 701 is used to determine that the target user has the intent to take an object when the frequency of the target user's viewpoint being located in a region outside the target region exceeds a preset number of times within a preset time period.

[0195] The location determination module 702 is used to identify target image data to determine the user location of the target user and at least one item in an area; based on the user information of the target user, it determines the target object from at least one item and determines the item location of the target object.

[0196] In this embodiment of the invention, it is possible to first detect whether the target user in the cabin intends to retrieve an object; when this intention is detected, target image data of the cabin is acquired, and the user's position and the object's position corresponding to the intention are determined based on the target image data; based on the user's position and the object's position, a robotic arm is controlled to deliver the object to the user. This invention enables automatic retrieval of objects for the user in the cabin while the vehicle is in motion, thereby avoiding distraction for the driver and ensuring the safety of the occupants.

[0197] This invention also provides a vehicle that may include a processor, a memory, and a computer program stored in the memory and capable of running on the processor. When the computer program is executed by the processor, it implements the above-described method for retrieving items from the vehicle cabin.

[0198] This invention also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the above-described method for retrieving items from a vehicle cabin.

[0199] As the apparatus embodiment is basically similar to the method embodiment, it is described in a relatively simple manner. For relevant details, please refer to the description of the method embodiment.

[0200] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on the differences from other embodiments. The same or similar parts between the various embodiments can be referred to each other.

[0201] Those skilled in the art will understand that embodiments of the present invention can be provided as methods, apparatus, or computer program products. Therefore, embodiments of the present invention can take the form of entirely hardware embodiments, entirely software embodiments, or embodiments combining software and hardware aspects. Furthermore, embodiments of the present invention can take the form of computer program products implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.

[0202] This invention is described with reference to flowchart illustrations and / or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the invention. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing terminal device to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal device, generate instructions for implementing the flowchart illustrations and / or block diagrams. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0203] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing terminal device to operate in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0204] These computer program instructions can also be loaded onto a computer or other programmable data processing terminal equipment, causing a series of operational steps to be performed on the computer or other programmable terminal equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable terminal equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0205] Although preferred embodiments of the present invention have been described, those skilled in the art, upon learning the basic inventive concept, can make other changes and modifications to these embodiments. Therefore, the appended claims are intended to be interpreted as including the preferred embodiments as well as all changes and modifications falling within the scope of the embodiments of the present invention.

[0206] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or terminal device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or terminal device. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or terminal device that includes said element.

[0207] The foregoing has provided a detailed description of a method, apparatus, vehicle, and medium for retrieving items from a vehicle cabin. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A method for retrieving items from a vehicle cabin, characterized in that, The cockpit is equipped with a robotic arm, and the method includes: Detect whether the target user inside the cabin has the intention to retrieve an item; When the target user is detected to have the intention to retrieve an object, target image data of the cabin is acquired, and the user's location and the location of the target object corresponding to the intention to retrieve the object are determined based on the target image data. Based on the user's location and the item's location, the robotic arm is controlled to deliver the target item to the target user; The robotic arm includes multiple grippers, with different grippers suitable for different types of items, and the relationship between the grippers and the item type is preset. The method further includes: The target object is identified to determine its type. Based on the type of item, the target gripper is determined from multiple grippers; The process of controlling the robotic arm to transfer the target object to the target user includes: The robotic arm is controlled to use the target gripper to pick up the target object and then transfer the target object to the target user. The step of detecting whether the target user inside the cabin intends to retrieve an item includes: When the target object is detected to have fallen, the frequency of the target user's viewpoint being in the falling area within a preset time period is detected. When the frequency exceeds a preset number of times, it is determined that the target user has the intention to retrieve the object.

2. The method according to claim 1, characterized in that, The step of controlling the robotic arm to deliver the target object to the target user based on the user's location and the object's location includes: Based on the user's location and the item's location, a movement trajectory for the robotic arm is generated; The robotic arm is controlled according to the movement trajectory.

3. The method according to claim 1, characterized in that, The detection of whether the target user inside the cabin intends to retrieve an item includes: Receive voice data input by the target user and determine whether the voice data includes a target trigger word; When the voice data includes the target trigger word, it is determined that the target user has the intention to retrieve the object; Determining the user location of the target user and the item location corresponding to the retrieval intention based on the target image data includes: The voice data is then recognized to identify the target object; The target image data is identified to determine the user location of the target user who input the voice data, and to determine the object location of the target object.

4. The method according to claim 1, characterized in that, The detection of whether the target user inside the cabin intends to retrieve an item includes: When the target object is detected to have fallen, it is checked whether the target user has been received to perform an operation on a preset button. When the target user's operation on the preset button is detected, it is determined that the target user has the intention to retrieve the object.

5. The method according to claim 1, characterized in that, The detection of whether the target user inside the cabin intends to retrieve an item includes: When it is detected that the frequency of the target user's viewpoint being located in an area outside the target area exceeds a preset number of times within a preset time period, it is determined that the target user has the intention to retrieve the object. Determining the user location of the target user and the item location corresponding to the retrieval intention based on the target image data includes: The target image data is identified to determine the user's location and at least one item within the area; Based on the user information of the target user, a target object is determined from the at least one item, and the location of the target object is determined.

6. A device for retrieving items in a vehicle cabin, characterized in that, The cockpit is equipped with a robotic arm, and the device includes: The intent detection module is used to detect whether the target user in the cabin has the intent to retrieve an object; The location determination module is used to acquire target image data of the cabin when it detects that the target user has the intention to retrieve an object, and determine the user location of the target user and the object location corresponding to the retrieval intention based on the target image data; The control module is used to control the robotic arm to transfer the target object to the target user based on the user's location and the object's location; The robotic arm includes multiple grippers, with different grippers suitable for different types of items, and the relationship between the grippers and the item type is preset. The device also includes: The gripper determination module is used to identify the target object, determine the item type of the target object, and determine the target gripper from the plurality of grippers based on the item type. Specifically, the control module is used to control the robotic arm to use the target gripper to pick up the target object and transfer the target object to the target user; Specifically, the intent detection module is used to detect the frequency of the target user's viewpoint being in the falling area within a preset time period when the target object is detected to have fallen; when the frequency exceeds a preset number of times, it is determined that the target user has the intent to retrieve the object.

7. A vehicle, characterized in that, It includes a processor, a memory, and a computer program stored in the memory and capable of running on the processor, wherein the computer program, when executed by the processor, implements the method of retrieving an item from the vehicle cabin as described in any one of claims 1 to 5.

8. A computer-readable storage medium, characterized in that, A computer program is stored on the computer-readable storage medium, which, when executed by a processor, implements the method for retrieving items from the vehicle cabin as described in any one of claims 1 to 5.

Images