Method, device, electronic device and storage medium for determining target object

Abstract

The present application discloses a method, device, electronic device and storage medium for determining a target object, and relates to the technical field of electronic devices. The method comprises: obtaining pointing information of an occupant in a vehicle, determining a pointing area corresponding to the pointing information, and obtaining one or more objects to be selected corresponding to the pointing area, and determining a target object from one or more objects to be selected in response to a selection instruction of the occupant for the one or more objects to be selected. The present application intuitively indicates the target object through the pointing information and selection instructions of the occupant in the vehicle, which can facilitate the development of subsequent interactive tasks and improve the naturalness and efficiency of the interaction.

Description

Technical Field

[0001] The present application relates to the technical field of electronic devices, and more specifically, to a method, device, electronic device and storage medium for determining a target object. Background Art

[0002] With the development of science and technology, vehicles are used more and more widely and have more and more functions, becoming one of the necessities in people's daily life. Among them, passengers in the vehicle have the need to point to objects inside or outside the vehicle. However, the current pointing recognition for passengers is not accurate enough, which makes it inconvenient to carry out subsequent interactive tasks and leads to poor interactive experience for passengers. Summary of the invention

[0003] In view of the above problems, the present application proposes a method, device, electronic device and storage medium for determining a target object to solve the above problems.

[0004] In a first aspect, an embodiment of the present application provides a method for determining a target object, the method comprising: obtaining pointing information of an occupant in a vehicle; determining a pointing area corresponding to the pointing information, and obtaining one or more objects to be selected corresponding to the pointing area; and determining a target object from the one or more objects to be selected in response to a selection instruction of the occupant for the one or more objects to be selected.

[0005] In a second aspect, an embodiment of the present application provides a device for determining a target object, the device comprising: a pointing information acquisition module, used to acquire pointing information of an occupant in a vehicle; a module for acquiring an object to be selected, used to determine a pointing area corresponding to the pointing information, and acquire one or more objects to be selected corresponding to the pointing area; and a target object determination module, used to determine a target object from the one or more objects to be selected in response to a selection instruction of the occupant for the one or more objects to be selected.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, comprising a memory and a processor, wherein the memory is coupled to the processor, the memory stores instructions, and when the instructions are executed by the processor, the processor executes the above method.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, in which a program code is stored, and the program code can be called by a processor to execute the above method.

[0008] The target object determination method, device, electronic device and storage medium provided in the embodiments of the present application obtain pointing information of an occupant in a vehicle, determine a pointing area corresponding to the pointing information, and obtain one or more objects to be selected corresponding to the pointing area, and determine the target object from the one or more objects to be selected in response to the occupant's selection instructions for the one or more objects to be selected, so that the target object can be intuitively indicated through the pointing information and selection instructions of the occupant in the vehicle, which can facilitate the development of subsequent interactive tasks and improve the naturalness and efficiency of the interaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without paying any creative work.

[0010] Figure 1 A schematic diagram showing a flow chart of a method for determining a target object provided by an embodiment of the present application;

[0011] Figure 2 A schematic diagram showing an embodiment of the present application using different angles of multiple pointing to eliminate ambiguity of an obscured target object or an inaccurate pointing direction;

[0012] Figure 3 A schematic diagram showing a flow chart of a method for determining a target object provided by an embodiment of the present application;

[0013] Figure 4 A schematic diagram showing a method of changing a selected object by moving a finger when previewing a pointing direction in AR glasses according to an embodiment of the present application;

[0014] Figure 5 A schematic diagram showing a flow chart of a method for determining a target object provided by an embodiment of the present application;

[0015] Figure 6 A schematic diagram showing a flow chart of a method for determining a target object provided by an embodiment of the present application;

[0016] Figure 7 A schematic diagram showing a conical area as a selection area constructed around a pointing direction when the selection range is expanded in an embodiment of the present application;

[0017] Figure 8 A schematic diagram showing a flow chart of a method for determining a target object provided by an embodiment of the present application;

[0018] Fig. 9The tracking result of the key points of the human body based on the depth camera provided in the embodiment of the present application is shown;

[0019] Fig.10 The figure shows the finger tracking result based on the depth camera provided in the embodiment of the present application;

[0020] Fig.11 A schematic diagram showing the arrangement of a motion capture system provided in an embodiment of the present application is shown;

[0021] Fig.12 A schematic diagram showing the attachment position of a reflective mark for full-body motion capture provided in an embodiment of the present application;

[0022] Fig.13 A schematic diagram of a glasses-type eye tracker provided in an embodiment of the present application is shown;

[0023] Fig.14 A schematic diagram of an AR handle / VR handle provided in an embodiment of the present application is shown;

[0024] Fig.15 This application shows Figure 8 The flowchart of step S510 of an embodiment of the method for determining a target object is shown;

[0025] Fig.16 This application shows Figure 8 The flowchart of step S510 of an embodiment of the method for determining a target object is shown;

[0026] Fig.17 A schematic diagram showing a flow chart of a method for determining a target object provided by an embodiment of the present application;

[0027] Fig.18 A module block diagram of a target object determination device provided by an embodiment of the present application is shown;

[0028] Fig.19 A block diagram of an electronic device for executing a method for determining a target object according to an embodiment of the present application is shown;

[0029] Fig. 20 A storage unit for storing or carrying a program code for implementing a method for determining a target object according to an embodiment of the present application is shown. DETAILED DESCRIPTION

[0030] In order to enable those skilled in the art to better understand the solution of the present application, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0031] As vehicles become more intelligent, the interactive experience of the occupants (including drivers and passengers) in the vehicle becomes increasingly important. In particular, the development of autonomous driving technology will turn more drivers into passengers, further increasing the demand for human-vehicle interactive experience. These interactive experiences often involve pointing at objects inside or outside the vehicle. The "pointing" here is not only the narrow pointing action of raising a finger, but also any information conveyed by the user in a broad sense that can be used to indicate the direction, such as the user's face direction, the direction of the gaze, the direction of the laser pen (physical or virtual), etc. Pointing information can be expressed in different ways, such as the coordinates of a point, a ray, a three-dimensional or two-dimensional selection box, etc. According to whether the pointed object moves, it can be divided into two categories: static objects (such as buildings, traffic signs, lane lines, natural landforms such as mountains and rivers, etc.) and dynamic objects (such as other vehicles, pedestrians, etc.). These objects are not limited to existing only in the real world. For example, in the future world of virtual and real integration, virtual objects can be pointed, such as virtual reality billboards. According to the purpose of interaction, it can be divided into information query, specified driving route and method, etc. Here are some common interaction scenarios involving pointing at objects outside the vehicle:

[0032] When the vehicle is picking up passengers, point to the target passenger outside the vehicle, such as "Stop next to the lady in the red hat."

[0033] Limit the driving strategy, such as "give way to those two elderly people".

[0034] Ask for information about buildings within the visible range, such as "Is that the Yangpu Bridge?", "Does this office building have parking spaces?", etc.

[0035] When parking temporarily on the side of the road, point to a reference point on the side of the road to indicate exactly where you are parking, such as "park behind this red car."

[0036] When driving on the highway, give instructions to other vehicles on the road, such as "stay away from this truck" or "report that car is driving dangerously."

[0037] While driving, mark the road condition information, such as "report a traffic accident there" or "mark that car as a broken-down vehicle".

[0038] Using targets outside the car as a reference to adjust the driving route is often encountered in self-driving tours. For example, "drive along this river" or "stop by that lake to take a picture."

[0039] In our daily life, the above scenarios are often met by the driver. With the increasing development and popularization of vehicle intelligence and autonomous driving technology, future smart vehicles need to be able to meet such needs, and the first problem is how to determine which object inside or outside the car the passenger is pointing to.

[0040] In view of the above problems, the inventors have discovered after long-term research and proposed the target object determination method, device, electronic device and storage medium provided in the embodiments of the present application. Through the pointing information and selection instructions of the passengers in the vehicle, the target object can be intuitively indicated, which can facilitate the development of subsequent interactive tasks and improve the naturalness and efficiency of the interaction. Among them, the specific method for determining the target object is described in detail in the subsequent embodiments.

[0041] See also Figure 1 , Figure 1 The flowchart of the method for determining a target object provided by an embodiment of the present application is shown. The method is used to intuitively indicate the target object through the pointing information and selection instructions of the occupants in the vehicle, which can facilitate the development of subsequent interactive tasks and improve the naturalness and efficiency of the interaction. In a specific embodiment, the method for determining a target object can be applied to Fig.18 The target object determination device 200 and the electronic device 100 equipped with the target object determination device 200 are shown in FIG. Fig.19 ). The following will take an electronic device as an example to illustrate the specific process of this embodiment. Of course, it can be understood that the electronic device used in this embodiment may include a vehicle, a vehicle system, etc., which is not limited here. Figure 1 The process shown is described in detail, and the method for determining the target object may specifically include the following steps:

[0042] Step S110: Acquire pointing information of a passenger in the vehicle.

[0043] In this embodiment, the occupants in the vehicle may include a driver and / or a passenger. That is, the occupants in the vehicle may include a person in the main driving seat of the vehicle, a person in the auxiliary seat of the vehicle, a person in the back seat of the vehicle, a person standing in the vehicle, etc., which are not limited here.

[0044] In some embodiments, the number of passengers in the vehicle can be obtained, and different ways of obtaining pointing information of passengers in the vehicle can be determined according to the number of passengers in the vehicle. As one way, when the number of passengers in the vehicle is one, the pointing information of the one passenger can be directly obtained. As another way, when the number of passengers in the vehicle is at least two, a target passenger can be determined from the at least two passengers, and the pointing information of the target passenger can be obtained. For example, the driver can be determined as the target passenger from the at least two passengers, and the passenger who meets the preset identity can be determined from the at least two passengers as the target passenger, etc. The preset identity can include the owner identity of the vehicle. As another way, when the number of passengers in the vehicle is at least two, the pointing information of the at least two passengers can be obtained, and the pointing information of one passenger can be selected from the pointing information of the at least two passengers.

[0045] In some embodiments, the occupants in the vehicle can express pointing in a variety of ways. For example: directly pointing to an object with a finger; pointing to an object through a camera of a mobile phone, tablet computer, etc.; selecting an object in the center of the occupant's sight by approximating the orientation of the occupant's face through the orientation of a head-mounted augmented reality (AR) device; selecting an object in the AR device through a virtual ray commonly used by an AR device controller; selecting an object in a virtual reality (VR) device through a virtual ray commonly used by a VR device controller; using eye tracking to take the occupant's gaze point as the direction of the pointing, and obtaining an object in that direction.

[0046] In this embodiment, the pointing information of the occupant in the vehicle can be the pointing information of the occupant in the vehicle in the world coordinate system, that is, the pointing information of the occupant in the vehicle and the object pointed by the vehicle are in the same coordinate system.

[0047] Step S120: determining a pointing area corresponding to the pointing information, and acquiring one or more objects to be selected corresponding to the pointing area.

[0048] In this embodiment, when the pointing information of the passenger in the vehicle is obtained, the pointing area corresponding to the pointing information can be determined, and one or more objects to be selected corresponding to the pointing area can be obtained. The one or more objects to be selected may include but are not limited to: buildings, traffic signs, lane lines, mountains and rivers, other vehicles, pedestrians, virtual reality billboards, and animals.

[0049] In some embodiments, the pointing information can be understood as a ray including a pointing starting point and a pointing direction. The pointing starting point can be defined as a certain position of the occupant, such as the head, the midpoint of the line connecting the two eyes, the chest, etc., or it can be defined as the position of a specific device, such as the position of a mobile phone camera, the position of an AR / VR handle, etc. The pointing direction can be defined as the direction of the line connecting the starting point to the user's fingertips, the direction of the user's face, the direction of the user's gaze, etc. Among them, when obtaining the pointing information (pointing starting point and pointing direction) of the occupants in the vehicle, the area intersecting with the pointing direction can be obtained, and the area intersecting with the pointing direction can be determined as the pointing area.

[0050] In some embodiments, when the pointing area is determined, one or more objects to be selected corresponding to the pointing area can be obtained. As an implementable manner, when the pointing area is determined, one or more objects located in the pointing area can be obtained, and the one or more objects located in the pointing area can be determined as one or more objects to be selected corresponding to the pointing area. As another implementable manner, when the pointing area is determined, one or more objects intersecting with the pointing area can be obtained, and the one or more objects intersecting with the pointing area can be determined as one or more objects to be selected corresponding to the pointing area.

[0051] Step S130: In response to the occupant's selection instruction for the one or more objects to be selected, determining a target object from the one or more objects to be selected.

[0052] Wherein, when one or more objects to be selected are obtained, the occupant can select or determine the one or more objects to be selected to determine the target object from the one or more objects to be selected, wherein, when there are multiple objects to be selected, the target object can be selected and determined based on the selection instruction triggered by the occupant, and when there is one object to be selected, the target object can be determined based on the selection instruction triggered by the occupant. In this embodiment, when one or more objects to be selected are obtained, the occupant's selection instruction for the one or more objects to be selected can be monitored, and when the occupant's selection instruction for the one or more objects to be selected is monitored, the target object can be determined from the one or more objects to be selected in response to the occupant's selection instruction for the one or more objects to be selected.

[0053] In some embodiments, when one or more objects to be selected are obtained, the one or more objects to be selected may be output, and the occupant may trigger a selection instruction for the one or more objects to be selected based on the output one or more objects to be selected, and then the target object may be determined from the one or more objects to be selected in response to the occupant's selection instruction for the one or more objects to be selected. Outputting multiple objects to be selected may include: displaying multiple objects to be selected at the same time, displaying multiple objects to be selected in sequence, and announcing the multiple objects to be selected by voice, etc., which is not limited here.

[0054] As a method, when one or more objects to be selected are obtained, the pointing area (one or more objects to be selected) can be displayed to the occupant. For example, the single object to be selected currently pre-selected by the occupant can be displayed through a display device such as a head-mounted AR device, a screen of a mobile phone, a window of a vehicle, or a preview on a display space. The occupant can change the pre-selected object to be selected by moving fingers, changing lines of sight, turning his head, moving the mobile phone, clicking the screen, etc., and obtain visual feedback, so that the target object can be determined from one or more objects to be selected.

[0055] As another way, when one or more objects to be selected are obtained, the target object can be determined from the one or more objects to be selected by continuous pointing when the vehicle is moving. That is, continuous pointing when the vehicle is moving can be regarded as multiple pointings to the same target object from different angles. By calculating the intersection of one or more objects to be selected in the multiple pointings, the target object can be determined more accurately from the one or more objects to be selected. For example, Figure 2 As shown, during the driving process, the passenger pointed at the car at different positions from t1 to t3. Although the single pointing was blocked by the trash can or the bicycle, causing ambiguity, through multiple pointings, it can be determined that the car is the target object pointed by the passenger.

[0056] As another way, when one or more objects to be selected are obtained, the target object can be determined from the one or more objects to be selected using the user's voice command. That is, the user's voice command often carries a description of the target object, such as a description of the type (vehicle, vehicle, sign, building, etc.), a description of the position (relative position, etc.), and a description of the behavior (whether moving, movement trajectory, etc.). By parsing the voice command, the features of these target objects can be extracted and compared with the feature information of the one or more objects to be selected, so that the target object can be effectively screened out from the one or more objects to be selected.

[0057] As another way, when one or more objects to be selected are obtained, the occupant can be asked about the target object to be pointed at through a voice assistant, or a combination of voice and display, and the occupant's reply based on the inquiry can be received to determine the target object from one or more objects to be selected according to the reply. For example, when the occupant points to other vehicles, if there are multiple other vehicles at hand, it will cause occlusion and interference. The voice assistant can seek to confirm the differentiated characteristics of the vehicle, so that the user can select the target more accurately. For example, the user can be asked "Is it the blue SUV with the Beijing A license plate that is turning?" At the same time, the image of the vehicle is displayed on the vehicle's computer, or the actual position of the vehicle is directly displayed through AR glasses. In this example, multiple information such as the type of target object (SUV vehicle), appearance characteristics (blue), semantic information (Beijing A license plate), and motion state (turning) are included, so that the target object can be effectively determined from one or more objects to be selected.

[0058] A method for determining a target object provided by an embodiment of the present application obtains pointing information of an occupant in a vehicle, determines a pointing area corresponding to the pointing information, and obtains one or more objects to be selected corresponding to the pointing area, and determines the target object from the one or more objects to be selected in response to the occupant's selection instructions for the one or more objects to be selected, thereby intuitively indicating the target object through the pointing information and selection instructions of the occupant in the vehicle, which can facilitate the development of subsequent interactive tasks and improve the naturalness and efficiency of the interaction.

[0059] See also Figure 3 , Figure 3 The flowchart of the method for determining the target object provided by an embodiment of the present application is shown in FIG. Figure 3 The process shown is described in detail, and the method for determining the target object may specifically include the following steps:

[0060] Step S210: Acquire pointing information of a passenger in the vehicle.

[0061] Step S220: determining a pointing area corresponding to the pointing information, and acquiring a plurality of objects to be selected corresponding to the pointing area.

[0062] For the detailed description of step S210 to step S220, please refer to step S110 to step S120, which will not be repeated here.

[0063] Step S230: displaying the pointing area to the occupant, and identifying a first to-be-selected object preselected from the plurality of to-be-selected objects in the pointing area by identification information.

[0064] In some embodiments, when multiple objects to be selected are obtained, one object to be selected can be pre-selected from the multiple objects to be selected as the first object to be selected, wherein the first object to be selected can be an object to be selected randomly selected from the multiple objects to be selected, can be an object to be selected from the multiple objects to be selected that is evaluated as the most likely to be pointed at by the occupant, or can be an object to be selected from the multiple objects to be selected that is closest to the occupant, etc., and is not limited here.

[0065] In some embodiments, when the pre-selected first object to be selected is determined, the pointing area can be displayed to the occupant, and the first object to be selected can be identified in the pointing area by identification information. As one way, the identification information can include a ray, and the first object to be selected can be pointed to by the ray in the pointing area. As another way, the identification information can include a target symbol, and the target symbol can be added to the first object to be selected in the pointing area. As another way, the identification information can include a target selection control, and the first object to be selected can be selected by the target selection control in the pointing area, etc., which are not limited here.

[0066] In some implementations, the pointing area may be displayed through a display device or display space such as an AR head display, a mobile phone screen, or a vehicle window.

[0067] Step S240: upon receiving the object switching instruction triggered by the occupant, determining a second object to be selected corresponding to the object switching instruction from the plurality of objects to be selected.

[0068] In this embodiment, during the process of displaying the pointing area, the instructions triggered by the occupant based on the pointing area can be monitored, and when the instructions triggered by the occupant based on the pointing area are monitored, the operation corresponding to the instruction can be executed.

[0069] In some embodiments, during the process of displaying the pointing area, the instructions triggered by the occupant based on the pointing area can be monitored. In the case of monitoring the object switching instruction triggered by the occupant, the second object to be selected corresponding to the object switching instruction can be determined from multiple objects to be selected in response to the object switching instruction triggered by the occupant. Among them, if it is detected that the occupant moves his finger, it can be determined that the occupant triggers the object switching instruction; if it is detected that the occupant changes his line of sight, it can be determined that the occupant triggers the object switching instruction; if it is detected that the occupant turns his head, it can be determined that the occupant triggers the object switching instruction; if it is detected that the occupant moves his mobile phone, it can be determined that the occupant triggers the object switching instruction; if it is detected that the occupant clicks the screen, it can be determined that the occupant triggers the object switching instruction, etc., which are not limited here.

[0070] In some embodiments, when an object switching instruction triggered by an occupant is monitored, the object switching instruction can be parsed to obtain the object to be selected indicated by the object switching instruction, and the object to be selected indicated by the object switching instruction is determined as the second object to be selected. For example, if the object switching instruction is triggered by the movement of the occupant's finger, the object to be selected indicated by the finger movement can be obtained, and the object to be selected indicated by the finger movement can be determined as the second object to be selected; if the object switching instruction is triggered by the occupant clicking the screen, the object to be selected corresponding to the position where the screen is clicked can be obtained, and the object to be selected corresponding to the position where the screen is clicked can be determined as the second object to be selected.

[0071] In some embodiments, during the process of displaying the pointing area, commands triggered by the occupant based on the pointing area can be monitored. When an object confirmation command triggered by the occupant is monitored, the first object to be selected can be determined as the target object, wherein the object confirmation command can be triggered by the occupant's voice, nod, blink, click the screen, etc., which are not limited here.

[0072] Step S250: In the pointing area, the identification information is switched from identifying the first object to be selected to identifying the second object to be selected, and the second object to be selected is determined as the target object.

[0073] In this embodiment, when the second object to be selected is determined, the identification information can be switched from identifying the first object to be selected to identifying the second object to be selected in the pointing area, and the second object to be selected can be determined as the target object. As one way, when the identification information is a ray, the ray pointing to the first object to be selected can be switched to pointing to the second object to be selected in the pointing area. As another way, when the identification information is a target symbol, the target symbol added to the first object to be selected can be switched to being added to the second object to be selected in the pointing area. As another way, when the identification information is a target box selection control, the target box selection control boxed on the first object to be selected can be switched to boxed on the second object to be selected in the pointing area, etc., which are not limited here.

[0074] like Figure 4 As shown, in the interaction of AR glasses, the object to be selected (the first object to be selected) initially selected by the occupant is a car, marked by ray A. When the occupant moves his finger, the ray also moves accordingly, and the object to be selected (the second object to be selected) selected as the target object following the movement of the finger is a bicycle, marked by ray B.

[0075] As an implementable manner, when the identification information is switched from identifying the first object to be selected to identifying the second object to be selected, the second object to be selected can be directly determined as the target object.

[0076] As another feasible method, when the identification information is switched from identifying the first object to be selected to identifying the second object to be selected, the instructions triggered by the occupant based on the pointing area can be monitored, and when the object confirmation instruction triggered by the occupant is monitored, the second object to be selected can be determined as the target object.

[0077] Compared with the method for determining the target object provided by an embodiment of the present application, Figure 1 The method for determining the target object shown in the figure, this embodiment also displays a pointing area to the occupant, and identifies a pre-selected first object to be selected by identification information in the pointing area. When an object switching instruction triggered by the occupant is received, the second object to be selected corresponding to the object switching instruction can be determined. In the pointing area, the identification information is switched from identifying the first object to be selected to identifying the second object to be selected, and the second object to be selected is determined as the target object, thereby improving the accuracy of target object determination and enhancing the user's interactive experience.

[0078] See also Figure 5 , Figure 5 The following is a flow chart of a method for determining a target object provided by an embodiment of the present application. Figure 5 The process shown is described in detail, and the method for determining the target object may specifically include the following steps:

[0079] Step S310: Acquire pointing information of a passenger in the vehicle.

[0080] Step S320: determining a pointing area corresponding to the pointing information, and acquiring one or more objects to be selected corresponding to the pointing area.

[0081] For the detailed description of step S310 to step S320, please refer to step S110 to step S120, which will not be repeated here.

[0082] Step S330: Acquire target feature information corresponding to each of the one or more objects to be selected, wherein the target feature information corresponding to each of the one or more objects to be selected is different.

[0083] In this embodiment, when one or more objects to be selected are obtained, target feature information corresponding to each of the one or more objects to be selected may be obtained, wherein the target feature information corresponding to each of the one or more objects to be selected is different.

[0084] In some embodiments, when one or more objects to be selected are obtained, feature information corresponding to each of the one or more objects to be selected can be extracted respectively to obtain feature information corresponding to each of the one or more objects to be selected. When feature information corresponding to each of the one or more objects to be selected is obtained, feature information corresponding to each of the one or more objects to be selected can be compared to obtain target feature information corresponding to each of the one or more objects to be selected, that is, differentiated feature information corresponding to each of the one or more objects to be selected (used to distinguish feature information of other objects to be selected) is obtained.

[0085] Step S340: output inquiry information to the occupant based on the target feature information corresponding to each of the one or more objects to be selected.

[0086] In this embodiment, when the target feature information corresponding to one or more objects to be selected is obtained, query information can be output to the occupant based on the target feature information corresponding to one or more objects to be selected. For example, the query information "Is it the blue SUV with the Beijing A license plate that is turning?" is output.

[0087] As one way, when the target feature information corresponding to one or more objects to be selected is obtained, the inquiry information in the form of voice can be output to the occupant through the voice assistant based on the target feature information corresponding to one or more objects to be selected. As another way, when the target feature information corresponding to one or more objects to be selected is obtained, the inquiry information can be output to the occupant in the form of displaying the image of the object to be selected based on the target feature information corresponding to one or more objects to be selected. As another way, when the target feature information corresponding to one or more objects to be selected is obtained, the inquiry information can be output to the occupant in the form of voice by displaying the image of the object to be selected while outputting the inquiry information in the form of voice to the occupant through the voice assistant based on the target feature information corresponding to one or more objects to be selected.

[0088] Step S350: In response to the occupant's response information to the inquiry information, determining the target object from the one or more objects to be selected.

[0089] In this embodiment, during the process of outputting inquiry information to the occupant, the occupant's reaction to the inquiry information can be detected. When response information of the occupant to the inquiry information is detected, a target object can be determined from one or more objects to be selected in response to the occupant's response information to the inquiry information.

[0090] In some implementations, when the response information characterizes a certainty, such as when the response information is “right”, “yes”, or “correct”, the object to be selected corresponding to the inquiry information may be determined as the target object.

[0091] In some implementations, when the response information indicates a negative answer, such as when the response information is “wrong”, “no”, or “uncertain”, the inquiry information may be re-output to the occupant.

[0092] Compared with the method for determining the target object provided by an embodiment of the present application, Figure 1 The method for determining the target object shown in the figure, this embodiment also obtains differentiated feature information corresponding to each of the one or more objects to be selected, outputs inquiry information to the occupant based on the differentiated feature information corresponding to each of the one or more objects to be selected, and determines the target object from the one or more objects to be selected in response to the occupant's response information to the inquiry information, thereby improving the accuracy of target object determination and enhancing the user's interactive experience.

[0093] See also Figure 6 , Figure 6 FIG. 1 is a flow chart showing a method for determining a target object provided by an embodiment of the present application. In this embodiment, the pointing information includes a pointing starting point and a pointing direction. Figure 6 The process shown is described in detail, and the method for determining the target object may specifically include the following steps:

[0094] Step S410: Acquire the pointing starting point and pointing direction of the occupant in the vehicle.

[0095] In this embodiment, the pointing information includes a pointing starting point and a pointing direction.

[0096] Step S420: constructing a conical area based on the pointing starting point and the pointing direction as the pointing area.

[0097] In some implementations, when the pointing starting point and pointing direction are obtained, the target object can be determined based on the pointing starting point and pointing direction. Wherein, when the pointing starting point and pointing direction are obtained, it can be detected whether there are intersecting objects in the pointing direction, and when it is determined that there is only one intersecting object in the pointing direction, the one intersecting object can be determined as the target object; when it is determined that there are multiple intersecting objects in the pointing direction, the object closest to the pointing starting point among the multiple intersecting objects can be determined as the target object.

[0098] It is understandable that the pointing direction of the passenger is not necessarily completely correct and is easily affected by the bumps of the vehicle, which will further reduce the accuracy of the pointing. Therefore, in order to improve the accuracy of pointing, a conical area can be constructed based on the pointing starting point and the pointing direction, and the constructed conical area is determined as the pointing area. The conical area can be a selection cone constructed around the pointing direction, which can be imagined as a light cone emitted by a flashlight. The size of the conical area can be changed, for example, the size of the conical area can be changed based on the user's operation, can be changed based on changes in the environment, etc., which are not limited here.

[0099] Step S430: Acquire a plurality of objects that intersect with the conical area and are not blocked as the one or more objects to be selected corresponding to the pointing area.

[0100] In some embodiments, when a conical area is determined, multiple objects that intersect with the conical area and are not obstructed can be obtained, and the multiple objects that intersect with the conical area and are not obstructed can be determined as one or more objects to be selected corresponding to the pointing area.

[0101] The objects intersecting with the conical region may include: objects located within the conical region, or objects intersecting with the edge of the conical region.

[0102] In some embodiments, being unobstructed may include: an occlusion ratio is less than a ratio threshold, wherein the ratio threshold may be updated as required, and an object whose occlusion ratio is less than the ratio threshold may be determined as an unobstructed object. For example, if the ratio threshold is 10%, an object whose occlusion ratio is less than 10% may be determined as an unobstructed object; if the ratio threshold is 5%, an object whose occlusion ratio is less than 5% may be determined as an unobstructed object, etc., which are not limited here.

[0103] In some embodiments, being unobstructed may include: an obstructed area is less than an area threshold, wherein the area threshold may be updated as required, and an object whose obstruction area is less than the area threshold may be determined as an unobstructed object. For example, if the area threshold is 5 square meters, an object whose obstruction area is less than 5 square meters may be determined as an unobstructed object; if the area threshold is 3 square meters, an object whose obstruction area is less than 3 square meters may be determined as an unobstructed object, etc., which is not limited here.

[0104] like Figure 7 As shown in the figure, the cone is a selection cone built around the pointing direction. It can be imagined as a light cone emitted by a flashlight. The illuminated target can be used as the object to be selected. Figure 7In the example, houses and vehicles are considered as objects to be selected because they intersect with the conical area, while cyclists and bicycles are not considered as objects to be selected because they do not intersect with the conical area.

[0105] Step S440: In response to the occupant's selection instruction for the one or more objects to be selected, determining a target object from the one or more objects to be selected.

[0106] For the detailed description of step S440, please refer to step S130, which will not be repeated here.

[0107] Compared with the method for determining the target object provided by an embodiment of the present application, Figure 1 The method for determining the target object shown in the figure, this embodiment also obtains the pointing starting point and pointing direction of the occupant in the vehicle, constructs a conical area as the pointing area based on the pointing starting point and the pointing direction, and obtains multiple objects that intersect with the conical area and are not blocked as one or more objects to be selected corresponding to the pointing area, thereby improving the accuracy of the obtained pointing information, and further improving the accuracy of the target object determination.

[0108] See also Figure 8 , Figure 8 The following is a flow chart of a method for determining a target object provided by an embodiment of the present application. Figure 8 The process shown is described in detail, and the method for determining the target object may specifically include the following steps:

[0109] Step S510: Acquire the pointing information of the occupant in the coordinate system of the detection device detected by the detection device located in the vehicle.

[0110] In this embodiment, a detection device may be provided in the vehicle, and the pointing information of the occupant may be detected by the detection device to obtain the pointing information of the occupant. It can be understood that the pointing information of the occupant detected by the detection device is the pointing information of the occupant in the detection device coordinate system corresponding to the detection device. Therefore, the pointing information of the occupant detected by the detection device located in the vehicle in the detection device coordinate system can be obtained.

[0111] In some embodiments, the detection device may include one or a combination of a depth camera, a motion capture system, a head-mounted augmented reality device, an eye tracker, an augmented reality handle, a virtual reality handle, and a mobile terminal.

[0112] Among them, from the perspective of the occupant, the essence of pointing information is to define a ray, and the object intersecting with the ray is the target object being pointed at. The two key information for defining this ray are the pointing starting point and the pointing direction. The starting point of the ray can be defined as a certain position of the human body, such as the head, the midpoint of the line connecting the two eyes, the chest, etc., or it can be defined as the position of a specific device, such as the position of a mobile phone camera, the position of an AR / VR handle, etc. The direction of the ray can be defined as the line from the starting point to the fingertips of the occupant, the orientation of the user's face, the direction of the user's gaze, etc. Table 1 lists the definitions and detection methods of the pointing starting point and pointing direction in several pointing behaviors, as shown in Table 1:

[0113]

[0114] Table 1

[0115] The methods involved in Table 1 are described in detail below.

[0116] As a first method, take the detection device as a depth camera as an example.

[0117] Depth cameras can obtain the distance between the sampling point on the surface of an object and the sensor within a certain viewing angle. Early depth cameras mainly used the principle of structured light for depth perception, while current depth cameras often use the principle of Time of Flight (TOF) for depth perception. When combined with human key point recognition and tracking algorithms, depth cameras can be used to identify the position and orientation of the user's head (such as Fig. 9 as shown) and fingers (as Fig.10 As shown in the figure), two types of pointing behaviors, head pointing and eye-hand connection, can be detected.

[0118] As a second method, take the detection device as a motion capture system as an example.

[0119] Motion capture systems perform similar functions to depth cameras and can track objects including human bodies. Fig.11 As shown in the figure, its core component is a set of infrared cameras C with infrared light sources. Before use, these cameras need to be arranged around the required space and their relative positions and other system parameters need to be calibrated. Fig.12 As shown, when in use, a number of specially made retroreflective markers D are attached to the surface of an object, and these markers can be identified and tracked by a multi-camera system, thereby tracking the object. Similar to the depth camera, the motion capture system can also detect two types of pointing behaviors: head pointing and eye-hand connection.

[0120] As a third method, take the detection device as a head-mounted augmented reality device as an example.

[0121] Among them, the head-mounted augmented reality device may include AR glasses. The head-mounted augmented display device includes a display component and various sensors. The display component is often near the user's eyes. One of the most important sensors is a number of inertial measurement units (IMUs), which can detect the posture information of the head-mounted augmented reality device and then determine the posture information of the user's head. The morphological design of the head-mounted augmented reality device defines how the occupant needs to wear it. The spatial relationship of the head position, eye position, and facial orientation relative to the head-mounted augmented reality device is relatively fixed and can be calibrated in advance more accurately, so the head-mounted augmented reality device can support pointing behavior of head pointing.

[0122] Another important sensor of head-mounted augmented reality devices is a number of cameras, including recognition, positioning and tracking of user fingers. Introducing multiple cameras can form a binocular or even trio of camera groups, thereby achieving more accurate positioning. Using the position of the user's fingers, head-mounted augmented reality devices can support eye-hand pointing behavior.

[0123] If an eye-tracking sensor is embedded in the device, the head-mounted augmented reality device can even track the user's gaze direction, thereby supporting pointing behavior of looking at the target object.

[0124] As a fourth method, take the detection device as an eye tracker as an example.

[0125] Among them, the eye tracker can be a desktop eye tracker. Desktop eye trackers generally use optical methods to detect the gaze direction of the occupant. The principle is to use a set of infrared light sources with known positions to illuminate the eyes, and use an infrared camera to obtain the image of the occupant's eyes. By analyzing the relative position of the reflection point of the infrared light source on the eyeball and the center point of the pupil, the direction of eyeball rotation, that is, the direction of the occupant's gaze, is obtained. For desktop eye trackers, in order to allow users to move their heads freely, binocular infrared cameras are often used to obtain the position of the user's eyes. Combining the eye position and the gaze direction can support pointing behavior of looking at the target object.

[0126] The eye tracker may be a glasses-type eye tracker. Fig.13 As shown in the figure, the principle of the glasses-type eye tracker is the same as that of the desktop eye tracker, but because it adopts the device form of glasses, the position of the camera and the occupant's eyes is relatively fixed, and the eye position can be calibrated in advance, which simplifies the calculation difficulty of real-time tracking. At the same time, because of its wearability, the glasses-type eye tracker can support pointing behavior of gazing at the target, and its eye tracking capability can be embedded in head-mounted devices such as AR glasses. Its main components are eight micro infrared light sources and two micro cameras for each eye.

[0127] As a fifth method, take the detection device as an augmented reality handle / virtual reality handle as an example.

[0128] Among them, Fig.14 As shown in Figure 1, AR / VR controllers generally have built-in IMUs to sense the spatial position of the AR / VR controllers. VR controllers can contain special infrared LED light strips, whose positions can be identified and tracked by infrared sensors or cameras on VR headsets. The position and orientation of the controller can define a ray to support the operator's pointing behavior with the controller.

[0129] As a sixth method, take the detection device as a mobile terminal as an example.

[0130] The camera of the mobile terminal can also be used for direction indication. In the simplest implementation, the sensor center point of the camera can be defined as the starting point of pointing, and the optical axis of the camera, that is, the axis connecting the sensor center point and the optical center point of the lens, can be defined as the direction of pointing. The occupant can point to different target objects by controlling the movement of the mobile terminal.

[0131] See also Fig.15 , Fig.15 This application shows Figure 8 In the present embodiment, the detection device includes a head-mounted augmented reality device. Fig.15 The process shown is described in detail, and the method may specifically include the following steps:

[0132] Step S511A: When the head-mounted augmented reality device is worn on the head of the occupant, determine the position information of the head of the occupant in the coordinate system of the head-mounted augmented reality device.

[0133] In this embodiment, the detection device is a head-mounted enhanced display device.

[0134] In some embodiments, it is possible to detect whether the head-mounted augmented reality device is in a wearing state, wherein, when it is detected that the head-mounted augmented reality device is worn on the head of the occupant, the position information of the occupant's head in the coordinate system of the head-mounted augmented reality device can be determined. As a method, the head-mounted augmented reality device can include multiple IMUs, and the head-mounted augmented reality device can detect the position information of the head-mounted augmented reality device through multiple IMUs, and determine the position information of the occupant's head in the head-mounted augmented reality coordinate system based on the detected position information of the head-mounted augmented reality device and the relative position relationship between the head-mounted augmented reality device and the occupant's head.

[0135] As an implementable method, the relative position relationship between the head-mounted augmented reality device and the occupant's head can be relatively fixed and pre-calibrated. Then, based on the posture information of the head-mounted augmented reality device and the relative position relationship between the head-mounted augmented reality device and the occupant's head, the posture information of the occupant's head in the coordinate system of the head-mounted augmented reality device can be determined.

[0136] Step S512A: Based on the posture information of the occupant's head in the coordinate system of the head-mounted augmented reality device, determine the pointing information of the occupant in the coordinate system of the head-mounted augmented reality device.

[0137] In this embodiment, when the position information of the occupant's head in the coordinate system of the head-mounted augmented reality device is determined, the pointing information of the occupant in the coordinate system of the head-mounted augmented reality device can be determined based on the position information of the occupant's head in the coordinate system of the head-mounted augmented reality device. As an implementable method, the occupant's face can be used to perform the pointing behavior, that is, the occupant's face can be used as the pointing starting point, and the occupant's facial orientation can be used as the pointing direction. Therefore, when the position information of the occupant's head in the coordinate system of the head-mounted augmented reality device is obtained, the occupant's facial position and facial orientation can be obtained to determine the occupant's pointing information in the coordinate system of the head-mounted augmented reality device based on the facial position and facial orientation.

[0138] As an application scenario, when the occupant is wearing a head-mounted augmented reality device, the occupant's face can be used as the starting point for pointing, the occupant's facial orientation can be used as the pointing direction, and the occupant's corresponding pointing information can be determined to point at the object. If the occupant's head turns, the occupant's turned face can be used as the pointing point, and the occupant's turned facial orientation can be used as the pointing direction to determine the occupant's corresponding pointing information to point at the object. As an implementable method, if the occupant's head turns, the occupant's head rotation angle can be obtained. If the rotation angle is greater than an angle threshold, the occupant's turned face can be used as the pointing point, and the occupant's turned facial orientation can be used as the pointing direction to determine the occupant's corresponding pointing information to point at the object; if the rotation angle is less than or equal to the angle threshold, the original pointing information can be maintained to point at the object.

[0139] See also Fig.16 , Fig.16 This application shows Figure 8 In the present embodiment, the detection device includes a mobile terminal. Fig.16 The process shown is described in detail, and the method may specifically include the following steps:

[0140] Step S511B: When the mobile terminal is in a gripping state and the camera is used to capture images, the center point of the camera and the optical center axis of the camera are obtained.

[0141] In this embodiment, the detection device is a mobile terminal, which may include a mobile phone, a tablet computer, etc., which is not limited here.

[0142] In some embodiments, it is possible to detect whether the mobile terminal is in a held state and image acquisition is performed through the camera. When it is detected that the mobile terminal is in a held state and image acquisition is performed through the camera, the center point of the camera and the optical center axis of the camera, that is, the axis connecting the center point of the sensor and the optical center point of the lens, can be obtained.

[0143] As an implementable manner, the number of cameras included in the mobile terminal may be one or more. When the number of cameras included in the mobile terminal is one, the center point and the optical center axis of the one camera may be directly obtained; when the number of cameras included in the mobile terminal is multiple, the center points of the multiple cameras may be determined, and the center points of the multiple cameras may be calculated based on the center points of the multiple cameras; the optical center axes of the multiple cameras may be determined, and the optical center axes of the multiple cameras may be calculated based on the optical center axes of the multiple cameras.

[0144] Step S512B: Based on the center point of the camera and the optical center axis of the camera, determine the pointing information of the occupant in the mobile terminal coordinate system.

[0145] In this embodiment, when the center point of the camera and the optical axis of the camera are obtained, the pointing information of the occupant in the coordinate system of the mobile terminal can be determined based on the center point of the camera and the optical axis of the camera. As an implementable manner, when the center point of the camera is obtained, the center point of the camera can be determined as the pointing starting point, and when the optical axis of the camera is obtained, the optical axis can be determined as the pointing direction, that is, the pointing information including the pointing starting point and the pointing direction can be obtained.

[0146] Step S520: Acquire the position and posture information of the vehicle in the world coordinate system, and acquire the position and posture information of the detection device in the vehicle coordinate system.

[0147] In this embodiment, the position and posture information of the vehicle in the world coordinate system can be obtained.

[0148] Among them, the perception capability of intelligent vehicles is a key component of autonomous driving technology, which outputs the vehicle's position in the world (i.e. positioning capability), traffic elements (such as lane lines, traffic lights, signs, road arrows, etc.), drivable areas, and dynamic traffic participant information (such as other vehicles, pedestrians and their direction, speed, behavior, etc.).

[0149] In some embodiments, common positioning technologies used in autonomous driving or assisted driving technologies include: global navigation satellite system, inertial navigation, and scan matching. These technologies are used in combination to provide the position and orientation of the vehicle body in the world coordinate system.

[0150] The Global Navigation Satellite System (GNSS) is a general term for a type of positioning system. Its principle is to detect the distance between the vehicle and multiple satellites. Since the positions of these satellites are known, the precise position of the vehicle on the surface of the earth can be calculated. The Global Positioning Satellite System (GPS), Beidou System, Quasi-Zenith Satellite System, etc., all belong to the category of GNSS. Due to factors such as electromagnetic interference in satellite signal propagation, satellite time errors, and noise inside components, GNSS positioning has large errors in practical use and cannot achieve centimeter-level positioning accuracy. The use of carrier phase differential technology (Real Time Kinematic, RTK) can eliminate some of these errors and achieve centimeter-level positioning accuracy. The use of GNSS requires good reception of electromagnetic wave signals from satellites, so positioning cannot be achieved underground or when blocked by buildings.

[0151] Inertial Navigation System (INS) uses the linear acceleration and angular velocity information provided by IMU to accurately estimate the relative displacement and rotation of the vehicle in a short period of time, thereby better supplementing the shortcomings of GNSS positioning.

[0152] Scan Matching is derived from the classic algorithm in the field of robotics, Simultaneous Localization and Mapping (SLAM). The core idea of ​​SLAM is to allow the robot to start from an unknown location in an unknown environment, locate its own position and posture through repeated observations of map features (such as corners, pillars, etc.) during movement, and then incrementally build a map based on its own position, thereby achieving the purpose of simultaneous positioning and map construction. In autonomous driving applications, the scope of use of autonomous driving is often limited to paved road networks within or between cities, so the mapping part of the SLAM algorithm can be pre-placed, that is, through special acquisition vehicles and high-precision sensors (sensors facing outside the vehicle for autonomous driving), high-precision maps can be pre-built, for example, including segmentation and marking of traffic elements and landmark information.

[0153] During the driving process of smart vehicles, their sensors (sensors facing outside the vehicle for autonomous driving), such as lidar, millimeter-wave radar, binocular cameras, etc., will also generate a large amount of real-time data. By comparing it with the information in the high-precision map, it is possible to calculate which segment of data in the high-precision map best matches the real-time data, thereby knowing the vehicle's position and posture in the high-precision map, that is, obtaining the vehicle's position and posture information in the world coordinate system.

[0154] The spatial positioning result of the vehicle expresses the position and posture (i.e., pose) of the vehicle in three-dimensional space, where the position is usually represented by three-dimensional coordinate points, and the orientation is usually expressed by quaternions, three-dimensional orthogonal matrices, Euler angles, etc. The specific mathematical expressions and calculation methods are public knowledge and will not be expanded here.

[0155] The vehicle positioning result described can be expressed in the form of a matrix. Let R w is the rotation transformation matrix from the vehicle coordinate system to the world coordinate system, t w is the displacement vector from the vehicle coordinate system to the world coordinate system after rotation, then the transformation matrix M from the vehicle coordinate system to the world coordinate system can be defined w :

[0156]

[0157] For the three-dimensional space discussed in this embodiment, R w is a third-order matrix, t w is a three-dimensional vector, M w is a fourth-order matrix. Using M wA matrix that converts a point or vector expressed in the vehicle coordinate system to an expression in the world coordinate system. When converting a point [x,y,z] in the vehicle coordinate system, 1 is appended to it to construct a four-dimensional vector [x,y,z,1] so that the displacement between the two coordinate systems can be applied in matrix multiplication. When converting a vector [x,y,z] in the vehicle coordinate system, 0 is appended to it to construct a four-dimensional vector [x,y,z,0] so that the displacement between the two coordinate systems can be ignored in matrix multiplication and only the rotation transformation can be applied.

[0158] In this embodiment, the position and posture information of the detection device in the vehicle coordinate system can be obtained.

[0159] The positioning result of the detection device in the vehicle can be expressed in the form of a matrix. c is the rotation transformation matrix from the detection device coordinate system to the vehicle coordinate system, t c is the displacement vector from the detection device coordinate system to the vehicle coordinate system after rotation, then the transformation matrix M from the detection device coordinate system to the vehicle coordinate system can be defined c :

[0160]

[0161] For the three-dimensional space discussed in this embodiment, R c is a third-order matrix, t c is a three-dimensional vector, M c is a fourth-order matrix. Using M c A matrix that can transform a point or vector expressed in the detection device coordinate system into an expression in the vehicle coordinate system. When transforming a point [x,y,z] in the detection device coordinate system, 1 is appended to it to construct a four-dimensional vector [x,y,z,1] so that the displacement between the two coordinate systems can be applied in the matrix multiplication. When transforming a vector [x,y,z] in the detection device coordinate system, 0 is appended to it to construct a four-dimensional vector [x,y,z,0] so that the displacement between the two coordinate systems can be ignored in the matrix multiplication and only the rotation transformation can be applied.

[0162] In some implementations, the detection device can be divided into two categories according to whether it is fixed in the vehicle.

[0163] As a method, if the detection device is fixed in the car, such as a depth camera, a motion capture system, or a desktop eye tracker, the position of the detection device relative to fixed feature points in the car can be directly and accurately measured, such as the center point of the upper edge of the rear window, the center line / upper end / lower end of the B / C pillar, etc., so as to determine the position of the detection device in the car, that is, to determine t c To determine R c, we can install accelerometers aligned with the corresponding coordinate systems on the vehicle and the detection device, and measure the direction of acceleration (i.e. gravity) when they are stationary. By the different directions of gravity acceleration in two different coordinate systems, we can calculate the rotation relationship between the two coordinate systems. Suppose the gravity acceleration vector measured in the vehicle coordinate system is g c , measured in the detection device coordinate system as g s , the relationship between the two satisfies g c =R c ·g s , R can be calculated by matrix operation and the properties of the third-order rotation transformation matrix (third-order orthogonal matrix with determinant 1) c Based on this, the position and posture information of the detection device in the vehicle coordinate system can be determined.

[0164] As another way, if the position of the detection device is not fixed relative to the vehicle body, such as AR glasses, eye trackers, mobile phones, etc., the SLAM algorithm (see the previous introduction to the SLAM algorithm) can be used to analyze the continuous video data obtained by the camera of the detection device, identify and track fixed feature points in the car, and thus detect the position of the detection device itself in the cabin in real time. Another method to obtain the position information of a non-fixed detection device relative to the vehicle body is to use the principle of motion capture. Reflective marks are attached to the detection device, and its position can be located through the fixed motion capture system in the car (see the previous paragraph for the method for determining the position of a fixed detection device). Both methods directly output M c Based on this, the position and posture information of the detection device in the vehicle coordinate system can be determined.

[0165] Step S530: Based on the posture information of the vehicle in the world coordinate system and the posture information of the detection device in the vehicle coordinate system, the pointing information of the occupant in the detection device coordinate system is converted into the pointing information of the occupant in the world coordinate system.

[0166] In this embodiment, after obtaining the position information of the vehicle in the world coordinate system and the position information of the detection device in the vehicle coordinate system, the pointing information of the occupant in the detection device coordinate system can be converted to the pointing information of the occupant in the world coordinate system based on the position information of the vehicle in the world coordinate system and the position information of the detection device in the vehicle coordinate system.

[0167] In some embodiments, the above method has constructed a conversion matrix from the detection device coordinate system to the vehicle coordinate system and a conversion matrix from the vehicle coordinate system to the world coordinate system, and obtained the pointing information of the occupant in the detection device coordinate system. In summary, the pointing information in the detection device coordinate system can be converted to the vehicle coordinate system and the world coordinate system in turn. Assume that the pointing information obtained by the detection device includes the pointing starting point ps and pointing direction v s Then the following matrix can be used to calculate the pointing information in the world coordinate system (pointing starting point p w 、Pointing direction v w ):

[0168]

[0169]

[0170] Step S540: determining a pointing area corresponding to the pointing information, and acquiring one or more objects to be selected corresponding to the pointing area.

[0171] Step S550: In response to the occupant's selection instruction for the one or more objects to be selected, determining a target object from the one or more objects to be selected.

[0172] For the detailed description of step S540 to step S550, please refer to step S120 to step S130, which will not be repeated here.

[0173] Compared with the method for determining the target object provided by an embodiment of the present application, Figure 1 The method for determining the target object shown in the figure, this embodiment also obtains the pointing information of the occupant in the detection device coordinate system detected by the detection device located in the vehicle, obtains the posture information of the vehicle in the world coordinate system, and obtains the posture information of the detection device in the vehicle coordinate system, and based on the posture information of the vehicle in the world coordinate system and the posture information of the detection device in the vehicle coordinate system, the pointing information of the occupant in the detection device coordinate system is converted to the pointing information of the occupant in the world coordinate system, so that through the conversion of the coordinate system, the position of the pointing information in the world coordinate system can be accurately obtained to improve the accuracy of target object determination.

[0174] See also Fig.17 , Fig.17 The following is a flow chart of a method for determining a target object provided by an embodiment of the present application. Fig.17 The process shown is described in detail, and the method for determining the target object may specifically include the following steps:

[0175] Step S610: Acquire pointing information of a passenger in the vehicle.

[0176] Step S620: determining a pointing area corresponding to the pointing information, and acquiring one or more objects to be selected corresponding to the pointing area.

[0177] Step S630: In response to the occupant's selection instruction for the one or more objects to be selected, determining a target object from the one or more objects to be selected.

[0178] For the detailed description of step S610 to step S630 , please refer to step S110 to step S130 , which will not be repeated here.

[0179] Step S640: Control the vehicle to perform an operation associated with the target object.

[0180] In this embodiment, when the target object is determined, the vehicle can be controlled to perform operations associated with the target object to achieve intelligent control of the vehicle.

[0181] In some embodiments, controlling the vehicle to perform an operation associated with the target object may include: querying a landmark outside the vehicle. Based on the current method, the occupant needs to open the map software of the electronic device, locate the location of the vehicle, identify the direction, and then try to find the location of the corresponding landmark on the map for query. Based on this solution, the occupant can directly point to the landmark outside the vehicle with a finger and query by voice gesture or voice.

[0182] In some embodiments, controlling the vehicle to perform an operation associated with a target object may include: "holding" a landmark outside the vehicle in hand and interacting with an information system inside the vehicle. The occupant points to a landmark outside the vehicle and drags it to the vehicle display device to save the landmark building information or add a navigation route.

[0183] In some embodiments, controlling the vehicle to perform operations associated with the target object may include: in the two-dimensional information control of the flat map, you can query landmark names, search for businesses in the building, and record and upload road conditions; in the three-dimensional information space perceived by the vehicle in real time, you can track target passengers, complete pick-up tasks, etc., without limitation here.

[0184] In some embodiments, controlling the vehicle to perform operations associated with the target object may include: opening the vehicle's sunroof, closing the vehicle's sunroof, opening the vehicle's doors, closing the vehicle's doors, etc., which are not limited here.

[0185] Compared with the method for determining the target object provided by an embodiment of the present application, Figure 1 The target object determination method shown in the figure, this embodiment also controls the vehicle to perform operations associated with the target object, thereby realizing automatic control of the vehicle and improving the user's driving experience.

[0186] See also Fig.18 , Fig.18The module block diagram of the target object determination device provided by an embodiment of the present application is shown. Fig.18 The block diagram shown in FIG. 1 is used to illustrate that the target object determination device 200 includes: a pointing information acquisition module 210, a to-be-selected object acquisition module 220, and a target object determination module 230, wherein:

[0187] The pointing information acquisition module 210 is used to acquire the pointing information of the occupant in the vehicle.

[0188] Furthermore, the pointing information acquisition module 210 includes: a first pointing information acquisition submodule, a posture information acquisition submodule and a second pointing information acquisition submodule, wherein:

[0189] The first pointing information acquisition submodule is used to acquire the pointing information of the occupant detected by a detection device located in the vehicle in a detection device coordinate system.

[0190] Further, the detection device includes a head-mounted augmented reality device, and the first pointing information acquisition submodule includes: a first posture acquisition unit and a first pointing information acquisition unit, wherein:

[0191] The first posture acquisition unit is used to determine the posture of the occupant's head in the coordinate system of the head-mounted augmented reality device when the head-mounted augmented reality device is worn on the occupant's head.

[0192] The first pointing information acquisition unit is used to determine the pointing information of the occupant in the coordinate system of the head-mounted augmented reality device based on the posture of the occupant's head in the coordinate system of the head-mounted augmented reality device.

[0193] Furthermore, the detection device includes a mobile terminal, and the first pointing information acquisition submodule includes: a camera parameter acquisition unit and a second pointing information acquisition unit, wherein:

[0194] The camera parameter acquisition unit is used to acquire the center point of the camera and the optical center axis of the camera when the mobile terminal is in a gripping state and the camera is used to capture images.

[0195] The second pointing information acquisition unit is used to determine the pointing information of the occupant in the mobile terminal coordinate system based on the center point of the camera and the optical center axis of the camera.

[0196] The posture information acquisition submodule is used to obtain the posture information of the vehicle in the world coordinate system and to obtain the posture information of the detection device in the vehicle coordinate system.

[0197] The second pointing information acquisition submodule is used to convert the pointing information of the occupant in the detection device coordinate system into the pointing information of the occupant in the world coordinate system based on the posture information of the vehicle in the world coordinate system and the posture information of the detection device in the vehicle coordinate system.

[0198] The to-be-selected object acquisition module 220 is used to determine a pointing area corresponding to the pointing information, and acquire one or more to-be-selected objects corresponding to the pointing area.

[0199] Furthermore, the pointing information includes a pointing starting point and a pointing direction. The to-be-selected object acquisition module 220 includes: a pointing area construction submodule and a to-be-selected object acquisition submodule, wherein:

[0200] The pointing area construction submodule is used to construct a conical area based on the pointing starting point and the pointing direction as the pointing area.

[0201] The to-be-selected object acquisition submodule is used to acquire one or more objects that intersect with the conical area and are not blocked, as the one or more to-be-selected objects corresponding to the pointing area.

[0202] The target object determination module 230 is configured to determine a target object from the one or more objects to be selected in response to a selection instruction from the occupant for the one or more objects to be selected.

[0203] Furthermore, if the number of the objects to be selected is multiple, the target object determination module 230 includes: a first to-be-selected object identification submodule, a second to-be-selected object determination submodule and a second to-be-selected object identification submodule, wherein:

[0204] The first to-be-selected object identification submodule is used to display the pointing area to the occupant and identify a first to-be-selected object preselected from the plurality of to-be-selected objects in the pointing area through identification information.

[0205] The second to-be-selected object determination submodule is used to determine a second to-be-selected object corresponding to the object switching instruction from the multiple to-be-selected objects when receiving the object switching instruction triggered by the occupant.

[0206] The second to-be-selected object identification submodule is used to switch the identification information from identifying the first to-be-selected object to identifying the second to-be-selected object in the pointing area, and determine the second to-be-selected object as the target object.

[0207] Furthermore, the target object determination module 230 includes: a target feature information acquisition submodule, a query information output submodule and a target object determination submodule, wherein:

[0208] The target feature information acquisition submodule is used to acquire the target feature information corresponding to each of the one or more objects to be selected, wherein the target feature information corresponding to each of the one or more objects to be selected is different.

[0209] The inquiry information output submodule is used to output inquiry information to the occupant based on the target feature information corresponding to each of the one or more objects to be selected.

[0210] The target object determination submodule is used to determine the target object from the one or more objects to be selected in response to the occupant's response information to the inquiry information.

[0211] Furthermore, the target object determination device 200 further includes: an operation control module, wherein:

[0212] An operation control module is used to control the vehicle to perform an operation associated with the target object.

[0213] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the above-described devices and modules can refer to the corresponding processes in the aforementioned method embodiments, and will not be repeated here.

[0214] In several embodiments provided in the present application, the coupling between modules may be electrical, mechanical or other forms of coupling.

[0215] In addition, each functional module in each embodiment of the present application can be integrated into a processing module, or each module can exist physically separately, or two or more modules can be integrated into one module. The above integrated modules can be implemented in the form of hardware or software functional modules.

[0216] See also Fig.19 , which shows a structural block diagram of an electronic device 100 provided in an embodiment of the present application. The electronic device 100 in the present application may include one or more of the following components: a processor 110, a memory 120, and one or more applications, wherein the one or more applications may be stored in the memory 120 and configured to be executed by one or more processors 110, and the one or more programs are configured to execute the method described in the aforementioned method embodiment.

[0217] Among them, the processor 110 may include one or more processing cores. The processor 110 uses various interfaces and lines to connect various parts of the entire electronic device 100, and executes various functions and processes data of the electronic device 100 by running or executing instructions, programs, code sets or instruction sets stored in the memory 120, and calling data stored in the memory 120. Optionally, the processor 110 can be implemented in at least one hardware form of digital signal processing (Digital Signal Processing, DSP), field programmable gate array (Field-Programmable Gate Array, FPGA), and programmable logic array (Programmable Logic Array, PLA). The processor 110 can integrate one or a combination of a central processing unit (Central Processing Unit, CPU), a graphics processing unit (Graphics Processing Unit, GPU) and a modem. Among them, the CPU mainly processes the operating system, user interface and application programs; the GPU is responsible for rendering and drawing the content to be displayed; and the modem is used to process wireless communications. It can be understood that the above-mentioned modem may not be integrated into the processor 110, but may be implemented separately through a communication chip.

[0218] The memory 120 may include a random access memory (RAM) or a read-only memory (ROM). The memory 120 may be used to store instructions, programs, codes, code sets or instruction sets. The memory 120 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for implementing at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the following various method embodiments, etc. The data storage area may also store data (such as a phone book, audio and video data, chat record data) created by the electronic device 100 during use.

[0219] See also Fig. 20 , which shows a structural block diagram of a computer-readable storage medium provided in an embodiment of the present application. The computer-readable medium 300 stores program codes, which can be called by a processor to execute the method described in the above method embodiment.

[0220] The computer readable storage medium 300 may be an electronic memory such as a flash memory, an EEPROM (electrically erasable programmable read-only memory), an EPROM, a hard disk, or a ROM. Optionally, the computer readable storage medium 300 includes a non-transitory computer-readable storage medium. The computer readable storage medium 300 has storage space for program code 310 that performs any method steps in the above method. These program codes can be read from or written to one or more computer program products. The program code 310 can be compressed, for example, in an appropriate form.

[0221] In summary, the target object determination method, device, electronic device and storage medium provided in the embodiments of the present application obtain the pointing information of the occupant in the vehicle, determine the pointing area corresponding to the pointing information, and obtain one or more objects to be selected corresponding to the pointing area, and determine the target object from the one or more objects to be selected in response to the occupant's selection instructions for the one or more objects to be selected, so that the target object can be intuitively indicated through the pointing information and selection instructions of the occupant in the vehicle, which can facilitate the development of subsequent interactive tasks and improve the naturalness and efficiency of the interaction.

[0222] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A method for determining a target object, characterized in that: The method comprises: Acquire pointing information of an occupant in the vehicle, wherein the pointing information is a ray including a pointing starting point and a pointing direction; constructing a conical area based on the pointing starting point and the pointing direction as the pointing area, wherein the size of the conical area supports changing based on a user operation or a change in the environment; Acquire one or more objects that intersect the conical area and are not blocked as one or more objects to be selected corresponding to the pointing area; In response to a selection instruction of the occupant for the one or more objects to be selected, determining a target object from the one or more objects to be selected; Wherein, in response to the occupant's selection instruction for the one or more objects to be selected, determining the target object from the one or more objects to be selected includes: Acquire target feature information corresponding to each of the one or more objects to be selected, wherein the target feature information corresponding to each of the one or more objects to be selected is different; outputting inquiry information to the occupant based on target feature information corresponding to each of the one or more objects to be selected; In response to the occupant's response information to the inquiry information, the target object is determined from the one or more objects to be selected.

2. The method according to claim 1, characterized in that If there are multiple objects to be selected, determining a target object from the one or more objects to be selected in response to the occupant's selection instruction for the one or more objects to be selected includes: Displaying the pointing area to the occupant, and identifying a first to-be-selected object preselected from the plurality of to-be-selected objects in the pointing area by identification information; In case of receiving the object switching instruction triggered by the occupant, determining a second object to be selected corresponding to the object switching instruction from the plurality of objects to be selected; In the pointing area, the identification information is switched from identifying the first object to be selected to identifying the second object to be selected, and the second object to be selected is determined as the target object.

3. The method according to claim 1 or 2, characterized in that: The step of obtaining the pointing information of the occupant in the vehicle includes: Acquiring pointing information of the occupant in a detection device coordinate system detected by a detection device located in the vehicle; Acquiring the position and posture information of the vehicle in the world coordinate system, and acquiring the position and posture information of the detection device in the vehicle coordinate system; Based on the position information of the vehicle in the world coordinate system and the position information of the detection device in the vehicle coordinate system, the pointing information of the occupant in the detection device coordinate system is converted into the pointing information of the occupant in the world coordinate system.

4. The method according to claim 3, characterized in that The detection device includes a head-mounted augmented reality device, and the step of obtaining the pointing information of the occupant in the coordinate system of the detection device detected by the detection device located in the vehicle includes: When the head-mounted augmented reality device is worn on the head of the occupant, determining the position and posture information of the head of the occupant in the coordinate system of the head-mounted augmented reality device; Based on the posture information of the occupant's head in the coordinate system of the head-mounted augmented reality device, the pointing information of the occupant in the coordinate system of the head-mounted augmented reality device is determined.

5. The method according to claim 4, characterized in that The detection device includes a mobile terminal, and the step of acquiring the pointing information of the occupant detected by the detection device located in the vehicle in the coordinate system of the detection device includes: When the mobile terminal is in a gripping state and an image is captured through a camera, obtaining a center point of the camera and an optical center axis of the camera; Based on the center point of the camera and the optical center axis of the camera, the pointing information of the occupant in the mobile terminal coordinate system is determined.

6. The method according to claim 1 or 2, characterized in that: After determining the target object from the one or more objects to be selected in response to the occupant's selection instruction for the one or more objects to be selected, the method further includes: The vehicle is controlled to perform an operation associated with the target object.

7. A device for determining a target object, characterized in that: The device comprises: A pointing information acquisition module, used to acquire pointing information of an occupant in the vehicle, wherein the pointing information is a ray including a pointing starting point and a pointing direction; A pointing area construction submodule, configured to construct a conical area based on the pointing starting point and the pointing direction as the pointing area, wherein the size of the conical area can be changed based on a user operation or a change in the environment; A to-be-selected object acquisition submodule, used to acquire one or more objects intersecting with the conical area and not blocked, as one or more to-be-selected objects corresponding to the pointing area; a target object determination module, configured to determine a target object from the one or more objects to be selected in response to a selection instruction of the occupant for the one or more objects to be selected; Wherein, the target object determination module includes: A target feature information acquisition submodule, used to acquire target feature information corresponding to each of the one or more objects to be selected, wherein the target feature information corresponding to each of the one or more objects to be selected is different; An inquiry information output submodule, configured to output inquiry information to the occupant based on target feature information corresponding to each of the one or more objects to be selected; The target object determination submodule is used to determine the target object from the one or more objects to be selected in response to the occupant's response information to the inquiry information.

8. An electronic device, characterized in that: The method comprises a memory and a processor, wherein the memory is coupled to the processor, and the memory stores instructions. When the instructions are executed by the processor, the processor executes the method according to any one of claims 1 to 6.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores program codes, which can be called by a processor to execute the method according to any one of claims 1 to 6.

Images