Method, device, and electronic device for unlocking target vehicle door

Abstract

The present application relates to the field of vehicle technology and discloses a method, device, and electronic device for unlocking a target vehicle door. The target vehicle door is provided with an image acquisition device. The method includes: determining a target object in a video stream acquired by the image acquisition device, and display information of a corresponding image frame of the target object in the video stream; determining multiple target distance values ​​between the target object and the target vehicle door based on the display information of the corresponding image frame, and using a target distance value less than a first preset distance value as a following distance; extracting facial features and eye features of the target object from the video stream, determining the target object's facial expression based on the facial features, and determining the target object's gaze duration based on the eye features; determining the target object's unlocking intention based on the following distance, facial expression, and gaze duration, and determining whether to unlock the target vehicle door based on the unlocking intention. The unlocking intention determined by the present application is more accurate.

Description

Technical Field

[0001] The present application relates to the field of vehicle technology, and in particular to a method, device, and electronic device for unlocking a target vehicle door. Background Art

[0002] Currently, there are many ways to unlock car doors, including remote control keys, mobile phone APPs, mechanical keys, etc. Each method has its own unique advantages, but also has some disadvantages. For example, the remote control key cannot be used normally if the battery is low, the key is lost or damaged; the mobile phone APP needs to be connected to the Internet to work normally. If there is no network signal or the mobile phone battery is low, the car door cannot be opened. Mechanical keys and keyholes are prone to wear during long-term use, which will make it impossible to unlock the door. Summary of the Invention

[0003] In view of the above problems, the present application provides a method, device, and electronic device for unlocking a target door, which is used to accurately determine the target object's unlocking intention based on real-time information collected near the target door, and then determine whether to unlock the target door.

[0004] According to one aspect of the present application, a method for unlocking a target vehicle door is provided, wherein the target vehicle door is provided with an image acquisition device, and the unlocking method comprises: determining a target object in a video stream acquired by the image acquisition device, and display information of a corresponding image frame of the target object in the video stream; determining a plurality of target distance values ​​between the target object and the target vehicle door based on the display information of the corresponding image frame, and taking a target distance value that is less than a first preset distance value as a following distance; extracting facial features and eye features of the target object from the video stream, determining the facial expression of the target object based on the facial features, and determining the gaze duration of the target object based on the eye features; determining the unlocking intention of the target object based on the following distance, facial expression, and gaze duration, and determining whether to perform an unlocking operation on the target vehicle door based on the unlocking intention.

[0005] In an optional manner, the display information includes the pixel coordinates and size information of the target object in the corresponding image frame; determining a plurality of target distance values ​​between the target object and the target vehicle door based on the display information of the corresponding image frame further includes: determining a plurality of three-dimensional coordinates of the target object in a built-in coordinate system based on each corresponding pixel coordinate and the size information; wherein the built-in coordinate system is a coordinate system constructed based on the intrinsic parameters of the image acquisition device; and converting each three-dimensional coordinate in the built-in coordinate system into a corresponding three-dimensional coordinate in a world coordinate system based on the extrinsic parameters of the image acquisition device, so as to calculate a plurality of target distance values ​​between the target object and the target vehicle door.

[0006] In an optional manner, determining multiple three-dimensional coordinates of the target object in the built-in coordinate system based on each corresponding pixel coordinate and the size information further includes: normalizing each pixel coordinate to obtain a corresponding multiple initial three-dimensional coordinates; determining the target depth of the target object in the corresponding image frame based on each size information; and calculating multiple three-dimensional coordinates of the target object in the built-in coordinate system based on each initial three-dimensional coordinate and the corresponding target depth.

[0007] In an optional manner, converting each three-dimensional coordinate in the built-in coordinate system into a corresponding three-dimensional coordinate in the world coordinate system according to the external parameters of the image acquisition device further includes: determining an external parameter matrix according to the external parameters of the image acquisition device; and converting all dimensional coordinate values ​​in each three-dimensional coordinate in the built-in coordinate system based on the external parameter matrix to obtain the corresponding three-dimensional coordinates in the world coordinate system.

[0008] In an optional manner, extracting the facial features and eye features of the target object from the video stream further includes: determining the facial area of ​​the target object in the target image frame based on an edge function and preset object information; wherein the target image frame is any image frame in the video stream; determining the facial features of the target object based on texture information of the facial area, and extracting the eye features of the target object from the facial area based on preset point information.

[0009] In an optional manner, determining the target object's willingness to unlock based on the following distance, facial expression, and gaze duration further includes: digitizing the following distance, facial expression, and gaze duration, and multiplying each numerical value by a corresponding preset weight coefficient to obtain a corresponding product; summing all the products, and using the preset unlocking intention corresponding to the calculated sum as the target object's willingness to unlock.

[0010] In an optional manner, the eye features include an eye focus point; determining the gaze duration of the target object based on the eye features further includes: taking the continuous image frames with the eye focus point as a preset focus point as designated continuous image frames; calculating the recording duration of each designated continuous image frame based on the recording time of the starting image frame and the ending image frame in each designated continuous image frame; taking the recording duration of the target designated continuous image frame as the gaze duration of the target object; wherein, the target designated continuous image frame is the continuous image frame with the longest recording duration among all designated continuous image frames.

[0011] In an optional embodiment, the target vehicle door is further provided with a distance measuring device; before determining the target object in the video stream captured by the image acquisition device, the unlocking method further includes: if the distance value measured by the distance measuring device is less than a second preset distance value, controlling the image acquisition device to capture the video stream; wherein, the distance value represents the distance between any object and the target vehicle door.

[0012] According to another aspect of the present application, an unlocking device for a target vehicle door is provided, which is provided with an image acquisition device, and the unlocking device includes: a display information determination module, which is used to determine the target object in the video stream acquired by the image acquisition device, and the display information of the corresponding image frame of the target object in the video stream; a following distance determination module, which is used to determine multiple target distance values ​​between the target object and the target vehicle door according to the display information of the corresponding image frame, and use the target distance value less than a first preset distance value as the following distance; a gaze duration determination module, which is used to extract the facial features and eye features of the target object from the video stream, determine the facial expression of the target object according to the facial features, and determine the gaze duration of the target object according to the eye features; an unlocking module, which is used to determine the unlocking intention of the target object according to the following distance, facial expression, and gaze duration, and determine whether to unlock the target vehicle door according to the unlocking intention.

[0013] According to one aspect of the present application, an electronic device is provided, including: a controller; and a memory for storing one or more programs, which, when executed by the controller, performs the above-mentioned unlocking method.

[0014] According to one aspect of the present application, a computer-readable storage medium is further provided, on which computer-readable instructions are stored. When the computer-readable instructions are executed by a processor of a computer, the computer executes the above-mentioned unlocking method.

[0015] According to one aspect of the present application, a computer program product or computer program is also provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the above-described unlocking method.

[0016] The present application provides a new method of unlocking a car door. An image acquisition device is set at the target car door to collect the surrounding video stream and extract three different types of data related to the target object's willingness to unlock from the video stream, which respectively represent the distance between the target object and the target car door, the target object's facial expression, and the length of time the target object stares at the target car door, reflecting the target object's willingness to unlock from different dimensions, making the determined unlocking intention more accurate.

[0017] The above description is only an overview of the technical solutions of the embodiments of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The accompanying drawings are incorporated into and constitute a part of the specification, illustrate embodiments consistent with the present application, and together with the specification, are used to explain the principles of the present application. Obviously, the drawings described below are only some embodiments of the present application, and it is clear that a person of ordinary skill in the art can derive other drawings based on these drawings without inventive effort.

[0019] Figure 1 It is a flowchart of a method for unlocking a target vehicle door shown in an exemplary embodiment of the present application.

[0020] Figure 2 is based on Figure 1 The exemplary embodiment shown is a flow chart of another method for unlocking a target vehicle door.

[0021] Figure 3 is based on Figure 1 The exemplary embodiment shown is a flow chart of another method for unlocking a target vehicle door.

[0022] Figure 4 is based on Figure 1 The exemplary embodiment shown is a flow chart of another method for unlocking a target vehicle door.

[0023] Figure 5 It is a schematic diagram of a vehicle showing an exemplary embodiment of the present application.

[0024] Figure 6 It is a schematic diagram of the application scenario of the unlocking method of the target vehicle door of this application.

[0025] Figure 7 1 is a schematic structural diagram of an unlocking device for a target vehicle door according to an exemplary embodiment of the present application.

[0026] Figure 8It is a structural diagram of a computer system of an electronic device shown in an exemplary embodiment of the present application. DETAILED DESCRIPTION

[0027] Exemplary embodiments will be described in detail herein, with examples illustrated in the accompanying drawings. When the following description refers to the drawings, identical numerals in different figures represent identical or similar elements unless otherwise indicated. The embodiments described in the following exemplary embodiments are not intended to represent all embodiments consistent with the present application. Rather, they are merely examples of apparatus and methods consistent with certain aspects of the present application, as detailed in the appended claims.

[0028] The block diagrams shown in the accompanying drawings are merely functional entities and do not necessarily correspond to physically separate entities. That is, these functional entities may be implemented in software, in one or more hardware modules or integrated circuits, or in different networks and / or processor devices and / or microcontroller devices.

[0029] The flowcharts shown in the accompanying drawings are for illustrative purposes only and do not necessarily include all contents and operations / steps, nor must they be executed in the order described. For example, some operations / steps may be decomposed, while others may be combined or partially combined. Therefore, the actual execution order may vary depending on the actual situation.

[0030] In this application, "plurality" refers to two or more. "And / or" describes the relationship between related objects, indicating that three possible relationships exist. For example, "A and / or B" can mean: A exists alone, A and B exist simultaneously, and B exists alone. The character " / " generally indicates that the related objects are in an "or" relationship.

[0031] There are many ways to unlock car doors, including remote control keys, mobile phone apps, mechanical keys, etc. Each method has its own unique advantages, but also has some disadvantages. For example, the remote control key cannot be used normally if the battery is low, the key is lost or damaged; the mobile phone app requires an Internet connection to work properly. If there is no network signal or the phone battery is low, the car door cannot be opened. The mechanical key and keyhole are prone to wear during long-term use, which will make it impossible to unlock the door.

[0032] To this end, one aspect of this application provides a new method for unlocking a target door to accurately determine the unlocking intention of the target object and then determine whether to unlock the target door. Figure 1 , Figure 1This is a flow chart of a method for unlocking a target vehicle door according to an exemplary embodiment of the present application. The unlocking method includes at least steps S110 to S140; wherein the target vehicle door is provided with an image acquisition device, which is described in detail as follows:

[0033] S110: Determine a target object in a video stream captured by an image capture device, and display information of a corresponding image frame of the target object in the video stream.

[0034] Existing unlocking methods generally only improve the main driver's seat, without considering the unlocking needs of other doors. Generally, other doors can only be unlocked after the main driver's seat is unlocked. It cannot meet the unlocking needs of multiple scenarios, especially the scenario where there is no need to unlock the main driver's door, but only other non-driver's doors need to be unlocked.

[0035] To this end, this embodiment solves the problem of unlocking a single door. The target door in this embodiment is not limited to the driver's door; it can be any door of the vehicle. Furthermore, this embodiment can include multiple target doors. The real-time data captured by the image acquisition device for each target door is used to determine whether to unlock the corresponding target door.

[0036] The image acquisition device may be an existing vehicle-mounted camera or other device capable of acquiring image frames and video streams, and is mainly used to acquire real-time images near the corresponding target vehicle door.

[0037] A target object is a person near the target door who has a potential unlocking intention. There can be one or more of these persons, who wish to unlock the target door to gain access to the vehicle. This embodiment can set the target object type to ensure that the identified target object type meets the preset type. This avoids determining unlocking intentions for objects outside the target type, thereby reducing unnecessary data processing.

[0038] A video stream consists of multiple continuous image frames, each of which corresponds to a different acquisition moment, and the display information in each image frame may also be different. This embodiment determines a preset type of object (i.e., target object) from the video stream and extracts image frames displaying the target object. It then determines the display information of the target object in the extracted image frame to characterize the imaging information of the target object in the corresponding image frame, for example, the pixel ratio of the target object in the image frame, the pixel coordinates of the target object, the facial features of the target object, etc.

[0039] S120: Determine a plurality of target distance values ​​between the target object and the target door according to display information of the corresponding image frame, and use a target distance value smaller than a first preset distance value as a following distance.

[0040] For example, according to the principle of near-far imaging, the closer an object is to the image capture device, the larger its image area in the image frame, and vice versa. This embodiment can use the size of the target object's image area in the image frame, combined with the built-in parameters of the image capture device, to quickly determine the distance between the target object and the target vehicle door.

[0041] Because a video stream is composed of multiple image frames, and the target object is not displayed in all image frames, the distance between the target object and the target door represented in each image frame will obviously vary. In this embodiment, a first preset distance value is used to filter out target distances smaller than the first preset distance value as the following distance. The following distance represents the distance between the target object and the target door, i.e., the distance required for the unlocking operation to occur between the target object and the target door.

[0042] S130: extracting facial features and eye features of the target object from the video stream, determining the facial expression of the target object based on the facial features, and determining the gaze duration of the target object based on the eye features.

[0043] Facial features characterize the target's facial expression. Based on these features, the target's emotions can be determined. For example, if a subject's brows are furrowed, the corners of their mouth are turned down, and their face is flushed, the result is anxiety or worry. If the target is anxious to unlock the car and get in, these facial features will indicate anxiety to some extent.

[0044] Eye features are features that characterize the target subject's eyes, including but not limited to eye position, eye focus point, and pupil distance. Based on these eye-related features, this embodiment can determine the duration of the target subject's gaze focused on a certain range of locations, characterizing the duration of the target subject's gaze on the target door, and thereby determining the target subject's willingness to unlock the vehicle. The following exemplifies the process for determining gaze duration, which includes at least S131 to S133. Eye features include the eye focus point, as described in detail below:

[0045] S131: Continuous image frames in which the eyeball focusing point is a preset focusing point are used as designated continuous image frames.

[0046] The preset focus point represents the point where the line of sight of the target object is focused on the target door. In this embodiment, the preset focus point can be adjusted to determine continuous image frames corresponding to different focus points of the target object.

[0047] Continuous image frames are those in which the target subject's eyes focus on the same point and are recorded continuously. Using only individual image frames, it is impossible to accurately determine the corresponding fixation duration. For example, interruptions in the fixation process or changes in the fixation focus will affect the calculation of fixation duration. Only by using continuous image frames can we determine the duration of the target subject's gaze focused on a certain point.

[0048] The designated continuous image frames are image frames in which the focus point of the target object's eyes is the same and is continuously recorded at a preset focus point.

[0049] S132: Calculate the recording time of each designated continuous image frame according to the recording time of the starting image frame and the ending image frame in each designated continuous image frame.

[0050] Each frame in a given continuous sequence of image frames is arranged in chronological order, meaning that the capture times corresponding to adjacent frames are sequential. The starting frame is the frame recorded at the moment the subject's line of sight begins focusing on the preset focal point within the given continuous sequence of image frames. Similarly, the ending frame is the last frame recorded within the given continuous sequence of image frames, meaning that its corresponding capture time is the last capture time within the given continuous sequence of image frames.

[0051] Exemplarily, a difference operation is performed between the recording time of the last image frame and the recording time of the starting image frame, and the obtained difference is the recording duration of the specified continuous image frames.

[0052] S133: Using the recording duration of the target designated continuous image frames as the gaze duration of the target object; wherein the target designated continuous image frames are the continuous image frames with the longest recording duration among all designated continuous image frames.

[0053] Because the target's eye focus in each of the designated consecutive image frames is the same as the preset focus point, meaning their gaze is always focused on the target door, the recorded duration of that consecutive image frame also reflects the target's gaze duration. It's worth noting that the gaze duration is the longest recorded duration of the designated consecutive image frames, as this best reflects the target's intention to unlock the vehicle.

[0054] S140: Determine the unlocking intention of the target object based on the following distance, facial expression, and gaze duration, and determine whether to unlock the target door based on the unlocking intention.

[0055] Following distance represents the distance between the target subject and the target door, facial expression represents the target subject's real-time emotion, and gaze duration represents the length of time the target subject gazes at the target door. Based on these three different types of data, the target subject's unlocking intention can be assessed from three different dimensions, providing a more comprehensive and accurate assessment. This allows the target door to be unlocked when the unlocking conditions are met. In certain embodiments, to enhance the security of the unlocking operation, the determined unlocking intention is transmitted to the vehicle user's user terminal to obtain feedback from the vehicle user before the unlocking operation is performed, further enhancing the security of the unlocking operation.

[0056] For example, three different types of data can be characterized to obtain three different feature vectors, and the three different dimensional features can be fused to obtain a one-dimensional vector to determine the target object's willingness to unlock. In some embodiments, a distance measurement device can also be installed on the target door to measure the distance between the target object and the corresponding target door. The three different feature vectors can be combined to determine the target object's willingness to unlock. The corresponding execution code is as follows:

[0057]

[0058]

[0059] Among them, fused_feature can determine the target object's willingness to unlock.

[0060] In another exemplary embodiment, the following distance, facial expression, and gaze duration are digitized, and each numerical value is multiplied by the corresponding preset weight coefficient to obtain the corresponding product; all products are summed up, and the preset unlocking intention corresponding to the calculated sum is used as the unlocking intention of the target object.

[0061] Numerical processing involves converting each data type into a constant to facilitate numerical calculations. Each type of data has a corresponding preset weight coefficient, which can be calculated through calibration experiments to adjust the influence ratio between different types of data.

[0062] Substituting the calculated sum value into Table 1, the preset unlocking intention corresponding to the sum value can be determined as the unlocking intention of the target object.

[0063] Table 1: Correspondence between preset values ​​and preset unlocking intentions

[0064]

[0065]

[0066] This embodiment provides a new method for unlocking a car door. An image acquisition device is set at the target car door to collect the surrounding video stream and extract three different types of data related to the target object's willingness to unlock from the video stream. The data respectively represent the distance between the target object and the target car door, the target object's facial expression, and the length of time the target object gazes at the target car door. The data reflects the target object's willingness to unlock from different dimensions, making the determined unlocking intention more accurate.

[0067] In another exemplary embodiment of the present application, it is described in detail how to determine multiple target distance values ​​between the target object and the target door according to the display information of the corresponding image frame. For details, please refer to Figure 2 , Figure 2 is based on Figure 1 The exemplary embodiment shown is a flow chart of another method for unlocking a target door. Figure 1 The illustrated S120 further includes S210 to S220; wherein the display information includes pixel coordinates and size information of the target object in the corresponding image frame, which is described in detail as follows:

[0068] S210: Determine multiple three-dimensional coordinates of the target object in a built-in coordinate system according to the corresponding pixel coordinates and size information; wherein the built-in coordinate system is a coordinate system constructed according to the intrinsic parameters of the image acquisition device.

[0069] Intrinsic parameters include but are not limited to focal length, principal point coordinates, etc.

[0070] This embodiment processes the corresponding image frames separately to calculate the three-dimensional coordinates of the target object in the built-in coordinate system based on the pixel coordinates and size information obtained by imaging the target object in the corresponding image frames. Compared with ordinary two-dimensional coordinates, it can more accurately represent the specific position of the target object in the built-in coordinate system.

[0071] According to the corresponding pixel coordinates and size information, combined with the imaging principle and triangulation principle of the image acquisition device, the three-dimensional coordinates of the target object in the built-in coordinate system can be determined.

[0072] Exemplarily, each pixel coordinate is normalized to obtain a corresponding plurality of initial three-dimensional coordinates; the target depth of the target object in the corresponding image frame is determined based on each size information; and based on each initial three-dimensional coordinate and the corresponding target depth, a plurality of three-dimensional coordinates of the target object in the built-in coordinate system are calculated.

[0073] The target depth is calculated based on the image acquisition device's built-in parameters and size information. The size information represents the target object's size within the image frame. This size information is then compared to the preset size of the built-in parameters, and this ratio is used as the target depth. The target depth is multiplied by the initial 3D coordinates to obtain the target object's 3D coordinates in the built-in coordinate system.

[0074] S220: Converting the three-dimensional coordinates in the built-in coordinate system into corresponding three-dimensional coordinates in the world coordinate system according to the external parameters of the image acquisition device, so as to calculate a plurality of target distance values ​​between the target object and the target door.

[0075] As is known, the world coordinate system is a universal coordinate system that can easily represent coordinate points in different coordinate systems and can also manage coordinate data in a more standardized manner. For example, any location in the world can be determined using the known latitude and longitude determination method.

[0076] The coordinate transformation is illustrated as follows: an extrinsic parameter matrix is ​​determined based on the extrinsic parameters of the image acquisition device; all dimensional coordinate values ​​in each three-dimensional coordinate in the built-in coordinate system are transformed based on the extrinsic parameter matrix to obtain the corresponding three-dimensional coordinates in the world coordinate system.

[0077] Specifically, the three-dimensional coordinates in the built-in coordinate system are (X c , Y c , Z c ), the external parameter matrix is ​​[R|t], then its three-dimensional coordinates in the world coordinate system (X w , Y w , Z w ), can be calculated by the following formula:

[0078] In this embodiment, the three-dimensional coordinates of the target object in the world coordinate system and the three-dimensional coordinates of the target door in the world coordinate system can be used to quickly calculate the distance between the target object and the target door, thereby calculating multiple target distance values.

[0079] This embodiment unifies the coordinate system to unify the corresponding three-dimensional coordinates into the world coordinate system, thereby facilitating the calculation of the target distance value between the target object and the target door, thereby obtaining the target distance values ​​corresponding to multiple image frames.

[0080] In another exemplary embodiment of the present application, how to extract the facial features and eye features of the target object from the video stream is described in detail. Figure 3 , Figure 3 is based on Figure 1 The exemplary embodiment shown is a flow chart of another method for unlocking a target door. Figure 1 The illustrated S130 further includes S310 to S320, which are described in detail as follows:

[0081] S310: Determine a facial region of a target object in a target image frame according to an edge function and preset object information; wherein the target image frame is any image frame in a video stream.

[0082] The edge function identifies the target area in the image frame and combines it with the object edge features in the preset object information, the preset area range of the object's face, etc., to quickly determine the facial area of ​​the target object in each image frame.

[0083] S320: Determine facial features of the target object based on texture information of the facial region, and extract eye features of the target object from the facial region based on preset point information.

[0084] For example, the facial features and eye features of the target object can be quickly determined according to the following code:

[0085] #Traverse the faces detected in the image frame

[0086] for k,d in enumerate(dets):

[0087] #Get the bounding box of the face area

[0088] x1 = d.left()

[0089] y1 = d.top()

[0090] x2=d.right()

[0091] y2 = d.bottom()

[0092] #Use shape_predictor to obtain facial key points (facial features)

[0093] shape = predictor(img_gray,d)

[0094] #Eyes are usually located between keypoint indices 36-41 (left eye) and 42-47 (right eye)

[0095] for iin range(36,48):#Assume that the key points of the left and right eyes are included

[0096] point = shape.part(i)

[0097] cv2.circle(img,(point.x,point.y),2,(0,255,0),-1)

[0098] After determining the facial area, the eye movement trajectory and gaze point are analyzed based on the eye tracking algorithm.

[0099] The code is as follows:

[0100] #Extract eye features (such as pupil position)

[0101] pupil_position=extract_pupil_position(eye)

[0102] #Tracking eye movements

[0103] if pupil_position is not None:

[0104] eye_tracker.update(pupil_position)

[0105] #Estimate gaze point (additional geometric model and calibration data are required here)

[0106] #gaze_point=estimate_gaze_point(eye_tracker.state,calibration_data)

[0107] Thus, the eye features can be accurately determined in the determined facial area.

[0108] In another exemplary embodiment of the present application, the activation prerequisites of the image acquisition device are described in detail. Figure 4 , Figure 4 is based on Figure 1 The exemplary embodiment shown is a flow chart of another method for unlocking a target door. Figure 1 Before S110 shown, at least S410 is also included; wherein the target door is further provided with a distance measuring device, which is described in detail as follows:

[0109] S410: If the distance value measured by the distance measuring device is less than a second preset distance value, the image acquisition device is controlled to acquire a video stream; wherein the distance value represents the distance between any object and the target door.

[0110] The second preset distance value is the distance value that activates the image acquisition device. Specifically, if the distance measurement device detects that the distance to any object is less than the second preset distance value, the image acquisition device activates its acquisition function. It is worth noting that the first preset distance value is used to filter the following distance value, and the following distance is determined based on image frames in the video stream captured by the image acquisition device. Obviously, the second preset distance value must be greater than or equal to the first preset distance value.

[0111] See also Figure 5 , Figure 5 The figure is a schematic diagram of a vehicle according to an exemplary embodiment of the present application. Each door is equipped with an image capture device and a distance measurement device. When the distance measurement device senses an object approaching the vehicle, that is, when the distance between the object and the target door is less than a second preset distance value, the image capture device is activated to capture the corresponding video stream.

[0112] In this embodiment, there is no need to turn on the image acquisition device in real time. The distance measuring device is used to measure any object near the target door. If the distance between any object and the target door is less than a second preset distance value, the image acquisition device is controlled to capture a video stream, so as to avoid turning on the image acquisition device in real time to capture the video stream and perform unnecessary unlocking operation judgments, thereby reducing the amount of data processing.

[0113] In certain embodiments, a distance measurement device may also be used to measure the distance between a target object and a target vehicle door. This distance may be determined based on the image frame display information, and a target distance value less than a first predetermined distance value may be used as the following distance. Specifically, the distance value determined based on the display information is used as the first target distance value, and the distance value measured by the distance measurement device is used as the second target distance value. The final target distance value is calculated by combining these values ​​with their respective predetermined weights.

[0114] In another exemplary embodiment of the present application, the application scenarios of the above multiple unlocking methods are exemplarily described. Figure 6 , Figure 6 Schematic diagram of the application scenario of the unlocking method of the target vehicle door of the present application, wherein the vehicle 100 and the server 200 are connected by wireless communication, and the present application does not limit the connection method between them.

[0115] One or more doors of the vehicle 100 are provided with an image acquisition device for acquiring a video stream near the corresponding door. Figure 6As shown in the vehicle 100, it can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, wherein multiple servers can form a blockchain, and the server is a node on the blockchain. The server 200 can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network) and big data and artificial intelligence platforms, and this is not restricted here.

[0116] The server 200 serves as the execution subject of the unlocking method shown in any of the above exemplary embodiments to execute any of the above unlocking methods, as exemplified below:

[0117] The server 200 determines the target object in the video stream captured by the image acquisition device, and the display information of the corresponding image frame of the target object in the video stream; the server 200 determines multiple target distance values ​​between the target object and the target vehicle door based on the display information of the corresponding image frame, and uses the target distance value less than the first preset distance value as the following distance; the server 200 extracts the facial features and eye features of the target object from the video stream, determines the facial expression of the target object based on the facial features, and determines the gaze duration of the target object based on the eye features; the server 200 determines the unlocking intention of the target object based on the following distance, facial expression, and gaze duration, and determines whether to unlock the target vehicle door based on the unlocking intention.

[0118] Another aspect of the present application also provides a target door unlocking device, such as Figure 7 As shown, Figure 7 The following is a schematic diagram of the structure of an unlocking device for a target vehicle door, shown in an exemplary embodiment of the present application. The target vehicle door is provided with an image acquisition device, and the unlocking device 700 includes: a display information determination module 710 for determining a target object in a video stream captured by the image acquisition device, as well as display information of the target object's corresponding image frame in the video stream; a following distance determination module 730 for determining multiple target distance values ​​between the target object and the target vehicle door based on the display information of the corresponding image frame, and using a target distance value less than a first preset distance value as the following distance; a gaze duration determination module 750 for extracting facial features and eye features of the target object from the video stream, determining the target object's facial expression based on the facial features, and determining the target object's gaze duration based on the eye features; and an unlocking module 770 for determining the target object's unlocking intention based on the following distance, facial expression, and gaze duration, and determining whether to unlock the target vehicle door based on the unlocking intention.

[0119] In another exemplary embodiment, the display information includes the pixel coordinates and size information of the target object in the corresponding image frame; the following distance determination module 730 further includes: a three-dimensional coordinate determination unit, used to determine multiple three-dimensional coordinates of the target object in the built-in coordinate system based on each corresponding pixel coordinate and size information; wherein the built-in coordinate system is a coordinate system constructed according to the internal parameters of the image acquisition device; a three-dimensional coordinate conversion unit, used to convert each three-dimensional coordinate in the built-in coordinate system into the corresponding three-dimensional coordinate in the world coordinate system according to the external parameters of the image acquisition device, so as to calculate multiple target distance values ​​between the target object and the target door.

[0120] In another exemplary embodiment, the three-dimensional coordinate determination unit further includes: a normalization block, used to normalize each pixel coordinate to obtain a corresponding multiple initial three-dimensional coordinates; a target depth determination block, used to determine the target depth of the target object in the corresponding image frame based on various size information; a three-dimensional coordinate determination block, used to calculate multiple three-dimensional coordinates of the target object in the built-in coordinate system based on each initial three-dimensional coordinate and the corresponding target depth.

[0121] In another exemplary embodiment, the three-dimensional coordinate conversion unit further includes: an external parameter matrix determination module, which is used to determine the external parameter matrix according to the external parameters of the image acquisition device; and a three-dimensional coordinate conversion module, which is used to convert all dimensional coordinate values ​​in each three-dimensional coordinate in the built-in coordinate system based on the external parameter matrix to obtain the corresponding three-dimensional coordinates in the world coordinate system.

[0122] In another exemplary embodiment, the gaze duration determination module 750 further includes: a facial area determination unit, used to determine the facial area of ​​the target object in the target image frame based on the edge function and preset object information; wherein the target image frame is any image frame in the video stream; a feature determination unit, used to determine the facial features of the target object based on the texture information of the facial area, and extract the eye features of the target object from the facial area based on the preset point information.

[0123] In another exemplary embodiment, the unlocking module 770 further includes: a product calculation unit, which is used to digitize the following distance, facial expression, and gaze duration, and multiply each numerical value with the corresponding preset weight coefficient to obtain the corresponding product; an unlocking intention calculation unit, which is used to sum all the products, and use the preset unlocking intention corresponding to the calculated sum as the unlocking intention of the target object.

[0124] In another exemplary embodiment, the eye features include an eye focus point; the gaze duration determination module 750 further includes: a designated continuous image frame determination unit, used to take continuous image frames with the eye focus point as a preset focus point as designated continuous image frames; a recording duration calculation unit, used to calculate the recording duration of each designated continuous image frame based on the recording time of the starting image frame and the ending image frame in each designated continuous image frame; a gaze duration determination unit, used to take the recording duration of the target designated continuous image frame as the gaze duration of the target object; wherein, the target designated continuous image frame is the continuous image frame with the longest recording duration among all designated continuous image frames.

[0125] In another exemplary embodiment, the target vehicle door is further provided with a distance measuring device; the unlocking device 700 further includes: a control module, configured to control the image acquisition device to capture a video stream if the distance value measured by the distance measuring device is less than a second preset distance value; wherein the distance value represents the distance between any object and the target vehicle door.

[0126] The present application provides a new method of unlocking a car door. An image acquisition device is set at the target car door to collect the surrounding video stream and extract three different types of data related to the target object's willingness to unlock from the video stream, which respectively represent the distance between the target object and the target car door, the target object's facial expression, and the length of time the target object stares at the target car door, reflecting the target object's willingness to unlock from different dimensions, making the determined unlocking intention more accurate.

[0127] It should be noted that the unlocking device provided in the above embodiment and the unlocking method provided in the above embodiment belong to the same concept, and the specific manner in which each module and unit performs the operation has been described in detail in the method embodiment and will not be repeated here.

[0128] Another aspect of the present application provides an electronic device, comprising: a controller; and a memory for storing one or more programs, which, when executed by the controller, executes the above-mentioned unlocking method.

[0129] See also Figure 8 , Figure 8 1 is a schematic diagram of the structure of a computer system of an electronic device shown in an exemplary embodiment of the present application, which shows a schematic diagram of the structure of a computer system of an electronic device suitable for implementing an embodiment of the present application.

[0130] It should be noted that Figure 8 The computer system 800 of the electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0131] like Figure 8As shown, the computer system 800 includes a central processing unit (CPU) 801, which can perform various appropriate actions and processes according to the program stored in the read-only memory (ROM) 802 or the program loaded from the storage part 808 into the random access memory (RAM) 803, such as executing the method in the above embodiment. Various programs and data required for system operation are also stored in the RAM 803. The CPU 801, ROM 802 and RAM 803 are connected to each other via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.

[0132] The following components are connected to the I / O interface 805: an input section 806 including a keyboard, a mouse, and the like; an output section 807 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage section 808 including a hard disk and the like; and a communication section 809 including a network interface card such as a LAN (Local Area Network) card or a modem. The communication section 809 performs communication processing via a network such as the Internet. A drive 810 is also connected to the I / O interface 805 as needed. Removable media 811, such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, is installed in the drive 810 as needed, so that computer programs read therefrom can be installed into the storage section 808 as needed.

[0133] In particular, according to an embodiment of the present application, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present application includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes a computer program for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 809, and / or installed from a removable medium 811. When the computer program is executed by the central processing unit (CPU) 801, the various functions defined in the system of the present application are executed.

[0134] It should be noted that the computer-readable medium shown in the embodiments of the present application can be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium can be, for example, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media can include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM), a flash memory, an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present application, a computer-readable storage medium can be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, device or device. In the present application, a computer-readable signal medium can include a data signal propagated in baseband or as part of a carrier wave, which carries a computer-readable computer program. Such propagated data signals may take various forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination thereof. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in connection with an instruction execution system, apparatus, or device. A computer program embodied on a computer-readable medium may be transmitted using any suitable medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0135] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present application. Among them, each box in the flowchart or block diagram can represent a module, program segment, or part of the code, and the above-mentioned module, program segment, or part of the code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0136] The units involved in the embodiments described in this application may be implemented by software or hardware, and the units described may also be set in a processor. In some cases, the names of these units do not constitute limitations on the units themselves.

[0137] Another aspect of the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the aforementioned unlocking method. The computer-readable storage medium may be included in the electronic device described in the above embodiments, or may exist independently and not be incorporated into the electronic device.

[0138] Another aspect of the present application provides a computer program product or computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the unlocking method provided in each of the above embodiments.

[0139] According to one aspect of an embodiment of the present application, a computer system is further provided, including a central processing unit (CPU), which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) or a program loaded from a storage portion into a random access memory (RAM), such as executing the method in the above embodiment. Various programs and data required for system operation are also stored in the RAM. The CPU, ROM, and RAM are connected to each other via a bus. An input / output (I / O) interface is also connected to the bus.

[0140] The following components are connected to the I / O interface: an input section including a keyboard, mouse, etc.; an output section including a cathode ray tube (CRT), liquid crystal display (LCD), and speakers; a storage section including a hard disk; and a communication section including a network interface card such as a LAN (Local Area Network) card and a modem. The communication section performs communication processing via a network such as the Internet. A drive is also connected to the I / O interface as needed. Removable media such as magnetic disks, optical disks, magneto-optical disks, semiconductor memories, etc. are installed in the drive as needed so that computer programs read from them can be installed into the storage section as needed.

[0141] The above content is only a preferred exemplary embodiment of the present application and is not intended to limit the implementation scheme of the present application. Ordinary technicians in this field can easily make corresponding changes or modifications based on the main ideas and spirit of the present application. Therefore, the scope of protection of the present application shall be based on the scope of protection required by the claims.

Claims

1. A method for unlocking a target vehicle door, characterized in that: The target door is provided with an image acquisition device, and the unlocking method includes: Determining a target object in the video stream captured by the image capture device, and display information of a corresponding image frame of the target object in the video stream; determining a plurality of target distance values ​​between the target object and the target door according to display information of the corresponding image frame, and using a target distance value smaller than a first preset distance value as a following distance; Extracting facial features and eye features of the target object from the video stream, determining a facial expression representing the real-time emotion of the target object based on the facial features, and determining a gaze duration of the target object based on the eye features; The unlocking intention of the target object is determined according to the following distance, facial expression, and gaze duration, and whether to perform an unlocking operation on the target door is determined according to the unlocking intention.

2. The unlocking method according to claim 1, characterized in that: The display information includes pixel coordinates and size information of the target object in the corresponding image frame; The determining of a plurality of target distance values ​​between the target object and the target door based on the display information of the corresponding image frame further includes: Determining a plurality of three-dimensional coordinates of the target object in a built-in coordinate system according to the respective corresponding pixel coordinates and the size information; wherein the built-in coordinate system is a coordinate system constructed according to the intrinsic parameters of the image acquisition device; According to the external parameters of the image acquisition device, each three-dimensional coordinate in the built-in coordinate system is converted into a corresponding three-dimensional coordinate in the world coordinate system to calculate a plurality of target distance values ​​between the target object and the target door.

3. The unlocking method according to claim 2, characterized in that: The step of determining a plurality of three-dimensional coordinates of the target object in a built-in coordinate system according to the corresponding pixel coordinates and the size information further includes: Normalizing each pixel coordinate to obtain a corresponding plurality of initial three-dimensional coordinates; Determine the target depth of the target object in the corresponding image frame according to each size information; A plurality of three-dimensional coordinates of the target object in the built-in coordinate system are calculated based on the respective initial three-dimensional coordinates and the corresponding target depths.

4. The unlocking method according to claim 2, characterized in that: The step of converting the three-dimensional coordinates in the built-in coordinate system into corresponding three-dimensional coordinates in the world coordinate system according to the external parameters of the image acquisition device further includes: Determining an extrinsic parameter matrix according to the extrinsic parameters of the image acquisition device; All dimensional coordinate values ​​in each three-dimensional coordinate in the built-in coordinate system are converted based on the external parameter matrix to obtain corresponding three-dimensional coordinates in the world coordinate system.

5. The unlocking method according to claim 1, characterized in that: The extracting facial features and eye features of the target object from the video stream further includes: Determine the facial region of the target object in the target image frame according to the edge function and the preset object information; wherein the target image frame is any image frame in the video stream; The facial features of the target object are determined based on the texture information of the facial area, and the eye features of the target object are extracted from the facial area based on the preset point information.

6. The unlocking method according to any one of claims 1 to 5, characterized in that: Determining the target object's unlocking intention based on the following distance, facial expression, and gaze duration further includes: Numerically processing the following distance, facial expression, and gaze duration, and multiplying each numerical value by a corresponding preset weight coefficient to obtain a corresponding product; All products are summed up, and the preset unlocking intention corresponding to the calculated sum value is used as the unlocking intention of the target object.

7. The unlocking method according to any one of claims 1 to 5, characterized in that: The eye features include the eye focus point; Determining the gaze duration of the target object based on the eye features further includes: taking the continuous image frames in which the eyeball focus point is the preset focus point as the designated continuous image frames; Calculating the recording time of each specified continuous image frame according to the recording time of the starting image frame and the ending image frame in each specified continuous image frame; The recording duration of the target-specified continuous image frames is used as the gaze duration of the target object; wherein the target-specified continuous image frames are the continuous image frames with the longest recording duration among all the specified continuous image frames.

8. The unlocking method according to any one of claims 1 to 5, characterized in that: The target door is also provided with a distance measuring device; Before determining the target object in the video stream captured by the image capture device, the unlocking method further includes: If the distance value measured by the distance measuring device is less than a second preset distance value, the image acquisition device is controlled to acquire a video stream; wherein the distance value represents the distance between any object and the target door.

9. A target door unlocking device, characterized in that: The target door is provided with an image acquisition device, and the unlocking device includes: a display information determination module, configured to determine a target object in the video stream captured by the image capture device, and display information of a corresponding image frame of the target object in the video stream; a following distance determination module, configured to determine a plurality of target distance values ​​between the target object and the target door according to display information of the corresponding image frame, and to use a target distance value smaller than a first preset distance value as a following distance; a gaze duration determination module, configured to extract facial features and eye features of the target object from the video stream, determine a facial expression representing the real-time emotion of the target object based on the facial features, and determine the gaze duration of the target object based on the eye features; The unlocking module is used to determine the unlocking intention of the target object based on the following distance, facial expression, and gaze duration, and determine whether to unlock the target door based on the unlocking intention.

10. An electronic device, characterized in that: include: Controller; A memory for storing one or more programs, which, when executed by a controller, enables the controller to implement the unlocking method according to any one of claims 1 to 8.

Images