A door control method, device, electronic equipment, and storage medium.

Abstract

This disclosure relates to a vehicle door control method and apparatus, electronic device, and storage medium. The method includes: acquiring a depth image of a passenger in a door area collected by a depth image sensor inside the vehicle; determining the relative positional relationship between the passenger and the door frame based on the depth image and the horizontal distance between the depth image sensor and the door frame; and suppressing a door closing command received by the vehicle when the relative positional relationship indicates that the passenger is within the door frame area. Embodiments of this disclosure can prevent passenger pinching when the door is closed.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to a door control method and apparatus, electronic device and storage medium. Background Technology

[0002] Currently, the opening and closing of public transportation vehicles (such as buses) is often subjectively controlled by the driver. Due to the limitations of the driver's perception and the subjectivity of human judgment, there are situations where the doors close before passengers have finished getting off, threatening the personal safety of passengers who are getting off. Summary of the Invention

[0003] This disclosure proposes a door control technology solution.

[0004] According to one aspect of this disclosure, a door control method is provided, comprising:

[0005] Acquire depth images of passengers in the door area captured by depth image sensors inside the vehicle;

[0006] Based on the depth image and the horizontal distance between the depth image sensor and the door frame, the relative positional relationship between the passenger and the door frame is determined.

[0007] When the relative positional relationship indicates that the passenger is within the door frame of the vehicle door, the vehicle door closing command received by the vehicle is suppressed.

[0008] In one possible implementation, determining the relative positional relationship between the passenger and the door frame based on the depth image and the horizontal distance between the depth image sensor and the door includes:

[0009] Based on the depth image, determine the location of at least one human key point of the passenger and a first distance between the at least one human key point and the depth sensor;

[0010] Based on the position of the at least one human key point, the first distance between the at least one human key point and the depth sensor, and the horizontal distance between the depth image sensor and the door frame range of the vehicle door, the relative positional relationship of the at least one human key point, the door frame range of the vehicle door, and the depth image sensor is determined.

[0011] The relative positional relationship between the passenger and the door frame is determined based on the relative positional relationship of the at least one human body key point, the door frame of the vehicle door, and the depth image sensor.

[0012] In one possible implementation, determining the relative positional relationship between the at least one human keypoint, the door frame of the vehicle door, and the depth image sensor based on the position of the at least one human keypoint, a first distance between the at least one human keypoint and the depth sensor, and the horizontal distance between the depth image sensor and the door frame of the vehicle door includes:

[0013] Determine the intersection point of the line connecting the depth image sensor and the at least one key point with the vertical plane where the door frame of the vehicle door is located, and calculate the second distance between the depth sensor and the intersection point;

[0014] Based on the comparison results of the first distance and the second distance, the relative positional relationship of the at least one human body key point, the door frame of the vehicle door, and the depth image sensor is determined.

[0015] In one possible implementation, determining the relative positional relationship between the passenger and the door frame based on the relative positional relationship of the at least one human body key point, the door frame, and the depth image sensor includes at least one of the following:

[0016] If the relative positional relationships between multiple preset human body key points, the door frame of the vehicle door, and the depth image sensor all meet the first condition, determine the range of the door frame from which the passenger leaves the vehicle and moves away from the door.

[0017] If the relative positional relationship between any of the multiple preset human body key points, the door frame of the vehicle door, and the depth image sensor all meet the second condition, it is determined that the passenger is located within the door frame of the vehicle door.

[0018] If the relative positions of multiple preset human body key points, the door frame of the vehicle door, and the depth image sensor all satisfy the third condition, the range of the door frame where the passenger is inside the vehicle and away from the door is determined.

[0019] The first condition includes: the first distance is greater than the second distance and the difference between the first distance and the second distance is greater than a first threshold.

[0020] The second condition includes: the absolute difference between the first distance and the second distance is less than a first threshold;

[0021] The third condition includes: the first distance is less than the second distance and the absolute difference between the first distance and the second distance is greater than the first threshold.

[0022] In one possible implementation, the plurality of preset human body key points include: head key point, shoulder key point, elbow key point, hand key point, leg key point, and foot key point.

[0023] In one possible implementation, the method further includes:

[0024] Passengers in the door area are detected based on image information of the door area acquired by the depth image sensor;

[0025] The acquisition of depth images of passengers in the door area collected by the depth image sensor inside the vehicle includes:

[0026] When a passenger is detected in the door area and a door-closing command is received, a depth image of the passenger in the door area is acquired by a depth image sensor inside the vehicle.

[0027] In one possible implementation, the method further includes:

[0028] The system detects whether there are passengers intending to get off the vehicle in the door area based on the image information of the door area collected by the depth image sensor.

[0029] The acquisition of depth images of passengers in the door area collected by the depth image sensor inside the vehicle includes:

[0030] When a passenger intending to get off is detected in the door area and a door-closing command is received, a depth image of the passenger in the door area is acquired by a depth image sensor inside the vehicle.

[0031] In one possible implementation, the method further includes:

[0032] When the relative positional relationship indicates that the passenger has left the vehicle and is away from the door, and a door-closing command is received, the door is controlled to close.

[0033] In one possible implementation, the depth sensor is positioned on the vehicle roof facing the door.

[0034] According to one aspect of this disclosure, a door control device is provided, comprising:

[0035] The acquisition module is used to acquire depth images of passengers in the door area collected by the depth image sensor inside the vehicle.

[0036] The relationship determination module is used to determine the relative positional relationship between the passenger and the door frame based on the depth image and the horizontal distance between the depth image sensor and the door frame.

[0037] A suppression module is used to suppress a door-closing command received by the vehicle when the relative positional relationship indicates that the passenger is within the door frame of the door.

[0038] In one possible implementation, the relationship determination module is configured to: determine the position of at least one human key point of the passenger and a first distance between the at least one human key point and the depth sensor based on the depth image; determine the relative positional relationship between the at least one human key point, the door frame of the vehicle door, and the depth image sensor based on the position of the at least one human key point, the first distance between the at least one human key point and the depth sensor, and the horizontal distance between the depth image sensor and the door frame of the vehicle door; and determine the relative positional relationship between the passenger and the door frame of the vehicle door based on the relative positional relationship between the at least one human key point, the door frame of the vehicle door, and the depth image sensor.

[0039] In one possible implementation, the relationship determination module is used to determine the intersection point of the line connecting the depth image sensor and the at least one key point with the vertical plane where the door frame of the vehicle door is located, and to calculate a second distance between the depth sensor and the intersection point; based on the comparison result of the first distance and the second distance, to determine the relative positional relationship between the at least one human key point, the door frame of the vehicle door, and the depth image sensor.

[0040] In one possible implementation, the relationship determination module is configured to perform at least one of the following:

[0041] If the relative positional relationships between multiple preset human body key points, the door frame of the vehicle door, and the depth image sensor all meet the first condition, determine the range of the door frame from which the passenger leaves the vehicle and moves away from the door.

[0042] If the relative positional relationship between any of the multiple preset human body key points, the door frame of the vehicle door, and the depth image sensor all meet the second condition, it is determined that the passenger is located within the door frame of the vehicle door.

[0043] If the relative positions of multiple preset human body key points, the door frame of the vehicle door, and the depth image sensor all satisfy the third condition, the range of the door frame where the passenger is inside the vehicle and away from the door is determined.

[0044] The first condition includes: the first distance is greater than the second distance and the difference between the first distance and the second distance is greater than a first threshold.

[0045] The second condition includes: the absolute difference between the first distance and the second distance is less than a first threshold;

[0046] The third condition includes: the first distance is less than the second distance and the absolute difference between the first distance and the second distance is greater than the first threshold.

[0047] In one possible implementation, the plurality of preset human body key points include: head key point, shoulder key point, elbow key point, hand key point, leg key point, and foot key point.

[0048] In one possible implementation, the device further includes:

[0049] The passenger detection module is used to detect passengers in the door area based on the image information of the door area collected by the depth image sensor.

[0050] The acquisition module is used to acquire a depth image of the passenger in the door area collected by a depth image sensor inside the vehicle when a passenger is detected in the door area and a door closing command is received.

[0051] In one possible implementation, the device further includes:

[0052] The disembarkation intention detection module is used to detect whether there are passengers with the intention to disembark in the door area based on the image information of the door area collected by the depth image sensor.

[0053] The acquisition module is used to acquire a depth image of the passenger in the door area collected by the depth image sensor inside the vehicle when a passenger with the intention to get off the vehicle is detected in the door area and a door closing command is received.

[0054] In one possible implementation, the device further includes:

[0055] The control module is configured to control the door to close when the relative positional relationship indicates that the passenger has left the vehicle and is away from the door, and a door-closing command has been received.

[0056] In one possible implementation, the depth sensor is positioned on the vehicle roof facing the door.

[0057] According to one aspect of this disclosure, an electronic device is provided, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to perform the method described above.

[0058] According to one aspect of this disclosure, a computer-readable storage medium is provided that stores computer program instructions thereon, which, when executed by a processor, implement the above-described method.

[0059] In this embodiment, a depth image of a passenger in the door area is acquired by a depth image sensor inside the vehicle. Based on the depth image and the horizontal distance between the depth image sensor and the door frame, the relative positional relationship between the passenger and the door frame is determined. If the relative positional relationship indicates that the passenger is within the door frame, the vehicle's door-closing command is suppressed. Therefore, by using the detected depth image of the passenger in the door area, the passenger's position can be accurately determined with high accuracy. Furthermore, the relative positional relationship between the passenger and the door frame can be determined based on the depth image and the horizontal distance between the depth image sensor and the door frame, without needing to detect vehicle body information such as the door and door frame, thus improving the efficiency of the door control process. Moreover, suppressing the vehicle's door-closing command when the relative positional relationship indicates that the passenger is within the door frame achieves the effect of preventing passenger pinching.

[0060] It should be understood that the foregoing general description and the following detailed description are exemplary and explanatory only, and are not intended to limit this disclosure. Other features and aspects of this disclosure will become clear from the following detailed description of exemplary embodiments with reference to the accompanying drawings. Attached Figure Description

[0061] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this disclosure and, together with the specification, serve to illustrate the technical solutions of this disclosure.

[0062] Figure 1 A flowchart of a door control method according to an embodiment of the present disclosure is shown.

[0063] Figure 2 This diagram illustrates the positional relationship between an occupant and a door frame, as provided in an embodiment of this disclosure.

[0064] Figure 3 A block diagram of a door control device provided in an embodiment of this disclosure is shown.

[0065] Figure 4 This diagram illustrates a block diagram of an electronic device 800 provided in an embodiment of the present disclosure.

[0066] Figure 5 A block diagram of an electronic device 1900 provided in an embodiment of this disclosure is shown. Detailed Implementation

[0067] Various exemplary embodiments, features, and aspects of this disclosure will now be described in detail with reference to the accompanying drawings. The same reference numerals in the drawings denote elements that have the same or similar functions. Although various aspects of the embodiments are shown in the drawings, they are not necessarily drawn to scale unless specifically indicated otherwise.

[0068] The term “exemplary” as used herein means “serving as an example, embodiment, or illustration.” Any embodiment illustrated herein as “exemplary” is not necessarily to be construed as superior to or better than other embodiments.

[0069] In this document, the term "and / or" is merely a description of the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent three cases: A alone, A and B simultaneously, and B alone. Furthermore, the term "at least one" in this document means any combination of at least two of any one or more elements. For example, including at least one of A, B, and C can mean including any one or more elements selected from the set consisting of A, B, and C.

[0070] Furthermore, to better illustrate this disclosure, numerous specific details are set forth in the following detailed description. Those skilled in the art will understand that this disclosure can be practiced without certain specific details. In some instances, methods, means, components, and circuits well known to those skilled in the art have not been described in detail in order to highlight the main points of this disclosure.

[0071] Currently, when public transportation vehicles stop to let passengers on and off, the driver often determines whether there are passengers within the door area by checking the rearview mirror on the side of the door or by the image captured by the camera installed at the door. Then, the driver manually controls the opening and closing of the door. This method of deciding whether to open or close the door based on the driver's observation is highly subjective and leads to the risk of passengers being trapped in the door.

[0072] In addition, some methods of installing detectors in the door frame have blind spots, which may not be able to accurately detect small parts such as hands. Moreover, due to their low sensitivity, they are prone to causing accidents.

[0073] In this embodiment, a depth image of a passenger in the door area is acquired by a depth image sensor inside the vehicle. Based on the depth image and the horizontal distance between the depth image sensor and the door frame, the relative positional relationship between the passenger and the door frame is determined. If the relative positional relationship indicates that the passenger is within the door frame, the vehicle's door-closing command is suppressed. Therefore, by using the detected depth image of the passenger in the door area, the passenger's position can be accurately determined with high accuracy. Furthermore, the relative positional relationship between the passenger and the door can be determined based on the depth image and the horizontal distance between the depth image sensor and the door, without needing to detect vehicle body information such as the door and door frame, thus improving the efficiency of the door control process. Moreover, suppressing the vehicle's door-closing command when the relative positional relationship indicates that the passenger is within the door frame achieves the effect of preventing passenger pinching.

[0074] In one possible implementation, the method can be executed by an intelligent driving control device installed on a vehicle. In another possible implementation, the method can be executed by a terminal device, a server, or other processing equipment. The terminal device can be an in-vehicle device, user equipment (UE), mobile device, user terminal, terminal, cellular phone, cordless phone, personal digital assistant (PDA), handheld device, computing device, or wearable device, etc. The in-vehicle device can be a vehicle-mounted system or domain controller within the vehicle cabin, or it can be a host device in an ADAS (Advanced Driving Assistance System), OMS (Occupant Monitoring System), or DMS (Driver Monitoring System) used to execute the door control method, etc. In some possible implementations, the door control method can be implemented by a processor calling computer-readable instructions stored in memory.

[0075] For ease of description, in one or more implementations of this specification, the executing entity of the door control method can be an on-board device in the vehicle. The following description uses an on-board device as the executing entity to illustrate the implementation of this method. It should be understood that using an on-board device as the executing entity is merely an illustrative example and should not be construed as a limitation of the method.

[0076] Figure 1 A flowchart of a door control method according to an embodiment of the present disclosure is shown, such as... Figure 1As shown, the door control method includes:

[0077] In step S11, depth images of passengers in the door area are acquired by a depth image sensor inside the vehicle.

[0078] The "door area" here refers to the area near the door of a vehicle, such as the area inside the vehicle cabin near the door, or the area outside the cabin near the door. This door can be a passenger boarding door or a passenger alighting door. The vehicle can be a car, aircraft, ship, etc. The vehicle here can be a public transportation vehicle, such as a bus, subway, train, etc. Furthermore, the vehicle here can also be at least one of the following types of vehicles: private car, shared car, ride-hailing vehicle, taxi, truck, etc. This disclosure does not limit the specific type of vehicle.

[0079] The depth image sensor here can acquire depth information of an object. In one example, the depth image sensor can be a color image sensor or an infrared image sensor equipped with a time-of-flight (TOF) sensor or a structured light sensor, or it can be a binocular image sensor, etc. Specifically, the TOF sensor uses the time of flight of a signal between two asynchronous transceivers (or a reflective surface) to measure the distance between nodes; the binocular sensor uses two cameras to perceive the depth information of an object; and the structured light sensor can be an coded structured light sensor or a speckle structured light sensor.

[0080] A depth image sensor can capture images containing depth information of the door area, thus obtaining high-precision depth information.

[0081] In step S12, the relative positional relationship between the passenger and the door frame is determined based on the depth image and the horizontal distance between the depth image sensor and the door frame.

[0082] The depth image contains the passenger's depth information, which can characterize the passenger's position in a coordinate system. This coordinate system can be a coordinate system established with the sensor as the origin, or it can be a world coordinate system. This disclosure does not limit this.

[0083] Optionally, the depth information of each pixel in the depth image represents the distance between the object represented by that pixel and the depth image sensor. Therefore, based on the depth image, the distance between the passenger and the depth image sensor can be determined.

[0084] The relative position of a passenger to the door frame indicates whether the passenger is within the door frame's area. The door frame's area can be the spatial region enclosed by the door frame. Since the four sides of the door frame are often on the same plane, the planar area enclosed by the four sides of the door frame can be considered the door frame's area. If a passenger is within the door frame's area, there is a possibility of being trapped when the door closes.

[0085] The horizontal distance between the depth image sensor and the door frame can be specifically the distance between the sensor and the plane containing the door frame, i.e., the length of a perpendicular line drawn from the sensor to that plane. Since the door frame is usually perpendicular to the ground, and the position and size of the door frame within the vehicle are fixed, the position of the entire door frame can be determined once the initial horizontal distance between the sensor and the door frame is determined, assuming the door frame is perpendicular to the ground.

[0086] The horizontal distance between the depth image sensor and the door frame can be pre-measured, for example, by directly measuring the distance between the sensor and the plane containing the door frame. Alternatively, during the installation of the depth image sensor, installation position parameters can be obtained, which include the horizontal distance between the depth sensor's positioning point and the door frame. Further methods for determining this horizontal distance are not detailed here.

[0087] Obviously, the passenger's position information can be determined based on the depth image, and the position of the door frame can be determined based on the horizontal distance between the depth image sensor and the door frame. Therefore, the relative positional relationship between the passenger in the door area and the door frame can be determined. For details, please refer to the possible implementation methods provided in this disclosure, which will not be elaborated here.

[0088] In step S13, if the relative positional relationship indicates that the passenger is within the door frame of the vehicle door, the vehicle door closing command received by the vehicle is suppressed.

[0089] The door-closing command can be issued by the driver operating the control element, or, in driverless or autonomous driving scenarios, the vehicle can automatically issue the door-closing command after arriving at the destination and detecting that a passenger has disembarked. In this case, if it is determined from the relative positional relationship obtained in step S12 that the passenger is within the door frame area, the door-closing command automatically triggered by the driver or vehicle controller can be suppressed.

[0090] The door control method provided in this disclosure can be executed in parking scenarios, such as after the driver issues a door closing command. When the relative positional relationship indicates that the passenger is within the door frame, it often means that closing the door will trap the passenger. Therefore, the door closing command received by the vehicle is suppressed to avoid trapping the passenger, thus ensuring high safety.

[0091] In this embodiment, a depth image of a passenger in the door area is acquired by a depth image sensor inside the vehicle. Based on the depth image and the horizontal distance between the depth image sensor and the door, the relative positional relationship between the passenger and the door is determined. If the relative positional relationship indicates that the passenger is within the door frame, the vehicle's door-closing command is suppressed. Therefore, by using the detected depth image of the passenger in the door area, the passenger's position can be accurately determined with high accuracy. Furthermore, the relative positional relationship between the passenger and the door can be determined based on the depth image and the horizontal distance between the depth image sensor and the door, without needing to detect vehicle body information such as the door and door frame, thus improving the efficiency of the door control process. Moreover, suppressing the vehicle's door-closing command when the relative positional relationship indicates that the passenger is within the door frame achieves the effect of preventing passenger pinching.

[0092] In one possible implementation, determining the relative positional relationship between the passenger and the vehicle door based on the depth image and the horizontal distance between the depth image sensor and the vehicle door includes: determining the position of at least one human key point of the passenger and a first distance between the at least one human key point and the depth sensor based on the depth image; determining the relative positional relationship between the at least one human key point, the vehicle door frame, and the depth image sensor based on the position of the at least one human key point, the first distance between the at least one human key point and the depth sensor, and the horizontal distance between the depth image sensor and the door frame; and determining the relative positional relationship between the passenger and the door frame based on the relative positional relationship between the at least one human key point, the vehicle door frame, and the depth image sensor.

[0093] The depth image of passengers in the door area acquired by the depth image sensor contains depth information of objects in the door area. This depth information can represent the distance between the objects and the depth sensor. Therefore, by performing human key point detection based on the depth image, the position of at least one human key point of the passenger and the first distance between at least one human key point and the depth sensor can be determined.

[0094] In one possible implementation, the relative positional relationship of the at least one human key point, the door frame of the vehicle door, and the depth image sensor is determined based on the position of the at least one human key point, a first distance between the at least one human key point and the depth sensor, and the horizontal distance between the depth image sensor and the door frame of the vehicle door. This includes: determining the intersection point of the line connecting the depth image sensor and the at least one key point with the vertical plane containing the door frame of the vehicle door, and calculating a second distance between the depth sensor and the intersection point; and determining the relative positional relationship of the at least one human key point, the door frame of the vehicle door, and the depth image sensor based on the comparison result of the first distance and the second distance.

[0095] It should be noted that the relative positional relationship between the human body key point, the door frame of the vehicle door, and the depth image sensor can include the positional relationship of any one of the three relative to the other two. In an exemplary case, in the horizontal direction (parallel to the ground), the human body key point is located on the side of the door frame away from the depth sensor. In other words, the door frame of the vehicle door is located between the human body key point and the depth sensor in the horizontal direction.

[0096] Please see Figure 2 This is a schematic diagram illustrating the positional relationship between an occupant and a door frame, provided by an embodiment of this disclosure. The line AB connecting the depth image sensor A and a key point B of the passenger is the second distance between the depth image sensor and the vertical plane containing the door frame, and the intersection point C of AB and the vertical plane of the door frame. The length of the line AC is the second distance between the depth image sensor and the intersection point.

[0097] The length b of AC can be determined based on the included angle and the distance a, specifically by formula (1).

[0098]

[0099] Wherein, the included angle α is the angle between AB and the horizontal direction. In one possible implementation, the specific method for determining this included angle is as follows: acquire image information of the car door area collected by a depth image sensor; perform human key point detection on the image information and determine the coordinates of the detected key points in the image information; based on the pre-calibrated correspondence between coordinates and included angles in the image, determine the included angle corresponding to the coordinates of the detected key points, as the included angle between the target line segment AB and the horizontal direction.

[0100] The depth image sensor can not only collect depth information, but also image information of the door area. For example, the depth image sensor can be a sensor that has both depth information acquisition function and RGB image acquisition function.

[0101] In one possible implementation, the depth sensor is positioned on the vehicle roof facing the door, such as... Figure 2 The location of point A is shown.

[0102] In one example, the sensor may be located on the top of the vehicle compartment near the door, facing the door side. The image information may be an RGB image or an infrared image. The image information may include a single frame image, a video stream, or multiple frames from a video stream; this disclosure does not limit the specific type of image information.

[0103] After acquiring the image information of the door area, human keypoint detection can be performed on the image information. If human keypoints are detected, it can be determined that there are passengers in the door area. There are various methods for detecting human keypoints. As an example of this implementation, multiple human keypoints to be detected can be pre-set. For example, the human skeleton can be set to contain 17 keypoints, indicating different parts of the human body, such as the head, hands, and elbows. By detecting these 17 keypoints, it can be determined whether there are passengers in the door area.

[0104] As another example of this implementation, image information can be input into a backbone network, which then extracts features from the image information to obtain feature maps. The locations of key points on the human body can then be detected based on these feature maps. The backbone network can employ network structures such as ResNet or MobileNet, and this is not a specific limitation.

[0105] The sensor can be fixed inside the vehicle cabin. Since the sensor's position is fixed, different positions in the image correspond to different angles. Therefore, after determining the human keypoints, the angle between the line connecting the human keypoints and the sensor and the horizontal direction can be determined based on the coordinates of these keypoints in the image information (e.g., a 2D image). Specifically, the correspondence between coordinates and angles in the image can be pre-calibrated, i.e., what angle corresponds to which coordinates in the image. Then, based on this pre-calibrated correspondence, the angle corresponding to the coordinates of the detected keypoints can be determined as the angle between the target line segment and the horizontal direction. The target line segment is the line connecting the depth image sensor and the human keypoints.

[0106] Furthermore, based on the coordinates of key points in the image information, the depth information of passenger key points in the depth information collected by the sensor can also be determined. There is a one-to-one correspondence between the image information and depth information collected by the sensor in terms of coordinates, and this correspondence can be pre-calibrated. Therefore, after determining the coordinates of key points in the image information collected by the sensor, the corresponding depth information, i.e., the depth information of the human body key points, can be determined based on these coordinates.

[0107] Therefore, by using the image information of the door area collected by the sensor and the pre-calibrated correspondence between coordinates and angles in the image, the angles corresponding to the coordinates of the key points detected in the image information can be determined as the angle between the target line segment and the horizontal direction, and the accuracy of the obtained angle is relatively high.

[0108] After determining the included angle α, since the horizontal distance a between the depth image sensor and the door frame of the car door is also known, the length b of AC can be determined based on formula (1), which is the second distance between the depth sensor and the intersection point.

[0109] Then, based on the comparison results of the first distance and the second distance, the relative positional relationship of at least one human body key point, the door frame of the car door, and the depth image sensor can be determined.

[0110] In this embodiment, the intersection point of the line connecting the depth image sensor and the at least one key point with the vertical plane of the door frame is determined, and a second distance between the depth sensor and the intersection point is calculated. Based on the comparison between the first distance and the second distance, the relative positional relationship between the at least one human key point, the door frame, and the depth image sensor is determined. Therefore, by accurately calculating the second distance between the depth sensor and the intersection point of the line connecting the depth image sensor and the at least one key point with the vertical plane of the door frame, the relative positional relationship between the at least one human key point, the door frame, and the depth image sensor can be accurately determined, thus accurately achieving door anti-pinch protection.

[0111] In one possible implementation, determining the relative position of the passenger to the door frame based on the relative positional relationships of the at least one human key point, the door frame, and the depth image sensor includes at least one of the following: determining the range of the door frame from which the passenger has left the vehicle and is away from the door, provided that the relative positional relationships of multiple preset human key points to the door frame and the depth image sensor all satisfy a first condition; determining the range of the door frame from which the passenger has left the vehicle, provided that the relative positional relationships of any one of the preset human key points to the door frame and the depth image sensor all satisfy a second condition; and determining the range of the door frame from which the passenger is inside the vehicle and away from the door, provided that the relative positional relationships of multiple preset human key points to the door frame and the depth image sensor all satisfy a third condition.

[0112] The first condition includes: the first distance is greater than the second distance and the difference between the first distance and the second distance is greater than a first threshold; the second condition includes: the absolute difference between the first distance and the second distance is less than the first threshold; the third condition includes: the first distance is less than the second distance and the absolute difference between the first distance and the second distance is greater than the first threshold.

[0113] The relative positional relationship between the passenger and the door frame of the vehicle door may include at least one of the following: the passenger leaves the vehicle and is away from the door frame; the passenger is within the door frame; the passenger is inside the vehicle and is away from the door frame.

[0114] Specifically, if the relative positions of multiple preset human body key points, the door frame of the vehicle door, and the depth image sensor all satisfy a first condition, the range of the door frame from which the passenger leaves the vehicle and moves away from the door is determined. The first condition includes: the first distance is greater than the second distance, and the difference between the first distance and the second distance is greater than a first threshold. The first threshold can be a pre-set empirical threshold.

[0115] If the relative positional relationship between any of the multiple preset human body key points, the door frame of the vehicle door, and the depth image sensor all satisfy the second condition, it is determined that the passenger is located within the door frame area of ​​the vehicle door. The second condition includes: the absolute difference between the first distance and the second distance is less than a first threshold.

[0116] If the relative positions of multiple preset human body key points, the door frame of the vehicle door, and the depth image sensor all satisfy a third condition, the range of the passenger within the vehicle and away from the door frame is determined. The third condition includes: the first distance is less than the second distance, and the absolute difference between the first distance and the second distance is greater than a first threshold.

[0117] In this embodiment of the disclosure, the relative positional relationship between the passenger and the door frame can be accurately determined based on the relative positional relationship of at least one human body key point, the door frame of the vehicle door, and the depth image sensor, so as to accurately achieve door anti-pinch based on the relative positional relationship.

[0118] In one possible implementation, the multiple preset key points of the human body include: top of head key point, shoulder key point, elbow key point, hand key point, leg key point, and foot key point.

[0119] In addition, key points of the human body can also be other key points of the human skeleton, indicating different parts of the human body, such as the head, hands, elbows, knees, etc.

[0120] In this embodiment of the disclosure, by monitoring the aforementioned key points of the human body, the passenger is detected, so as to accurately determine the relative positional relationship between the passenger and the door, and to accurately implement door anti-pinch based on the relative positional relationship.

[0121] In one possible implementation, the method further includes: detecting passengers in the door area based on image information of the door area acquired by the depth image sensor; acquiring the depth image of the passengers in the door area acquired by the depth image sensor inside the vehicle includes: when a passenger in the door area is detected and a door closing command is received, acquiring the depth image of the passengers in the door area acquired by the depth image sensor inside the vehicle.

[0122] The presence of passengers in the vehicle door area can be detected using image object detection technology. Specifically, human keypoint detection can be performed on the image information. If human keypoints are detected, it can be determined that passengers are present in the vehicle door area. There are various methods for detecting human keypoints. As an example of this implementation, multiple human keypoints to be detected can be pre-set. For example, a human skeleton can be set to contain 17 keypoints, indicating different parts of the human body, such as the head, hands, and elbows. By detecting these 17 keypoints, it can be determined whether passengers are present in the vehicle door area.

[0123] As another example of this implementation, image information can be input into a backbone network, which then extracts features from the image information to obtain a feature map. The location of key points on the human body can then be detected based on the feature map, thereby detecting passengers in the door area. The backbone network can employ network structures such as ResNet or MobileNet; no specific limitation is made here.

[0124] The execution scenario of this disclosure embodiment can be that it is executed only when a manual door closing command is received. In this case, after detecting that there is a passenger in the door area and receiving the door closing command, the depth image of the passenger in the door area collected by the depth image sensor inside the vehicle can be obtained, and then the door control method provided by this disclosure can be executed to prevent the driver from accidentally trapping the passenger due to inaccurate observation.

[0125] In one possible implementation, the method further includes: detecting whether there is a passenger intending to get off the vehicle in the door area based on the image information of the door area collected by the depth image sensor; the step of acquiring the depth image of the passenger in the door area collected by the depth image sensor inside the vehicle includes: when a passenger intending to get off the vehicle is detected in the door area and a door closing command is received, acquiring the depth image of the passenger in the door area collected by the depth image sensor inside the vehicle.

[0126] Based on the video stream of the door area collected by the depth image sensor, the action of a passenger approaching the door can be identified to determine whether there is a passenger intending to get off. If the action of a passenger approaching the door is identified, it can be determined that there is a passenger intending to get off.

[0127] Alternatively, in a bus scenario where passengers swipe their cards to get off, the video stream can be used to identify the passenger's card-swiping action to determine if the passenger intends to get off. Once the card-swiping action is detected, it can be determined that the passenger intends to get off.

[0128] The specific methods for recognizing a passenger's approach to the door and swiping their card can be implemented based on network structures such as ResNet and MobileNet, which will not be elaborated upon in this disclosure.

[0129] The execution scenario of this disclosure embodiment can be that it is executed only when a door closing command is received. In this case, if there is a passenger in the door area who intends to get off the vehicle and the door closing command is received, the depth image of the passenger in the door area collected by the depth image sensor inside the vehicle can be obtained. Then, the door control method provided in this disclosure is executed to prevent the driver from accidentally trapping the passenger due to inaccurate observation.

[0130] In one possible implementation, the method further includes: controlling the door to close when the relative positional relationship indicates that the passenger has left the vehicle and is away from the door, and a door-closing command is received.

[0131] If the relative positional relationship indicates that the passenger has left the vehicle and is away from the door, and a door-closing instruction has been received, it means there is no risk of the passenger being caught in the door, and the door-closing instruction can be executed.

[0132] The aforementioned door-closing command can be issued by the driver. In an autonomous driving scenario, the door-closing command can be issued by the vehicle controller. Specifically, the door-closing command issued by the driver can be triggered by the driver's actions. In one or more embodiments provided in this disclosure, upon receiving a door-closing command from the driver, it can be determined whether the passenger in the door area is within the door frame and whether to suppress the door-closing command. That is, the manually triggered door-closing command is not directly sent to the door control unit that controls door closing, but is first sent to the execution body of this method to further determine whether to send the door-closing command to the door control unit that controls door closing.

[0133] Therefore, although the driver manually or the vehicle controller automatically triggers the door closing command after arriving at the station or destination, whether or not the door closing command is sent to the door control unit that controls the door closing still needs to be determined by this method. This method detects whether the passenger is within the door frame. If the relative position indicates that the passenger is within the door frame, the door closing command received by the vehicle is suppressed to improve passenger safety. At the same time, a voice reminder can be issued to remind the passenger that the door is about to close and to pay attention to safety.

[0134] The following describes an application scenario of an embodiment of this disclosure. Please refer to... Figure 2 The application scenario is a bus disembarkation scenario. In this scenario, the TOF camera located at the bus disembarkation door collects image information and depth information of the disembarkation door area. The image information is used to detect key human points in the door area. When key human points are detected, the angle α between the line connecting the camera and the key point and the horizontal direction is obtained through the depth information. Using the angle α between the line connecting the key point and the camera and the horizontal direction, and the distance a between the camera and the door frame in the horizontal direction, the distance b is calculated using formula (1). The straight-line distance AB between the key point obtained by the TOF camera and the camera is compared with b. If AB is greater than b, it means that the key point has moved away from the door frame. According to the above logic, it is judged whether all key points of the same passenger have moved away from the door frame. If any key point has not moved away from the door frame, the driver's door closing command received by the door control unit is temporarily suppressed until all key points have moved away from the door frame before the door closing action is performed.

[0135] In this application scenario, after the bus driver observes that all passengers have disembarked, he closes the doors. The camera then begins to sense and determine whether all people have left the door area. If the camera detects that key human figures have not yet left the door area, the doors are not closed until all key figures have been identified as having left the door area, at which point the doors are closed.

[0136] It is understood that the various method embodiments mentioned above in this disclosure can be combined with each other to form combined embodiments without violating the principle and logic. Due to space limitations, this disclosure will not elaborate further. Those skilled in the art will understand that in the above methods of specific implementation, the specific execution order of each step should be determined by its function and possible internal logic.

[0137] In addition, this disclosure also provides a door control device, electronic device, computer-readable storage medium, and program, all of which can be used to implement any of the door control methods provided in this disclosure. The corresponding technical solutions and descriptions are described in the corresponding descriptions in the method section and will not be repeated here.

[0138] Figure 3A block diagram of a door control device according to an embodiment of the present disclosure is shown, such as Figure 3 As shown, the device includes:

[0139] The acquisition module 31 is used to acquire depth images of passengers in the door area collected by the depth image sensor inside the vehicle.

[0140] The relationship determination module 32 is used to determine the relative positional relationship between the passenger and the door frame based on the depth image and the horizontal distance between the depth image sensor and the door frame.

[0141] Suppression module 33 is used to suppress a door closing command received by the vehicle when the relative positional relationship indicates that the passenger is located within the door frame of the vehicle door.

[0142] In one possible implementation, the relationship determination module is configured to: determine the position of at least one human key point of the passenger and a first distance between the at least one human key point and the depth sensor based on the depth image; determine the relative positional relationship between the at least one human key point, the door frame of the vehicle door, and the depth image sensor based on the position of the at least one human key point, the first distance between the at least one human key point and the depth sensor, and the horizontal distance between the depth image sensor and the door frame of the vehicle door; and determine the relative positional relationship between the passenger and the door frame of the vehicle door based on the relative positional relationship between the at least one human key point, the door frame of the vehicle door, and the depth image sensor.

[0143] In one possible implementation, the relationship determination module is used to determine the intersection point of the line connecting the depth image sensor and the at least one key point with the vertical plane where the door frame of the vehicle door is located, and to calculate a second distance between the depth sensor and the intersection point; based on the comparison result of the first distance and the second distance, to determine the relative positional relationship between the at least one human key point, the door frame of the vehicle door, and the depth image sensor.

[0144] In one possible implementation, the relationship determination module is configured to perform at least one of the following:

[0145] If the relative positional relationships between multiple preset human body key points, the door frame of the vehicle door, and the depth image sensor all meet the first condition, determine the range of the door frame from which the passenger leaves the vehicle and moves away from the door.

[0146] If the relative positional relationship between any of the multiple preset human body key points, the door frame of the vehicle door, and the depth image sensor all meet the second condition, it is determined that the passenger is located within the door frame of the vehicle door.

[0147] If the relative positions of multiple preset human body key points, the door frame of the vehicle door, and the depth image sensor all satisfy the third condition, the range of the door frame where the passenger is inside the vehicle and away from the door is determined.

[0148] The first condition includes: the first distance is greater than the second distance and the difference between the first distance and the second distance is greater than a first threshold.

[0149] The second condition includes: the absolute difference between the first distance and the second distance is less than a first threshold;

[0150] The third condition includes: the first distance is less than the second distance and the absolute difference between the first distance and the second distance is greater than the first threshold.

[0151] In one possible implementation, the plurality of preset human body key points include: head key point, shoulder key point, elbow key point, hand key point, leg key point, and foot key point.

[0152] In one possible implementation, the device further includes:

[0153] The passenger detection module is used to detect passengers in the door area based on the image information of the door area collected by the depth image sensor.

[0154] The acquisition module is used to acquire a depth image of the passenger in the door area collected by a depth image sensor inside the vehicle when a passenger is detected in the door area and a door closing command is received.

[0155] In one possible implementation, the device further includes:

[0156] The disembarkation intention detection module is used to detect whether there are passengers with the intention to disembark in the door area based on the image information of the door area collected by the depth image sensor.

[0157] The acquisition module is used to acquire a depth image of the passenger in the door area collected by the depth image sensor inside the vehicle when a passenger with the intention to get off the vehicle is detected in the door area and a door closing command is received.

[0158] In one possible implementation, the device further includes:

[0159] The control module is configured to control the door to close when the relative positional relationship indicates that the passenger has left the vehicle and is away from the door, and a door-closing command has been received.

[0160] In one possible implementation, the depth sensor is positioned on the vehicle roof facing the door.

[0161] This method is specifically technically related to the internal structure of computer systems and can solve technical problems of how to improve hardware computing efficiency or execution performance (including reducing data storage, reducing data transmission, and increasing hardware processing speed), thereby achieving technical effects that improve the internal performance of computer systems in accordance with natural laws.

[0162] In some embodiments, the functions or modules of the apparatus provided in this disclosure can be used to perform the methods described in the above method embodiments. The specific implementation can be referred to the description of the above method embodiments, and for the sake of brevity, it will not be repeated here.

[0163] This disclosure also proposes a computer-readable storage medium storing computer program instructions that, when executed by a processor, implement the above-described method. The computer-readable storage medium can be volatile or non-volatile.

[0164] This disclosure also proposes an electronic device, including: a processor; and a memory for storing processor-executable instructions; wherein the processor is configured to invoke the instructions stored in the memory to execute the above-described method.

[0165] This disclosure also provides a computer program product, including computer-readable code, or a non-volatile computer-readable storage medium carrying computer-readable code, wherein when the computer-readable code is run in a processor of an electronic device, the processor in the electronic device performs the above-described method.

[0166] Electronic devices can be provided as terminals, servers, or other forms of devices.

[0167] Figure 4 This diagram illustrates a block diagram of an electronic device 800 according to an embodiment of the present disclosure. For example, the electronic device 800 may be a user equipment (UE), a mobile device, a user terminal, a terminal, a cellular phone, a cordless phone, a personal digital assistant (PDA), a handheld device, a computing device, an in-vehicle device, a wearable device, or other terminal devices.

[0168] Reference Figure 4 The electronic device 800 may include one or more of the following components: processing component 802, memory 804, power supply component 806, multimedia component 808, audio component 810, input / output (I / O) interface 812, sensor component 814, and communication component 816.

[0169] Processing component 802 typically controls the overall operation of electronic device 800, such as operations associated with display, telephone calls, data communication, camera operation, and recording operations. Processing component 802 may include one or more processors 820 to execute instructions to complete all or part of the steps of the methods described above. Furthermore, processing component 802 may include one or more modules to facilitate interaction between processing component 802 and other components. For example, processing component 802 may include a multimedia module to facilitate interaction between multimedia component 808 and processing component 802.

[0170] Memory 804 is configured to store various types of data to support the operation of electronic device 800. Examples of this data include instructions for any application or method operating on electronic device 800, contact data, phonebook data, messages, pictures, videos, etc. Memory 804 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0171] Power supply component 806 provides power to various components of electronic device 800. Power supply component 806 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to electronic device 800.

[0172] Multimedia component 808 includes a screen that provides an output interface between the electronic device 800 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 808 includes a front-facing camera and / or a rear-facing camera. When the electronic device 800 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0173] Audio component 810 is configured to output and / or input audio signals. For example, audio component 810 includes a microphone (MIC) configured to receive external audio signals when electronic device 800 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 804 or transmitted via communication component 816. In some embodiments, audio component 810 also includes a speaker for outputting audio signals.

[0174] I / O interface 812 provides an interface between processing component 802 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, power buttons, and lock buttons.

[0175] Sensor assembly 814 includes one or more sensors for providing state assessments of various aspects of electronic device 800. For example, sensor assembly 814 may detect the on / off state of electronic device 800, the relative positioning of components such as the display and keypad of electronic device 800, changes in position of electronic device 800 or a component of electronic device 800, the presence or absence of user contact with electronic device 800, orientation or acceleration / deceleration of electronic device 800, and temperature changes of electronic device 800. Sensor assembly 814 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 814 may also include an optical sensor, such as a complementary metal-oxide-semiconductor (CMOS) or charge-coupled device (CCD) image sensor, for use in imaging applications. In some embodiments, sensor assembly 814 may also include an accelerometer, gyroscope, magnetometer, pressure sensor, or temperature sensor.

[0176] Communication component 816 is configured to facilitate wired or wireless communication between electronic device 800 and other devices. Electronic device 800 can access wireless networks based on communication standards, such as Wi-Fi, 2G, 3G, 4G, LTE, 5G, or combinations thereof. In one exemplary embodiment, communication component 816 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 816 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID), Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0177] In an exemplary embodiment, the electronic device 800 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the methods described above.

[0178] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 804 including computer program instructions that can be executed by a processor 820 of an electronic device 800 to perform the above-described method.

[0179] This disclosure relates to the field of augmented reality (AR). It involves acquiring image information of target objects in a real-world environment and then using various visual algorithms to detect or identify the relevant features, states, and attributes of these objects, thereby achieving an AR effect that combines virtual and real elements to suit specific applications. For example, target objects may include human features such as faces, limbs, gestures, and movements; objects such as signs and markers; or venues such as sand tables, display areas, or displayed items. Visual algorithms may include visual localization, SLAM, 3D reconstruction, image registration, background segmentation, keypoint extraction and tracking of objects, and pose or depth detection. Specific applications can include interactive scenarios related to real-world scenes or objects, such as guided tours, navigation, explanations, reconstruction, and virtual effect overlay displays, as well as human-related special effects processing, such as makeup enhancement, limb enhancement, special effects displays, and virtual model displays. Convolutional neural networks (CNNs) can be used to detect or identify the relevant features, states, and attributes of target objects. The aforementioned CNNs are network models trained using deep learning frameworks.

[0180] Figure 5 A block diagram of an electronic device 1900 according to an embodiment of the present disclosure is shown. For example, the electronic device 1900 may be provided as a server or a terminal device. (Refer to...) Figure 5 The electronic device 1900 includes a processing component 1922, which further includes one or more processors, and memory resources represented by memory 1932 for storing instructions, such as application programs, that can be executed by the processing component 1922. The application programs stored in memory 1932 may include one or more modules, each corresponding to a set of instructions. Furthermore, the processing component 1922 is configured to execute instructions to perform the methods described above.

[0181] Electronic device 1900 may also include a power supply component 1926 configured to perform power management of electronic device 1900, a wired or wireless network interface 1950 configured to connect electronic device 1900 to a network, and an input / output (I / O) interface 1958. Electronic device 1900 can operate on an operating system stored in memory 1932, such as Microsoft Server operating system (Windows Server). TM Apple's graphical user interface-based operating system (Mac OSX) TM ), a multi-user, multi-process computer operating system (Unix) TM Linux is a free and open-source Unix-like operating system. TM ), the open-source Unix-like operating system (FreeBSD) TM (or similar.)

[0182] In an exemplary embodiment, a non-volatile computer-readable storage medium is also provided, such as a memory 1932 including computer program instructions that can be executed by a processing component 1922 of an electronic device 1900 to perform the above-described method.

[0183] This disclosure can be a system, method, and / or computer program product. A computer program product may include a computer-readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of this disclosure.

[0184] Computer-readable storage media can be tangible devices capable of holding and storing instructions for use by an instruction execution device. Computer-readable storage media can be, for example, (but not limited to) electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination of the foregoing. More specific examples (a non-exhaustive list) of computer-readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination of the foregoing. The computer-readable storage media used herein are not to be construed as transient signals themselves, such as radio waves or other freely propagating electromagnetic waves, electromagnetic waves propagating through waveguides or other transmission media (e.g., light pulses through fiber optic cables), or electrical signals transmitted through wires.

[0185] The computer-readable program instructions described herein can be downloaded from computer-readable storage media to various computing / processing devices, or downloaded via a network, such as the Internet, local area network, wide area network, and / or wireless network, to an external computer or external storage device. The network may include copper transmission cables, fiber optic transmission, wireless transmission, routers, firewalls, switches, gateway computers, and / or edge servers. A network adapter card or network interface in each computing / processing device receives the computer-readable program instructions from the network and forwards them to the computer-readable storage media in the respective computing / processing device.

[0186] Computer program instructions used to perform the operations of this disclosure may be assembly instructions, instruction set architecture (ISA) instructions, machine instructions, machine-dependent instructions, microcode, firmware instructions, status setting data, or source code or object code written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Smalltalk, C++, etc., and conventional procedural programming languages ​​such as the "C" language or similar programming languages. The computer-readable program instructions may execute entirely on the user's computer, partially on the user's computer, as a standalone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer via any type of network—including a local area network (LAN) or a wide area network (WAN)—or may be connected to an external computer (e.g., via the Internet using an Internet service provider). In some embodiments, electronic circuitry, such as programmable logic circuitry, field-programmable gate arrays (FPGAs), or programmable logic arrays (PLAs), is personalized by utilizing the status information of the computer-readable program instructions to implement various aspects of this disclosure.

[0187] Various aspects of this disclosure are described herein with reference to flowchart illustrations and / or block diagrams of methods, apparatus (systems), and computer program products according to embodiments of this disclosure. It should be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer-readable program instructions.

[0188] These computer-readable program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing apparatus to produce a machine such that, when executed by the processor of the computer or other programmable data processing apparatus, they create means for implementing the functions / actions specified in one or more blocks of the flowchart and / or block diagram. These computer-readable program instructions can also be stored in a computer-readable storage medium that causes a computer, programmable data processing apparatus, and / or other device to operate in a particular manner; thus, the computer-readable medium storing the instructions comprises an article of manufacture that includes instructions for implementing aspects of the functions / actions specified in one or more blocks of the flowchart and / or block diagram.

[0189] Computer-readable program instructions may also be loaded onto a computer, other programmable data processing apparatus, or other device to cause a series of operational steps to be performed on the computer, other programmable data processing apparatus, or other device to produce a computer-implemented process, thereby causing the instructions executed on the computer, other programmable data processing apparatus, or other device to perform the functions / actions specified in one or more boxes of a flowchart and / or block diagram.

[0190] The flowcharts and block diagrams in the accompanying drawings illustrate the architecture, functionality, and operation of possible implementations of systems, methods, and computer program products according to various embodiments of the present disclosure. In this regard, each block in a flowchart or block diagram may represent a module, segment, or portion of an instruction containing one or more executable instructions for implementing a specified logical function. In some alternative implementations, the functions marked in the blocks may occur in a different order than those shown in the drawings. For example, two consecutive blocks may actually be executed substantially in parallel, and they may sometimes be executed in reverse order, depending on the functions involved. It should also be noted that each block in the block diagrams and / or flowcharts, and combinations of blocks in the block diagrams and / or flowcharts, may be implemented using a dedicated hardware-based system that performs the specified function or action, or using a combination of dedicated hardware and computer instructions.

[0191] The computer program product can be implemented specifically through hardware, software, or a combination thereof. In one alternative embodiment, the computer program product is specifically embodied in a computer storage medium; in another alternative embodiment, the computer program product is specifically embodied in a software product, such as a software development kit (SDK), etc.

[0192] The description of the various embodiments above tends to emphasize the differences between the various embodiments. The similarities or similarities between them can be referred to, and for the sake of brevity, they will not be repeated here.

[0193] Those skilled in the art will understand that, in the above-described method of the specific implementation, the order in which each step is written does not imply a strict execution order and does not constitute any limitation on the implementation process. The specific execution order of each step should be determined by its function and possible internal logic.

[0194] If the technical solution of this application involves personal information, the product using this technical solution has clearly informed the user of the personal information processing rules and obtained the user's voluntary consent before processing the personal information. If the technical solution of this application involves sensitive personal information, the product using this technical solution has obtained the user's separate consent before processing the sensitive personal information, and also meets the requirement of "express consent". For example, at personal information collection devices such as cameras, clear and prominent signs are set up to inform users that they have entered the scope of personal information collection and that personal information will be collected. If an individual voluntarily enters the collection scope, it is deemed that they have agreed to the collection of their personal information; or on the personal information processing device, with clear signs / information informing users of the personal information processing rules, authorization is obtained from the individual through pop-up information or by asking the individual to upload their personal information; wherein, the personal information processing rules may include information such as the personal information processor, the purpose of personal information processing, the processing method, and the types of personal information processed.

[0195] The various embodiments of this disclosure have been described above. These descriptions are exemplary and not exhaustive, nor are they limited to the disclosed embodiments. Many modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described embodiments. The terminology used herein is chosen to best explain the principles, practical application, or improvement of the technology in the market, or to enable others skilled in the art to understand the embodiments disclosed herein.

Claims

1. A vehicle door control method characterized by, The method comprises: acquiring a depth image of a passenger in a door area of a vehicle captured by a depth image sensor inside the vehicle; determining a relative position relationship between the passenger and a door frame of the door based on the depth image and a horizontal distance between the depth image sensor and the door frame; in a case where the relative position relationship indicates that the passenger is located within a range of the door frame of the door, suppressing a received door closing instruction of the vehicle; the determining of the relative position relationship between the passenger and the door frame based on the depth image and the horizontal distance between the depth image sensor and the door frame comprises at least one of: in a case where a relative position relationship between a plurality of preset human body key points and the door frame of the door and the depth image sensor all satisfy a first condition, determining that the passenger is away from the vehicle and away from the range of the door frame of the door; in a case where a relative position relationship between any one of the plurality of preset human body key points and the door frame of the door and the depth image sensor all satisfy a second condition, determining that the passenger is located within the range of the door frame of the door; in a case where a relative position relationship between the plurality of preset human body key points and the door frame of the door and the depth image sensor all satisfy a third condition, determining that the passenger is within the vehicle and away from the range of the door frame of the door; wherein the first condition comprises that a first distance is greater than a second distance and a difference between the first distance and the second distance is greater than a first threshold value; the first distance is a distance between a position of at least one human body key point of the passenger and the depth image sensor; the second distance is a distance between the depth image sensor and an intersection point; the intersection point is an intersection point of a line connecting the depth image sensor and the at least one key point and a vertical plane on which the door frame of the door is located; the second condition comprises that an absolute difference between the first distance and the second distance is less than the first threshold value; the third condition comprises that the first distance is less than the second distance and an absolute difference between the first distance and the second distance is greater than the first threshold value.

2. The method of claim 1, wherein, the determining of the relative position relationship between the passenger and the door frame based on the depth image and the horizontal distance between the depth image sensor and the door frame comprises: determining a position of at least one human body key point of the passenger and a first distance between the at least one human body key point and the depth image sensor based on the depth image; determining a relative position relationship between the at least one human body key point, the door frame of the door, and the depth image sensor based on the position of the at least one human body key point, the first distance between the at least one human body key point and the depth image sensor, and the horizontal distance between the depth image sensor and the door frame of the door; determining the relative position relationship between the passenger and the door frame of the door according to the relative position relationship between the at least one human body key point, the door frame of the door, and the depth image sensor.

3. The method of claim 2, wherein, The determining the relative position relationship of the at least one human body key point, the door frame of the vehicle door, and the depth image sensor based on the position of the at least one human body key point, the first distance between the at least one human body key point and the depth image sensor, and the horizontal distance between the depth image sensor and the door frame of the vehicle door comprises: determining the intersection of the line connecting the depth image sensor and the at least one key point and the vertical plane on which the door frame of the vehicle door is located, and calculating the second distance between the depth image sensor and the intersection; determining the relative position relationship of the at least one human body key point, the door frame of the vehicle door, and the depth image sensor according to the comparison result of the first distance and the second distance.

4. The method of claim 1, wherein, The plurality of preset human body key points include: a head key point, a shoulder key point, an elbow key point, a hand key point, a leg key point, and a foot key point.

5. The method according to any of claims 1 to 4, characterized in that The method further comprises: detecting a passenger in the vehicle door area according to image information of the vehicle door area collected by the depth image sensor; The method further comprises: detecting a passenger in the vehicle door area according to image information of the vehicle door area collected by the depth image sensor; 6. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: detecting a passenger in the vehicle door area according to image information of the vehicle door area collected by the depth image sensor; The method further comprises: detecting a passenger in the vehicle door area according to image information of the vehicle door area collected by the depth image sensor; 7. The method according to any one of claims 1 to 4, characterized in that, The method further comprises: controlling the vehicle door to close in the case that the relative position relationship indicates that the passenger is leaving the vehicle and moving away from the vehicle door, and a close vehicle door instruction is received.

8. The method according to any one of claims 1 to 4, characterized in that, The depth image sensor is arranged on the vehicle roof in a position facing the vehicle door.

9. A vehicle door control device characterized by comprising: The method further comprises: an acquisition module configured to acquire a depth image of a passenger in a vehicle door area collected by a depth image sensor in the vehicle interior; a relationship determination module configured to determine a relative position relationship between the passenger and a door frame of the vehicle door based on the depth image and a horizontal distance between the depth image sensor and the door frame of the vehicle door; an inhibition module configured to inhibit a close vehicle door instruction received by the vehicle in the case that the relative position relationship indicates that the passenger is located within the door frame range of the vehicle door; The relationship determination module is configured to perform at least one of: determining that the passenger is leaving the vehicle and moving away from the door frame range of the vehicle door in the case that the relative position relationship of each of a plurality of preset human body key points and the door frame of the vehicle door, and the depth image sensor satisfies a first condition; determining that the passenger is located within the door frame range of the vehicle door in the case that the relative position relationship of any one of a plurality of preset human body key points and the door frame of the vehicle door, and the depth image sensor satisfies a second condition; In a case where the relative position relationship among the plurality of preset human body key points, the door frame of the vehicle door, and the depth image sensor satisfies a third condition, it is determined that the passenger is in the vehicle and away from the door frame of the vehicle door. The first condition includes: a first distance is greater than a second distance, and a difference between the first distance and the second distance is greater than a first threshold; the first distance is a distance between at least one human body key point of the passenger and the depth image sensor; the second distance is a distance between the depth image sensor and an intersection point; and the intersection point is an intersection point of a line connecting the depth image sensor and the at least one human body key point and a vertical plane on which the door frame of the vehicle door is located. The second condition includes: an absolute difference between the first distance and the second distance is less than the first threshold. The third condition includes: the first distance is less than the second distance, and an absolute difference between the first distance and the second distance is greater than the first threshold.

10. An electronic device, comprising: The computer program instructions are executed by the processor to implement the method in any one of claims 1 to 8. The computer program instructions are executed by the processor to implement the method in any one of claims 1 to 8. ​ ​ 11. A computer-readable storage medium having stored thereon computer program instructions, wherein, ​

Images