A control method, device, equipment and medium

Abstract

The present application relates to a control method, device, equipment and medium, which includes obtaining an image to be detected; using an elevator detection model to identify the elevator area of ​​the image to be detected to obtain an identification result; when the identification result includes elevator pixel area information, determining the actual position information of the elevator according to the elevator pixel area information; determining a warning area according to the actual position information of the elevator, and controlling the movement of the detection subject according to the warning area. The present application has the effect of accurately identifying the elevator and obtaining the accurate elevator position to achieve the effect of controlling the movement of the robot.

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a control method, device, equipment and medium. Background Art

[0002] With the development of artificial intelligence technology, robots have entered people's daily life from the industrial field. Robots can be seen in many large supermarkets. Robots can independently complete tasks such as welcoming guests, explaining business, and pushing information.

[0003] In order to ensure the safety of the robot's operation, the relevant technology uses laser radar detection or depth camera detection to collect the robot's activity area. However, the inventor found that when using the relevant technology to detect escalators, only the point cloud data of the handrail area on both sides can be obtained. If it is a descending escalator, the middle area of ​​the escalator collected is vacant, and it may not be possible to accurately determine whether there is an escalator in the activity area, which makes the robot continue to move without knowing that there is an escalator area ahead, which may lead to danger.

[0004] Therefore, how to solve the above technical problems has become an urgent problem to be solved by those skilled in the art. Summary of the invention

[0005] The purpose of this application is to provide a control method, device, equipment and medium for solving at least one of the above technical problems.

[0006] The above invention objectives of the present application are achieved through the following technical solutions:

[0007] In a first aspect, the present application provides a control method, which adopts the following technical solution:

[0008] A control method, the method comprising:

[0009] Acquire the image to be detected;

[0010] Using the elevator detection model to perform elevator area recognition on the image to be detected to obtain a recognition result;

[0011] When the recognition result includes elevator pixel area information, determining the actual position information of the elevator according to the elevator pixel area information;

[0012] The early warning area is determined according to the actual position information of the elevator, and the movement of the detection body is controlled according to the early warning area.

[0013] By adopting the above technical solution, the image to be detected is obtained, and the elevator detection model can be used to identify the elevator area based on the image to be detected. When the image to be detected includes the elevator image, accurate elevator pixel area information can be identified, and then the actual position information of the elevator is determined based on the elevator pixel area information. The elevator pixel area information is accurate data obtained through the elevator detection model, and thus the actual position information of the elevator determined based on the elevator pixel area information is also accurate. Furthermore, the warning area can be determined based on the actual position information of the elevator, and the movement of the detection subject can be controlled based on the warning area to reduce the occurrence of the detection subject falling due to incorrect identification of the escalator.

[0014] In a preferred example, the present application may be further configured as follows: determining the actual position information of the elevator according to the elevator pixel area information includes:

[0015] Determine the elevator pixel position information according to the elevator pixel area information;

[0016] The relative position information of the elevator and the detection subject is determined according to the elevator pixel position information using the camera projection relationship, and the relative position information is used as the actual position information of the elevator.

[0017] By adopting the above technical solution, the elevator pixel position information is determined by the elevator pixel area information, and the relative position information of the elevator and the detection subject is determined by using the camera projection relationship. The elevator pixel position information in the pixel coordinate system can be quickly converted into the position relative to the detection subject in the world coordinate system, which is simple to operate and convenient to calculate.

[0018] In a preferred example, the present application may be further configured as follows: controlling the movement of the detection subject according to the warning area includes:

[0019] If the detection body is within the warning area, the detection body is controlled to stop moving, or the detection body is controlled to move in a target movement direction, wherein the target movement direction is a direction away from the elevator area.

[0020] By adopting the above technical solution, when the detection subject is in the warning area, the detection subject is controlled to move away from the warning area or stop moving, thereby reducing the probability of danger to the detection subject.

[0021] In a preferred example, the present application may be further configured as follows: controlling the movement of the detection subject according to the warning area includes:

[0022] If the detection subject is in the warning area, determining whether the detection subject is in the elevator area;

[0023] If the detection body is within the elevator area, the detection body is controlled to stop moving, or the detection body is controlled to move in a target movement direction.

[0024] By adopting the above technical solution, when the detection body is in the warning area and the detection body is in the elevator area, it is determined that the detection body is already at the elevator entrance, and then an emergency stop method is taken or the detection body is controlled to move in the target movement direction to keep the detection body away from the elevator, which can effectively avoid the occurrence of danger.

[0025] In a preferred example, the present application may be further configured as follows: the determining whether the detection subject is in the elevator area includes:

[0026] Obtaining the vibration frequency of the detection body, and determining whether the vibration frequency is within a preset vibration frequency range;

[0027] If the vibration frequency is within the preset vibration frequency range, it is determined that the detection body is in the elevator area.

[0028] By adopting the above technical solution, the vibration frequency of the detection body is obtained, and it is determined whether the vibration frequency is within the preset vibration frequency range. If the vibration frequency is within the preset vibration frequency range, it can be accurately determined that the detection body is in the elevator area, thereby improving the accuracy of the judgment.

[0029] In a preferred example, the present application may be further configured as follows: the training process of the elevator detection model includes: obtaining an elevator image training sample set, wherein the elevator image training sample set includes a plurality of elevator sample images and respective corresponding elevator area annotation information;

[0030] The elevator image training sample set is used to train a convolutional neural network to obtain the elevator detection model.

[0031] By adopting the above technical solution, the convolutional neural network is trained using the elevator image training sample set to obtain an elevator detection model, and then when the elevator detection model is used to perform elevator recognition, the elevator recognition rate and recognition accuracy are improved.

[0032] In a preferred example, the present application may be further configured as follows: the step of obtaining an elevator image training sample set includes: obtaining a first training set, wherein the first training set includes a plurality of first elevator image training samples and respective corresponding elevator area annotation information;

[0033] Acquire a second training set, wherein the second training set includes a plurality of second elevator image training samples and respective corresponding elevator region annotation information, and the plurality of second elevator image training samples are obtained by cutting and recombining the plurality of first elevator image training samples;

[0034] The first training set and the second training set are used as the elevator image training sample sets.

[0035] By adopting the above technical solution, a second training set is obtained by cutting and recombining multiple first elevator image training samples. Both the first training set and the second training set are used as elevator image training sample sets to train the convolutional neural network. The training set is enriched by using cutting and recombining methods, thereby improving the model training effect, so that the trained elevator detection model has a higher recognition rate.

[0036] In a second aspect, the present application provides a control device, which adopts the following technical solution:

[0037] A control device, comprising:

[0038] An image acquisition module, used for acquiring an image to be detected;

[0039] An elevator area recognition module is used to perform elevator area recognition on the image to be detected using an elevator detection model to obtain a recognition result;

[0040] an actual position information determination module, configured to determine the actual position information of the elevator according to the elevator pixel area information when the recognition result includes the elevator pixel area information;

[0041] The control module determines a warning area according to the actual position information of the elevator, and controls the movement of the detection body according to the warning area.

[0042] By adopting the above technical solution, the image to be detected is obtained, and the elevator detection model can be used to identify the elevator area based on the image to be detected. When the image to be detected includes the elevator image, accurate elevator pixel area information can be identified, and then the actual position information of the elevator is determined based on the elevator pixel area information. The elevator pixel area information is accurate data obtained through the elevator detection model, and thus the actual position information of the elevator determined based on the elevator pixel area information is also accurate. Furthermore, the warning area can be determined based on the actual position information of the elevator, and the movement of the detection subject can be controlled based on the warning area to reduce the occurrence of the detection subject falling due to incorrect identification of the escalator.

[0043] In a preferred example, the present application may be further configured as follows: the actual position information determination module determines the actual position information of the elevator according to the elevator pixel area information, for:

[0044] Determine the elevator pixel position information according to the elevator pixel area information;

[0045] The relative position information of the elevator and the detection subject is determined according to the elevator pixel position information and the camera projection relationship.

[0046] In a preferred example, the present application may be further configured as follows: when the control module controls the movement of the detection subject according to the warning area, it is used to:

[0047] When the detection body is in the warning area, the detection body is controlled to stop moving, or the detection body is controlled to move in a target movement direction, wherein the target movement direction is a direction away from the elevator area.

[0048] In a preferred example, the present application may be further configured as follows: when the control module controls the movement of the detection subject according to the warning area, it is used to:

[0049] When the detection subject is in the warning area, determining whether the detection subject is in the elevator area;

[0050] When the detection body is within the elevator area, the detection body is controlled to stop moving, or the detection body is controlled to move in a target moving direction.

[0051] In a preferred example, the present application may be further configured as follows: when the control module determines whether the detection subject is in the elevator area, it is used to:

[0052] Obtaining the vibration frequency of the detection body, and determining whether the vibration frequency is within a preset vibration frequency range;

[0053] When the vibration frequency is within the preset vibration frequency range, it is determined that the detection body is within the elevator area.

[0054] In a preferred example, the present application can be further configured as follows: the control device further includes:

[0055] An elevator image training sample set acquisition module is used to acquire an elevator image training sample set, wherein the elevator image training sample set includes a plurality of elevator sample images and their corresponding elevator area annotation information;

[0056] A training module is used to train a convolutional neural network using the elevator image training sample set to obtain the elevator detection model.

[0057] In a preferred example, the present application may be further configured as follows: when the elevator image training sample set acquisition module executes the acquisition of the elevator image training sample set, it is used to:

[0058] Acquire a first training set, wherein the first training set includes a plurality of first elevator image training samples and respective corresponding elevator area annotation information;

[0059] Acquire a second training set, wherein the second training set includes a plurality of second elevator image training samples and respective corresponding elevator region annotation information, and the plurality of second elevator image training samples are obtained by cutting and recombining the plurality of first elevator image training samples;

[0060] The first training set and the second training set are used as the elevator image training sample sets.

[0061] In a third aspect, the present application provides an electronic device, which adopts the following technical solution:

[0062] An electronic device, comprising:

[0063] at least one processor;

[0064] Memory;

[0065] At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute any of the above methods.

[0066] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:

[0067] A computer-readable storage medium stores at least one instruction, at least one program, a code set or an instruction set, wherein the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by a processor to implement any of the methods described above.

[0068] In summary, the present application includes at least one of the following beneficial technical effects:

[0069] 1. Obtain the image to be detected, and use the elevator detection model to identify the elevator area based on the image to be detected. When the image to be detected includes the elevator image, the accurate elevator pixel area information can be identified, and then the actual position information of the elevator is determined based on the elevator pixel area information. The elevator pixel area information is accurate data obtained through the elevator detection model, and thus the actual position information of the elevator determined based on the elevator pixel area information is also accurate. Furthermore, the warning area can be determined based on the actual position information of the elevator, and the movement of the detection subject can be controlled based on the warning area to reduce the occurrence of the detection subject falling due to incorrect identification of the escalator. BRIEF DESCRIPTION OF THE DRAWINGS

[0070] Figure 1 It is a flow chart of a control method according to one embodiment of the present application.

[0071] Figure 2It is a schematic diagram of the relationship between four coordinate systems of one embodiment of the present application.

[0072] Figure 3 It is a schematic diagram of the relationship between the world coordinate system and the camera coordinate system according to one embodiment of the present application.

[0073] Figure 4 It is a schematic diagram of the relationship between the camera coordinate system and the image coordinate system of one embodiment of the present application.

[0074] Figure 5 It is a schematic diagram of the relationship between the image coordinate system and the pixel coordinate system according to one embodiment of the present application.

[0075] Figure 6 It is a structural schematic diagram of a control device according to one embodiment of the present application.

[0076] Figure 7 It is a structural schematic diagram of an electronic device according to one embodiment of the present application. DETAILED DESCRIPTION

[0077] The present application is further described in detail below in conjunction with the accompanying drawings.

[0078] This specific embodiment is merely an explanation of the present application and is not a limitation of the present application. After reading this specification, a person skilled in the art may make non-creative modifications to the present embodiment as needed, but such modifications are protected by the patent law as long as they are within the scope of the present application.

[0079] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0080] In addition, the term "and / or" in this article is only a description of the association relationship of associated objects, indicating that there can be three relationships. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. In addition, the character " / " in this article, unless otherwise specified, generally means that the associated objects before and after are in an "or" relationship.

[0081] The embodiments of the present application are further described in detail below in conjunction with the drawings in the specification.

[0082] In order to ensure the safety of robot operation, relevant technologies use laser radar detection or depth camera detection to collect the robot's activity area. The following working conditions may exist:

[0083] The robot uses laser radar detection or depth camera detection to collect the robot's activity area, and draws the point cloud data fed back by the hardware device into an electronic map. The areas where the robot is not expected to move and dangerous areas (such as escalator areas, stall placement areas, glass door areas, etc.) are framed in the electronic map using "virtual walls" to limit the robot's range of movement and prevent the robot from moving to the "virtual wall" area. However, when forming an electronic map, the robot needs to detect the environment. When detecting escalators, the robot's detector can only obtain point cloud data of the handrail areas on both sides, which is prone to detection errors, thus causing danger.

[0084] Furthermore, during the actual movement of the robot, the robot will comprehensively judge the robot's location by combining the electronic map and the point cloud data of the real-time detection of the environment during the movement. The inventor found that in the related technology, when performing real-time detection, the point cloud data fed back by the hardware device is combined to determine whether the front is a dangerous area, but the detection of the escalator is prone to errors, resulting in errors in the robot's judgment of its own position in the electronic map. In turn, the robot has deviations when calculating the distance between its current position and the "virtual wall" area in the electronic map, and is therefore unable to correctly judge whether it has entered the "virtual wall" area in the electronic map, resulting in the occurrence of danger. Of course, there may be other working conditions that are no longer limited in the embodiments of this application.

[0085] Of course, machine vision can also be used to collect the robot's activity area. However, using machine vision to identify elevators requires high computing power and is very inefficient. Even if some simpler classification and recognition schemes are used, only the elevator-related areas can be roughly marked, which is difficult to use as an effective reference for distance measurement.

[0086] Specifically, an embodiment of the present application provides a control method, which is executed by an electronic device, such as Figure 1 As shown, the method includes step S101, step S102, step S103 and step S104, wherein:

[0087] Step S101: Acquire an image to be detected.

[0088] For the embodiments of the present application, the method of obtaining the image to be detected may include: collecting the image to be detected through the camera of the detection subject so that the electronic device obtains the image to be detected; of course, it may also include: collecting the image to be detected through the camera of the shopping mall so that the electronic device obtains the image to be detected. The embodiments of the present application are no longer limited, as long as the electronic device can obtain the image to be detected.

[0089] The images to be detected include: images including an elevator and images not including an elevator, wherein the elevator may be an ascending elevator or a descending elevator.

[0090] Step S102: using the elevator detection model to perform elevator area recognition on the image to be detected to obtain a recognition result.

[0091] In an embodiment of the present application, the elevator detection model can identify the elevator area to obtain a recognition result. When the image to be detected includes an elevator, the recognition result may include: elevator area pixel information, specifically elevator area information in a pixel coordinate system, and the elevator area pixel information can specifically reflect the location information of the elevator area; when the image to be detected does not include an elevator, the recognition result may include: information corresponding to no elevator, specifically 0 or no output of any information.

[0092] The elevator detection model can be obtained by training based on multiple elevator sample images and their corresponding elevator area annotation information. Furthermore, the convolutional neural network is trained through multiple elevator sample images and their corresponding elevator area annotation information to obtain the elevator detection model, wherein the structure of the convolutional neural network can be set by the user according to actual needs, and the embodiment of the present application will not be limited. Furthermore, the elevator detection model can be called to perform elevator area recognition on the image to be detected.

[0093] Step S103: When the recognition result includes the elevator pixel area information, the actual position information of the elevator is determined according to the elevator pixel area information.

[0094] For the embodiment of the present application, when the image to be detected includes an elevator, the recognition result includes the elevator pixel area information, and then the actual position information of the elevator can be determined based on the elevator pixel area information. The embodiment of the present application can accurately identify the elevator for the up and down elevators. Specifically, the actual position information of the elevator is the position information of the elevator in the world coordinate system. In some cases, the position information of the elevator is the relative coordinates of the elevator relative to the detection subject. In other cases, the relative position information includes the distance between the two and the relative direction of the two. Among them, the relative position information can be determined according to the elevator pixel area information using the camera projection relationship, or the relative position information can be determined by the relationship between the pixel coordinate change and the actual moving distance. Specifically, by directly anchoring the elevator pixel coordinates and recording the cumulative moving distance of the robot, the relative position information of the elevator and the detection subject can be obtained by calculating the change in pixel coordinates and the actual moving distance.

[0095] Step S104: Determine the warning area according to the actual position information of the elevator, and control the movement of the detection body according to the warning area.

[0096] For the embodiment of the present application, the elevator position in the world coordinate system with the detection subject as the origin is determined according to the actual position information of the elevator, and the warning area is determined with the elevator position as the reference point. The present application does not limit the warning area, and the user can set it according to actual needs. For example, the reference point is used as the center to extend in four directions according to the extension length of each direction to obtain a rectangular warning area. For example, the reference point is used as the center of the circle to obtain a circular warning area based on a preset radius. For example, an irregular warning area is obtained with the reference point as the center. Among them, the detection subject can be a robot. If the detection subject enters the warning area, the detection subject is controlled to move, that is, the detection subject is controlled to stop or the detection subject is controlled to move in the target movement direction, and the target movement direction is the direction away from the elevator area.

[0097] It can be seen that in the embodiment of the present application, an image to be detected is obtained, and the elevator detection model can be used to perform elevator area recognition based on the image to be detected. When the image to be detected includes an elevator image, accurate elevator pixel area information can be identified, and then the actual position information of the elevator is determined based on the elevator pixel area information. Among them, the elevator pixel area information is accurate data obtained through the elevator detection model, and thus the actual position information of the elevator determined based on the elevator pixel area information is also accurate. Furthermore, the warning area can be determined based on the actual position information of the elevator, and the movement of the detection subject can be controlled based on the warning area to reduce the occurrence of the detection subject falling due to incorrect identification of the escalator.

[0098] Further, in order to simply and quickly obtain accurate relative position information, in the embodiment of the present application, step S103 determines the actual position information of the elevator according to the elevator pixel area information, including:

[0099] Determine the elevator pixel position information according to the elevator pixel area information;

[0100] For the embodiment of the present application, the elevator pixel area information can be the pixel coordinates of any point in the elevator feature annotation box and the height and width of the annotation box. The point pixel coordinates and the height and width are combined to form the annotation box. Therefore, the center point pixel coordinates of the annotation box can be calculated based on the obtained point pixel coordinates and the length and width, and the obtained center point pixel coordinates are determined as the elevator pixel position information.

[0101] Furthermore, in one case, when the obtained elevator pixel area information is the bottom elevator area information, the elevator pixel area information includes: the center point pixel coordinates determined based on the pixel coordinates of any point of the annotation box and the width and height of the annotation box as the elevator pixel position information; in another case, when the obtained elevator area information is the front elevator area information, the elevator pixel area information includes: the center point pixel coordinates determined based on the pixel coordinates of any point of the annotation box and the width and height of the annotation box as the elevator pixel position information; in another case, when the obtained elevator area information is the side elevator area information, the elevator pixel area information includes: the center point pixel coordinates determined based on the pixel coordinates of any point of the annotation box and the width and height of the annotation box as the elevator pixel position information. Of course, there may be other ways to determine the elevator pixel position information, which can be customized by the user, and the embodiments of the present application do not impose any restrictions.

[0102] According to the elevator pixel position information, the camera projection relationship is used to determine the relative position information of the elevator and the detection subject, and the relative position information is used as the actual position information of the elevator.

[0103] Specifically, for the embodiment of the present application, the position information of the elevator in the pixel coordinate system can be converted into the relative position information relative to the detection subject in the world coordinate system by using the camera projection relationship. The specific implementation process is as follows:

[0104] The relationship between the world coordinate system, camera coordinate system, image coordinate system, and pixel coordinate system can be referred to Figure 2 .

[0105] O w -X w Y w Z w : World coordinate system, describing the camera position, unit m; O c -X c Y c Z c : camera coordinate system, with the optical center as the origin, unit m; o-xy: image coordinate system, with the origin as the midpoint of the imaging plane, unit mm; uv: pixel coordinate system, with the origin as the upper left corner of the image, unit pixel; P: a point in the world coordinate system, that is, a real point in life; p: the imaging point of point p in the image, with coordinates (x, y) in the image coordinate system and (u, v) in the pixel coordinate system; f: camera focal length, equal to the distance between o and O c The distance, f = |oO c |.

[0106] (1) Coordinate conversion from world coordinate system to camera coordinate system.

[0107] The transformation from the world coordinate system to the camera coordinate system is a rigid body transformation, that is, the object will not be deformed, only rotation and translation are required. R and T are the external parameter matrices of the camera, where R is a 3*3 rotation matrix and T is a 3*1 translation matrix. Different cameras may have different R and T. Users can choose cameras according to actual needs. For details on the relationship between the world coordinate system and the camera coordinate system, please refer to Figure 3 , the transformation from the world coordinate system to the camera coordinate system only involves the rotation and translation process of the coordinate system, and we can get:

[0108]

[0109] Among them, for any point P in the world coordinate system, the coordinates of point P in the world coordinate system are (X w , Y w , Z w ), the coordinates of point P in the camera coordinate system are (X c , Y c , Z c ).

[0110] Convert the above formula into homogeneous coordinate form, which is expressed as follows:

[0111]

[0112] (2) Coordinate conversion from camera coordinate system to image coordinate system.

[0113] The specific relationship between the camera coordinate system and the image coordinate system can be referred to Figure 4 , where the coordinates of point A in the camera coordinate system are (0,0, Z c ), the coordinates of point B are (X c ,0,Z c ),O c The coordinates of point p are (0,0,0), and the coordinates of point o are (0,0,f); in the image coordinate system, the coordinates of point p are (x,y), and the coordinates of point o are (0,0).

[0114] According to the triangle similarity relationship, the conversion relationship between camera coordinates and image coordinates is derived. The specific derivation process is as follows: ΔABO c ~ΔoCO c ; ΔPBO c ~ΔpCO c ;

[0115]

[0116] Then we can get:

[0117] (3) Conversion between image coordinate system and pixel coordinate system.

[0118] The specific relationship between the image coordinate system and the pixel coordinate system can be found in Figure 5 , where the coordinates of point o in the image coordinate system are (0,0), and the coordinates of point o in the pixel coordinate system are (u0,v0), and then the transformation relationship between image coordinates and pixel coordinates is derived. Specifically: based on get

[0119] (4) Conversion equation from world coordinate system to pixel coordinate system.

[0120] Combining the conversion relationships of the four coordinate systems described above, we can get the conversion relationship between world coordinates and pixel coordinates, specifically:

[0121]

[0122] (u, v) is: the position information of the elevator in the pixel coordinate system, that is, the elevator pixel position information;

[0123] (X w ,Y w ,Z w ) is: the relative position information of the elevator relative to the detection subject in the world coordinate system, that is, the actual position information of the elevator.

[0124] When determining the relative position information, this application takes the ground as the standard and calculates X w , Y w The relative position information can be determined, so Z w Set to 0 and use Z in expressions const To represent Z w , both represent 0, so the above expression can be expressed as follows:

[0125]

[0126] in,

[0127] The matrix calculated according to the camera intrinsic parameters and camera extrinsic parameters has known values. According to the calculation process, the above projection relationship expression can be equivalent to:

[0128]

[0129] Therefore, the above three-variable cubic equation can be solved to obtain X w , Y w , Z c , we can get (X w , Y w ) is the actual relative coordinates between the elevator and the detection subject, that is, the relative position information, and the relative position is used as the actual position information of the elevator.

[0130] Therefore, according to the (u, v) elevator pixel position information, the camera projection relationship is used Determine the actual position information of the elevator (X w , Y w , Z w ), if the ground is taken as the standard, for Z w The value of is not required, so (X w , Y w ) can determine the actual position information of the elevator. w The value of is not limited in this embodiment.

[0131] It can be seen that in the embodiment of the present application, the position information of the elevator is determined by the elevator area information, and the relative position information of the elevator and the detection subject is determined by using the camera projection relationship. The elevator area information of the elevator in the pixel coordinate system can be quickly converted into the position relative to the detection subject in the world coordinate system, which is simple to operate and convenient to calculate.

[0132] Further, in order to reduce the danger in the warning area, in the embodiment of the present application, step S104 controls the movement of the detection subject according to the warning area, including:

[0133] If the detection body is in the warning area, the detection body is controlled to stop moving, or the detection body is controlled to move in a target movement direction, wherein the target movement direction is a direction away from the elevator area.

[0134] For the embodiments of the present application, specifically, in some embodiments, the actual position of the detection subject can be obtained by using positioning technology, and whether the detection subject enters the warning area can be determined based on the actual position of the detection subject, wherein the actual position of the detection subject can be obtained by using positioning technology, and one or more external sensors such as ultrasonic sensors, visual sensors, laser radars, etc. can be used to detect and obtain the surrounding environment characteristics, and then the absolute position of the detection subject in the reference coordinate system is estimated based on the surrounding environment characteristics to obtain the actual position of the detection subject. Furthermore, based on the actual position of the detection subject, when the detection subject enters the warning area, the detection subject can be controlled to stop moving to avoid danger to the detection subject; or, the detection subject can be controlled to move in a direction away from the elevator area, for example, the detection subject can be controlled to move in a target direction to first approach the elevator area and then move away from the elevator area, or the detection subject can be controlled to move in a target direction to move away from the elevator area, as long as it does not enter the elevator area, the embodiments of the present application will not be limited.

[0135] It can be seen that in the embodiment of the present application, when the detection subject is in the warning area, the detection subject is controlled to move away from the warning area or stop moving, thereby reducing the probability of danger occurring to the detection subject.

[0136] Furthermore, in order to effectively prevent the detection subject from being in danger at the elevator entrance, in an embodiment of the present application, the movement of the detection subject is controlled according to the warning area, including:

[0137] If the detection subject is in the warning area, it is determined whether the detection subject is in the elevator area;

[0138] If the detection subject is within the elevator area, the detection subject is controlled to stop moving, or the detection subject is controlled to move in the target movement direction.

[0139] For the embodiment of the present application, when the detection subject is in the warning area, it can be determined whether the detection subject is in the elevator area based on the actual position of the detection subject, wherein the elevator area can be the bed cover area at the elevator entrance, which can be specifically determined based on the elevator position information. If the detection subject is in the elevator area, the detection subject is controlled to stop moving or to move in the target movement direction, both of which are intended to prevent the detection subject from moving to the elevator entrance and causing danger.

[0140] It can be seen that in the embodiment of the present application, when the detection body is in the warning area and the detection body is in the elevator area, it is determined that the detection body is already in the elevator entrance part, and then an emergency stop method is taken or the detection body is controlled to move in the target movement direction to make the detection body stay away from the elevator area, which can effectively avoid the occurrence of danger.

[0141] Furthermore, the method for determining whether the detection subject is in the elevator area can be based on the vibration frequency of the detection subject, or can be based on the distance between the detection subject and the elevator position.

[0142] Therefore, for example, in one achievable method, determining whether the detection subject is in the elevator area includes:

[0143] Obtaining the vibration frequency of the detection subject and determining whether the vibration frequency is within a preset vibration frequency range;

[0144] If the vibration frequency is within the preset vibration frequency range, it is determined that the detection subject is in the elevator area.

[0145] For the embodiments of the present application, the electronic device can obtain the vibration frequency of the detection subject collected by the gyroscope; or, determine the vibration frequency of the detection subject through the vibration amplitude of the camera image, specifically, the electronic device pre-stores a correspondence between the vibration amplitude and the vibration frequency, and the correspondence is obtained by actual measurement; or, determine the vibration frequency of the detection subject through the slope change of the ground point cloud by a multi-line laser radar or a depth camera, specifically, the electronic device pre-stores a correspondence between the slope change and the vibration frequency, and the correspondence is obtained by actual measurement. A preset vibration frequency range is pre-stored in the electronic device, and when the vibration frequency is within the preset vibration frequency range, it is determined that the elevator area has been reached. The size of the preset vibration frequency range can be set by the user based on experience, or it can be based on data obtained by actual measurement. The process of determining data based on actual measurements may specifically include: placing the detection body in the elevator area to collect the actual vibration frequency, wherein multiple actual vibration frequencies can be obtained by performing multiple measurements at multiple locations in the elevator area, and the minimum vibration frequency and maximum vibration frequency of the detection body in the elevator area are calculated based on the multiple actual vibration frequencies. The minimum vibration frequency and the maximum vibration frequency are combined to form a preset vibration frequency range; of course, there may be other forms, which are no longer limited in the embodiments of the present application, and users can customize the settings.

[0146] It can be seen that in the embodiment of the present application, the vibration frequency of the detection subject can be obtained by the gyroscope, and it is determined whether the vibration frequency is within a preset vibration frequency range, wherein the preset vibration frequency range is pre-stored and can be set based on user experience or based on actual measurement. If the vibration frequency is within the preset vibration frequency range, it can be accurately determined that the detection subject is in the elevator area, thereby improving the accuracy of the determination.

[0147] For example, in another feasible method, the relative position information between the elevator and the detection subject is converted into the distance and direction between the detection subject and the elevator through a distance formula, and it is determined whether the distance between the detection subject and the elevator is less than a preset distance corresponding to the direction. If the distance between the two is less than the preset distance, it can be determined that the detection subject has entered the elevator area, wherein the preset distance can be set to the maximum value from the point where the relative position information is obtained to the boundary of the elevator area. Of course, a specific value can also be set, which is not limited in the embodiments of the present application.

[0148] Furthermore, in order to improve the accuracy of elevator area recognition, after obtaining the image to be detected, the following steps are also included:

[0149] Perform distortion correction on the image to be detected to obtain a corrected elevator image;

[0150] Accordingly, the elevator detection model is used to identify the elevator area of ​​the image to be detected, including:

[0151] The elevator detection model is used to identify the elevator area of ​​the rectified elevator image.

[0152] After obtaining the image to be detected, the embodiment of the present application can perform distortion correction on the detected image to obtain a corrected elevator image. The specific distortion correction method is not limited in the embodiment of the present application, and the user can set it according to actual needs as long as it can achieve the purpose of the embodiment of the present application.

[0153] By adopting the above technical solution, the present application first performs distortion correction on the image to be detected to obtain a corrected elevator image, and then uses the corrected elevator image to perform elevator area recognition, which can avoid the error problem of elevator area recognition caused by distortion and improve the recognition accuracy.

[0154] Furthermore, in order to improve the elevator recognition rate and the accuracy of the position, the training process of the elevator detection model includes: obtaining an elevator image training sample set, wherein the elevator image training sample set includes a plurality of elevator sample images and the corresponding elevator area annotation information;

[0155] The convolutional neural network is trained using the elevator image training sample set to obtain the elevator detection model.

[0156] In the embodiment of the present application, elevator sample images can be obtained from the network, local storage, etc. The elevator sample images can be images of multiple shopping malls. After obtaining the elevator sample images, the elevator sample images can be expanded by cropping, affine transformation, integration, etc., so that the model obtained by subsequent training based on these elevator sample images has higher accuracy and better recall rate. The specific annotation method can be manual annotation or automatic annotation, which is no longer limited in the embodiment of the present application. The user can annotate according to actual needs. The elevator area annotation information corresponding to the elevator sample image can be in the form of an annotation box, which can include any of the following: structural feature information of the bottom, front, and side of the elevator. Specifically, when annotating the structural feature information of the bottom of the elevator, the distance between the two handrails of the elevator can be used as the width of the annotation box, and the distance from the elevator floor plate to the bottom of the elevator handrail can be used as the height of the annotation box; when annotating the structural feature information of the front of the elevator, the distance between the two handrails of the elevator can be used as the width of the annotation box, and the distance from the elevator floor plate to the top of the elevator handrail can be used as the height of the annotation box; for the structural feature information of the side elevator, the unstepped part of the apron plate at the starting point of the elevator is used as the width of the annotation box, and the height of the annotation box is used from the ground to the top of the elevator handrail.

[0157] The convolutional neural network is trained using the elevator image training sample set to obtain an elevator detection model for detecting elevator areas. Specifically, the training elevator area information is obtained through the convolutional neural network based on multiple elevator sample images; for each elevator sample image, the similarity of the training elevator area information and the elevator area annotation information is determined; based on the similarity of multiple elevator sample images, the loss is obtained, and the loss is back-propagated to train the convolutional neural network; the weighted summation of each loss of the trained convolutional neural network is performed to obtain the total loss, and when the total loss meets the set loss threshold range, the trained convolutional neural network is determined to be an elevator detection model. Among them, in the embodiment of the present application, the convolutional neural network can be various convolutional networks, for example, Resnet network, yolov5 network.

[0158] Further, in an embodiment of the present application, using the elevator image training sample set to train the convolutional neural network to obtain the elevator detection model may include: using the elevator image training sample set to train the convolutional neural network to obtain a first elevator detection model; using the test set to test the first elevator detection model to obtain a test result; when the test result meets the preset result threshold, the first elevator detection model is determined to be the final elevator detection model; when the test result does not meet the preset result threshold, the first elevator detection model is retrained using the training set to obtain a second elevator detection model, and the second elevator detection model is tested using the test set until the final elevator detection model that meets the preset result threshold is obtained. Further, after the trained elevator detection model is tested using the test set, the test images that failed the test can be added to the elevator image training sample set to update the elevator image training sample set, which can effectively improve the training effect. When the detection subject calls the elevator detection model, the image to be detected can be processed to obtain the elevator area information.

[0159] It can be seen that in the embodiment of the present application, the convolutional neural network is trained using the elevator image training sample set to obtain an elevator detection model, which improves the elevator recognition rate and also makes the acquired elevator position more accurate.

[0160] Furthermore, in order to improve the recognition rate of the elevator detection model, an elevator image training sample set is obtained, including:

[0161] Acquire a first training set, wherein the first training set includes a plurality of first elevator image training samples and respective corresponding elevator area annotation information;

[0162] For the embodiment of the present application, the corresponding elevator area annotation information is information obtained by annotating the bottom, front or side features of the elevator according to the characteristics of the elevator in the first elevator image training sample.

[0163] Obtaining a second training set, wherein the second training set includes a plurality of second elevator image training samples and respective corresponding elevator region annotation information, wherein the plurality of second elevator image training samples are obtained by cutting and recombining the plurality of first elevator image training samples;

[0164] For the embodiment of the present application, the specific operation of cutting and recombining the multiple first elevator image training samples is to intercept the multiple first elevator image training samples according to different positions and different sizes, and combine any two or more of the intercepted images into a second elevator image training sample, wherein the number of images to be assembled and the proportion of the image sizes can be set according to actual conditions. Then, the multiple second elevator image training samples and their corresponding annotation information constitute the second training set.

[0165] The first training set and the second training set are used as elevator image training sample sets.

[0166] It can be seen that in the embodiment of the present application, the second training set is obtained by cutting and recombining multiple first elevator image training samples. The first training set and the second training set are both used as elevator image training sample sets to train the convolutional neural network. The cutting and recombining methods are used to enrich the training set, thereby improving the model training effect, so that the elevator detection model has a higher recognition rate.

[0167] The above-mentioned embodiment introduces a control method from the perspective of method flow, and the following embodiment introduces a control device from the perspective of a virtual module or a virtual unit. Please refer to the following embodiment for details.

[0168] The present application embodiment provides a control device 200, such as Figure 6 As shown, the control device 200 may specifically include: a control device 200, including:

[0169] An image acquisition module 210 is used to acquire an image to be detected;

[0170] The elevator area recognition module 220 is used to perform elevator area recognition on the image to be detected using the elevator detection model to obtain a recognition result;

[0171] An actual position information determination module 230, when the recognition result includes elevator pixel area information, determines the actual position information of the elevator according to the elevator pixel area information;

[0172] The control module 240 determines the warning area according to the actual position information of the elevator, and controls the movement of the detection body according to the warning area.

[0173] It can be seen that the embodiment of the present application can obtain the image to be detected, and the elevator detection model can be used to perform elevator area recognition based on the image to be detected. When the image to be detected includes an elevator image, accurate elevator pixel area information can be identified, and then the actual position information of the elevator can be determined based on the elevator pixel area information. Among them, the elevator pixel area information is accurate data obtained through the elevator detection model, and thus the actual position information of the elevator determined based on the elevator pixel area information is also accurate, and then the warning area can be determined based on the actual position information of the elevator, and the movement of the detection subject can be controlled based on the warning area to reduce the occurrence of the detection subject falling due to incorrect identification of the escalator.

[0174] Preferably, when determining the actual position information of the elevator according to the elevator pixel area information, the actual position information determining module 230 is used to:

[0175] Determine the elevator pixel position information according to the elevator pixel area information;

[0176] According to the pixel position information of the elevator, the relative position information of the elevator and the detection subject is determined by using the camera projection relationship.

[0177] Preferably, when executing the control of detecting the movement of the subject according to the warning area, the control module 240 is used to:

[0178] When the detection body is in the warning area, the detection body is controlled to stop moving, or the detection body is controlled to move in a target movement direction, wherein the target movement direction is a direction away from the elevator area.

[0179] Preferably, when executing the control of detecting the movement of the subject according to the warning area, the control module 240 is used to:

[0180] When the detection subject is in the warning area, determine whether the detection subject is in the elevator area;

[0181] When the detection body is in the elevator area, the detection body is controlled to stop moving, or the detection body is controlled to move in the target movement direction.

[0182] Preferably, when the control module 240 determines whether the detection subject is in the elevator area, it is used to:

[0183] Obtaining the vibration frequency of the detection subject and determining whether the vibration frequency is within a preset vibration frequency range;

[0184] When the vibration frequency is within a preset vibration frequency range, it is determined that the detection subject is in the elevator area.

[0185] Preferably, the control device 200 further includes:

[0186] A distortion correction module is used to perform distortion correction on the image to be detected to obtain a corrected elevator image;

[0187] Accordingly, when the elevator area recognition module 220 performs elevator area recognition on the image to be detected using the elevator detection model, it is used to:

[0188] The elevator detection model is used to identify the elevator area of ​​the rectified elevator image.

[0189] Preferably, the control device 200 further includes:

[0190] An elevator image training sample set acquisition module is used to acquire an elevator image training sample set, wherein the elevator image training sample set includes a plurality of elevator sample images and their corresponding elevator area annotation information;

[0191] The training module is used to train the convolutional neural network using the elevator image training sample set to obtain the elevator detection model.

[0192] Preferably, when executing the step of acquiring the elevator image training sample set, the elevator image training sample set acquisition module is used to: acquire a first training set, wherein the first training set includes a plurality of first elevator image training samples and the corresponding elevator area annotation information;

[0193] Obtaining a second training set, wherein the second training set includes a plurality of second elevator image training samples and respective corresponding elevator region annotation information, wherein the plurality of second elevator image training samples are obtained by cutting and recombining the plurality of first elevator image training samples;

[0194] The first training set and the second training set are used as elevator image training sample sets.

[0195] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the device 200 described above can refer to the corresponding process in the aforementioned method embodiment, and will not be repeated here.

[0196] An electronic device is provided in an embodiment of the present application, such as Figure 7 As shown, Figure 7 The electronic device 300 shown includes: a processor 301 and a memory 303. The processor 301 and the memory 303 are connected, such as through a bus 302. Optionally, the electronic device 300 may also include a transceiver 304. It should be noted that in actual applications, the transceiver 304 is not limited to one, and the structure of the electronic device 300 does not constitute a limitation on the embodiments of the present application.

[0197] Processor 301 may be a CPU (Central Processing Unit), a general purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array) or other programmable logic devices, transistor logic devices, hardware components or any combination thereof. It may implement or execute various exemplary logic blocks, modules and circuits described in conjunction with the disclosure of this application. Processor 301 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, a combination of a DSP and a microprocessor, etc.

[0198] The bus 302 may include a path to transmit information between the above components. The bus 302 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. The bus 302 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0199] The memory 303 can be a ROM (Read Only Memory) or other types of static storage devices that can store static information and instructions, a RAM (Random Access Memory) or other types of dynamic storage devices that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compressed optical disk, laser disk, optical disk, digital versatile disk, Blu-ray disk, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store the desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited to these.

[0200] The memory 303 is used to store the application code for executing the solution of the present application, and the execution is controlled by the processor 301. The processor 301 is used to execute the application code stored in the memory 303 to implement the contents shown in the above method embodiment.

[0201] The electronic devices include, but are not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), and fixed terminals such as digital TVs, desktop computers, etc. It can also be a server, etc. Figure 7 The electronic device shown is merely an example and should not bring any limitation to the functions and scope of use of the embodiments of the present application.

[0202] The embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding content in the aforementioned method embodiment. Compared with the related art, the embodiment of the present application can use the elevator detection model to identify the elevator area based on the image to be detected. When the image to be detected includes an elevator image, accurate elevator pixel area information can be identified, and then the actual position information of the elevator can be determined based on the elevator pixel area information. The elevator pixel area information is accurate data obtained by the elevator detection model, and the actual position information of the elevator determined based on the elevator pixel area information is also accurate. Then, the warning area can be determined based on the actual position information of the elevator, and the movement of the detection subject can be controlled based on the warning area to reduce the occurrence of the detection subject falling due to the escalator.

[0203] It should be understood that, although the steps in the flowchart of the accompanying drawings are displayed in sequence as indicated by the arrows, these steps are not necessarily executed in sequence in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least a part of the steps in the flowchart of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a part of the sub-steps or stages of other steps.

[0204] The above are only some implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A control method, characterized in that: include: Acquire the image to be detected; Using the elevator detection model to perform elevator area recognition on the image to be detected to obtain a recognition result; When the recognition result includes elevator pixel area information, determining the actual position information of the elevator according to the elevator pixel area information; Determine a warning area according to the actual position information of the elevator, and control the movement of the detection subject according to the warning area; Determining the actual position information of the elevator according to the elevator pixel area information includes: Determine the elevator pixel position information according to the elevator pixel area information, wherein the elevator pixel area information is: the pixel coordinates of any point in the elevator feature annotation box, the height and width of the pixel box; the elevator pixel position information is the pixel coordinates of the center point corresponding to the elevator feature annotation box; According to the elevator pixel position information, the relative position information of the elevator and the detection subject is determined by using the camera projection relationship, and the relative position information is used as the actual position information of the elevator, wherein the conversion relationship between the world coordinates and the pixel coordinates in the camera projection relationship is: ； is the elevator pixel position information; The actual location information of the elevator; is the focal length of the camera; R and T are the external parameter matrices of the camera, R is a 3*3 rotation matrix, and T is a 3*1 translation matrix; the coordinates in the world coordinate system are ( , , ) point, the coordinates in the camera coordinate system are ( , , ); The point with coordinates (0,0) in the image coordinate system has coordinates ( , ); , ; The controlling the movement of the detection subject according to the warning area comprises: If the detection body is within the warning area, obtaining the vibration frequency of the detection body, and determining whether the vibration frequency is within a preset vibration frequency range; If the vibration frequency is within the preset vibration frequency range, it is determined that the detection subject is within the elevator area; If the detection body is within the elevator area, the detection body is controlled to stop moving, or the detection body is controlled to move in a target movement direction.

2. The control method according to claim 1, characterized in that: The controlling the movement of the detection subject according to the warning area comprises: The target movement direction is a direction away from the elevator area.

3. The control method according to any one of claims 1 to 2, characterized in that: The training process of the elevator detection model includes: Obtain an elevator image training sample set, wherein the elevator image training sample set includes a plurality of elevator sample images and respective corresponding elevator area annotation information; The elevator image training sample set is used to train a convolutional neural network to obtain the elevator detection model.

4. The control method according to claim 3, characterized in that: The step of obtaining an elevator image training sample set includes: Acquire a first training set, wherein the first training set includes a plurality of first elevator image training samples and respective corresponding elevator area annotation information; Acquire a second training set, wherein the second training set includes a plurality of second elevator image training samples and respective corresponding elevator region annotation information, and the plurality of second elevator image training samples are obtained by cutting and recombining the plurality of first elevator image training samples; The first training set and the second training set are used as the elevator image training sample sets.

5. A control device, characterized in that: include: An image acquisition module, used for acquiring an image to be detected; An elevator area recognition module, used to perform elevator area recognition on the image to be detected using an elevator detection model to obtain a recognition result; an actual position information determination module, configured to determine the actual position information of the elevator according to the elevator pixel area information when the recognition result includes the elevator pixel area information; A control module, which determines a warning area according to the actual position information of the elevator, and controls the movement of the detection body according to the warning area; When the actual position information determining module determines the actual position information of the elevator according to the elevator pixel area information, it is used to: Determine the elevator pixel position information according to the elevator pixel area information, wherein the elevator pixel area information is: the pixel coordinates of any point in the elevator feature annotation box, the height and width of the pixel box; the elevator pixel position information is the pixel coordinates of the center point corresponding to the elevator feature annotation box; According to the elevator pixel position information, the relative position information of the elevator and the detection subject is determined by using the camera projection relationship, and the relative position information is used as the actual position information of the elevator, wherein the conversion relationship between the world coordinates and the pixel coordinates in the camera projection relationship is: ； is the elevator pixel position information; The actual location information of the elevator; is the focal length of the camera; R and T are the external parameter matrices of the camera, R is a 3*3 rotation matrix, and T is a 3*1 translation matrix; the coordinates in the world coordinate system are ( , , ) point, the coordinates in the camera coordinate system are ( , , ); The point with coordinates (0,0) in the image coordinate system has coordinates ( , ); , ; When executing the control of detecting the movement of the subject according to the warning area, the control module is used to: If the detection body is within the warning area, obtaining the vibration frequency of the detection body, and determining whether the vibration frequency is within a preset vibration frequency range; If the vibration frequency is within the preset vibration frequency range, it is determined that the detection subject is within the elevator area; If the detection body is within the elevator area, the detection body is controlled to stop moving, or the detection body is controlled to move in a target movement direction.

6. An electronic device, characterized in that: include: at least one processor; Memory; At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute the method according to any one of claims 1 to 4.

7. A computer-readable storage medium, characterized in that: The storage medium stores at least one instruction, at least one program, a code set or an instruction set, and the at least one instruction, the at least one program, the code set or the instruction set is loaded and executed by the processor to implement the method according to any one of claims 1 to 4.

Images