Method, device, electronic device and readable storage medium for determining hoisting status

Abstract

The present application provides a method, device, electronic device and readable storage medium for determining a hoisting status, and belongs to the field of hoisting technology. The method comprises: acquiring a two-dimensional image containing a hoisting scene sent by a shooting device; when it is determined that the two-dimensional image includes a target hoist, obtaining the target area and target two-dimensional coordinates of the target hoist in the two-dimensional image by semantic segmentation technology, wherein the target hoist is a target boom or a target hanging object; determining the target three-dimensional coordinates of the target hoist in the actual space by a perspective transformation scheme, wherein the target two-dimensional coordinates and the target area are used to determine the depth coordinates and height coordinates of the target hoist in the target three-dimensional coordinates; determining the current hoisting status of the target hoist according to the target three-dimensional coordinates. The present application improves the accuracy of detecting the hoisting status of a hoist in a scenario where no additional signal support can be provided.

Description

Technical Field

[0001] The present application relates to the field of hoisting technology, and in particular to a method, device, electronic device and readable storage medium for determining a hoisting state. Background Art

[0002] In the hoisting scenario, manual participation is required during the hoisting process, including tying up the objects and hanging them on the boom before hoisting, and checking or tightening the tying of the objects when they are temporarily placed on the ground. To ensure personal safety, if the objects are in the hoisted state, an alarm will be issued if someone enters the hoisting area during the hoisting process.

[0003] The current method of detecting hoisting is to mark the hoisting parts (boom or hoisting object) in the two-dimensional image through a detection frame. If it is determined that the position of the detection frame exists in the hoisting area, it is determined that the hoisting parts are in the hoisted state, and personnel are not allowed to enter the hoisting area.

[0004] Since the detection frame cannot be close to the detection hanger, the accuracy of judging whether the hanger is in the hoisted state by the position of the detection frame is low. In addition, since the two-dimensional image lacks accurate actual depth of field information, when the hanger is at a certain height, it is impossible to accurately judge whether the hanger is placed on the ground or suspended in the air, nor is it possible to accurately judge whether the hanger is in the hoisted state. This results in low accuracy in detecting the hoisting state of the hanger. If someone enters the hoisting area during the hoisting process, safety accidents are likely to occur. Summary of the invention

[0005] The purpose of the embodiments of the present application is to provide a method, device, electronic device and readable storage medium for determining the hoisting state, so as to solve the problem of low accuracy in detecting whether the hoisting part has left the ground. The specific technical solution is as follows:

[0006] In a first aspect, a method for determining a hoisting state is provided, the method comprising:

[0007] Acquire a two-dimensional image including a hoisting scene sent by a shooting device;

[0008] When it is determined that the two-dimensional image includes a target hanging part, obtaining a target area and a target two-dimensional coordinate of the target hanging part in the two-dimensional image by using a semantic segmentation technology, wherein the target hanging part is a target hanging arm or a target hanging object;

[0009] Determine the target three-dimensional coordinates of the target hanging object in the actual space by a perspective transformation scheme, wherein the target two-dimensional coordinates and the target area are used to determine the depth coordinates and the height coordinates of the target hanging object in the target three-dimensional coordinates;

[0010] The current hoisting state of the target hanging component is determined according to the target three-dimensional coordinates.

[0011] Optionally, before obtaining the target area and target two-dimensional coordinates of the target hanging part in the two-dimensional image by using a semantic segmentation technology, the method further includes:

[0012] Inputting the two-dimensional image into a target detection model to obtain a detection result output by the target detection model;

[0013] In a case where the detection result is a detection frame of the target hanging part, determining that the target hanging part is included in the two-dimensional image, wherein the detection frame includes a target image of the target hanging part;

[0014] The target image is segmented from the two-dimensional image.

[0015] Optionally, obtaining the target area of ​​the target hanging part in the two-dimensional image by using a semantic segmentation technology includes:

[0016] Inputting the target image into a semantic segmentation model to obtain two-dimensional matrix elements output by the semantic segmentation model, wherein each of the two-dimensional matrix elements is used to indicate a pixel point in the two-dimensional image, and each of the pixel points has a corresponding category label;

[0017] Determine a target pixel point of the target hanging part according to the category label;

[0018] The number of pixels of the target pixel point is used as the target area of ​​the target hanging part.

[0019] Optionally, obtaining the target two-dimensional coordinates of the target hanging part in the two-dimensional image by using a semantic segmentation technology includes:

[0020] Determine a first two-dimensional coordinate of the detection frame in the two-dimensional image, and a second two-dimensional coordinate of the target hanging part in the detection frame;

[0021] The sum of the first two-dimensional coordinate and the second two-dimensional coordinate is used as the target two-dimensional coordinate of the target hanging part in the two-dimensional image.

[0022] Optionally, before determining the target three-dimensional coordinates of the target hanging component in the actual space by using a perspective transformation scheme, the method further includes:

[0023] Determine the sample height coordinate and the sample depth coordinate of the sample hanging part in the actual space;

[0024] Acquire a sample two-dimensional image corresponding to the sample height coordinate and the sample depth coordinate by a photographing device;

[0025] Obtaining the sample area and sample two-dimensional coordinates of the sample hanging part in the sample two-dimensional image by using the target detection model and the semantic segmentation technology;

[0026] A corresponding relationship between the sample height coordinate, the sample depth of field coordinate, the sample area, and the sample two-dimensional coordinate is established, and the corresponding relationship is stored in a database.

[0027] Optionally, determining the current hoisting state of the target hanging component according to the target three-dimensional coordinates includes:

[0028] When it is determined that the target hanging component is located in the hanging area according to the lateral coordinates and the depth coordinates in the target three-dimensional coordinates, obtaining the height coordinates of the target hanging component;

[0029] When it is determined that the height coordinate exceeds the first height threshold, it is determined that the target hanging component is currently in a hoisted state.

[0030] Optionally, after obtaining the height coordinate of the target hanging component, the method further includes:

[0031] When it is determined that the height coordinate is greater than a second height threshold and less than the first height threshold, acquiring a target positive slope of the target hanging object, wherein the target hanging object is a strip-shaped object arranged in a horizontal direction, and the target positive slope is used to indicate a degree of inclination of the target hanging object;

[0032] When the target positive slope satisfies a preset slope range, it is determined that the target hanging component is currently in a hoisted state.

[0033] In a second aspect, a device for determining a hoisting state is provided, the device comprising:

[0034] An acquisition module, which acquires a two-dimensional image including a hoisting scene sent by a shooting device;

[0035] an obtaining module, for obtaining a target area and a target two-dimensional coordinate of the target hanging part in the two-dimensional image by using a semantic segmentation technology when it is determined that the target hanging part is included in the two-dimensional image, wherein the target hanging part is a target hanging arm or a target hanging object;

[0036] A first determination module is used to determine the target three-dimensional coordinates of the target hanging object in the actual space through a perspective transformation scheme, wherein the target two-dimensional coordinates and the target area are used to determine the depth coordinates and height coordinates of the target hanging object in the target three-dimensional coordinates;

[0037] The second determining module is used to determine the current hoisting state of the target hanging component according to the target three-dimensional coordinates.

[0038] In a third aspect, an electronic device is provided, comprising a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other via the communication bus;

[0039] Memory, used to store computer programs;

[0040] The processor is used to implement any of the steps of the method for determining the hoisting state when executing the program stored in the memory.

[0041] In a fourth aspect, a computer-readable storage medium is provided, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, any of the steps of the method for determining the hoisting status is implemented.

[0042] Beneficial effects of the embodiments of the present application:

[0043] An embodiment of the present application provides a method for determining a hoisting status, the method comprising: acquiring a two-dimensional image containing a hoisting scene sent by a photographing device; when it is determined that the two-dimensional image includes a target hoist, obtaining a target area and a target two-dimensional coordinate of the target hoist in the two-dimensional image by using a semantic segmentation technique, wherein the target hoist is a target boom or a target hanging object; determining a target three-dimensional coordinate of the target hoist in actual space by using a perspective transformation scheme, wherein the target two-dimensional coordinate and the target area are used to determine a depth coordinate and a height coordinate of the target hoist in the target three-dimensional coordinate; and determining the current hoisting status of the target hoist according to the target three-dimensional coordinate.

[0044] The present application obtains the depth coordinates and height coordinates of the target three-dimensional coordinates of the target hanging part by detecting the target two-dimensional coordinates and the target area of ​​the target hanging part. In this way, the target three-dimensional coordinates of the target hanging part can be used to detect whether the target hanging part is located in the hoisting area and whether it is off the ground, thereby improving the accuracy of detecting the hoisting state of the hanging part. In addition, the present application does not rely on the detection frame to determine whether the target hanging part is in the hoisted state, thus avoiding the problem of low fit between the detection frame and the target hanging part, and also improving the accuracy of the detection.

[0045] Of course, implementing any product or method of the present application does not necessarily require achieving all of the above advantages at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0046] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, for ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative labor.

[0047] Figure 1 A schematic diagram of the hardware environment of a method for determining a hoisting state provided in an embodiment of the present application;

[0048] Figure 2 A flow chart of a method for determining a hoisting state provided in an embodiment of the present application;

[0049] Figure 3 A processing flow chart of a method for determining a hoisting state provided in an embodiment of the present application;

[0050] Figure 4 A schematic diagram of the structure of a device for determining a hoisting state provided in an embodiment of the present application;

[0051] Figure 5 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0052] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0053] In the subsequent description, the suffixes such as "module", "component" or "unit" used to represent elements are only used to facilitate the description of the present application and have no specific meaning. Therefore, "module" and "component" can be used interchangeably.

[0054] In order to solve the problem mentioned in the background technology, according to one aspect of an embodiment of the present application, an embodiment of a method for determining a hoisting state is provided.

[0055] Optionally, in the embodiment of the present application, the above-mentioned method for determining the hoisting state can be applied to: Figure 1 In the hardware environment composed of the photographing device 101 and the server 103 shown in FIG. Figure 1 As shown, the server 103 is connected to the shooting device 101 via a network, and can be used to provide services for a terminal or a client installed on the terminal. A database 105 can be set on the server or independently of the server to provide data storage services for the server 103. The above-mentioned network includes but is not limited to: a wide area network, a metropolitan area network or a local area network, and the shooting device 101 includes but is not limited to a camera, a camcorder, etc.

[0056] A method for determining a hoisting state in an embodiment of the present application may be executed by the photographing device 101 , may be executed by the server 103 , or may be executed jointly by the server 103 and the photographing device 101 .

[0057] An embodiment of the present application provides a method for determining a hoisting status, which can be applied to a server to determine the hoisting status of a target hanging component.

[0058] The following will describe in detail a method for determining a hoisting state provided by an embodiment of the present application in conjunction with a specific implementation manner. Figure 2 As shown, the specific steps are as follows:

[0059] Step 201: Acquire a two-dimensional image including a hoisting scene sent by a shooting device.

[0060] In the embodiment of the present application, a shooting device is provided in the hoisting scene, and the shooting device sends a two-dimensional image containing the hoisting scene to the server, and the server receives the two-dimensional image sent by the shooting device. The server determines whether the two-dimensional image contains the target hanging part. If the server determines that the two-dimensional image does not contain the target hanging part, it indicates that no hoisting is currently being performed in the hoisting scene, and the target hanging part is in an un-hoisted state, and then the two-dimensional image continues to be acquired. If the server determines that the two-dimensional image contains the target hanging part, step 202 is executed.

[0061] The target hoisting part is a target boom or a target hoisting object. For example, the hoisting scene is the hoisting of bars in a bar workshop of a steel plant.

[0062] As an optional implementation, before obtaining the target area and target two-dimensional coordinates of the target hanger in the two-dimensional image through semantic segmentation technology, the method also includes: inputting the two-dimensional image into a target detection model to obtain a detection result output by the target detection model; when the detection result is a detection frame of the target hanger, determining that the two-dimensional image includes the target hanger, wherein the detection frame contains a target image of the target hanger; and cutting the target image out of the two-dimensional image.

[0063] In the present application, the specific method for the server to determine whether the target hanging part is included in the two-dimensional image is as follows: the server inputs the two-dimensional image into the target detection model, and obtains the detection result output by the target detection model. If the server determines that the detection result is empty, it is determined that the target hanging part is not included in the two-dimensional image. If the server determines that the detection result is a detection frame of the target hanging part, it is determined that the target hanging part is included in the two-dimensional image, and the target image containing the target hanging part in the detection frame is cut out from the two-dimensional image.

[0064] Due to the complexity of actual industrial scenes, cluttered backgrounds, and large differences in light between morning and evening, if semantic segmentation or instance segmentation is performed directly on two-dimensional images, inaccurate image content recognition is likely to occur. This application obtains a detection frame of the target hanger through a target detection model. The image content in the detection frame is smaller than the image content in the two-dimensional image, and the server performs a semantic segmentation model on the target image of the detection frame. Since the image content in the detection frame is smaller than the image content in the two-dimensional image, the complexity of the semantic segmentation model calculation is reduced, and the amount of calculation is also reduced, which improves the efficiency and accuracy of the semantic segmentation model output of the two-dimensional matrix elements, and also improves the efficiency and accuracy of determining the hoisting status. Among them, the target detection model can be a YOLO model, and this application does not make specific restrictions on the target detection model.

[0065] Step 202: when it is determined that the two-dimensional image includes the target hanging part, the target area and target two-dimensional coordinates of the target hanging part in the two-dimensional image are obtained by using semantic segmentation technology.

[0066] The target hanging part is a target hanging arm or a target hanging object.

[0067] In an embodiment of the present application, if the server determines that the two-dimensional image includes a target hanger, the target hanger in the target image is identified through semantic segmentation technology, and then the target area of ​​the target hanger in the two-dimensional image is obtained based on the number of pixels of the target hanger in the two-dimensional image, and the target two-dimensional coordinates of the target hanger in the two-dimensional image are obtained based on the coordinates of the detection frame in the two-dimensional image and the coordinates of the target hanger in the detection frame.

[0068] Step 203: Determine the target three-dimensional coordinates of the target hanging part in the actual space through a perspective transformation scheme.

[0069] The target two-dimensional coordinates and the target area are used to determine the depth coordinates and height coordinates of the target hanging part in the target three-dimensional coordinates.

[0070] The server determines the horizontal coordinates of the target hanging part in the target three-dimensional coordinates through the perspective transformation scheme. The database stores the correspondence between the two-dimensional coordinates and area of ​​the target hanging part in the two-dimensional image and the depth coordinates and height coordinates in the actual three-dimensional coordinates. Therefore, the server can obtain the depth coordinates and height coordinates of the target hanging part in the target three-dimensional coordinates according to the target two-dimensional coordinates and target area of ​​the target hanging part. The server uses the horizontal coordinates, depth coordinates and height coordinates as the target three-dimensional coordinates of the target hanging part.

[0071] Step 204: Determine the current hoisting state of the target hanging component according to the target three-dimensional coordinates.

[0072] The server determines the current hoisting state of the target hanging part according to the target three-dimensional coordinates, and the hoisting state includes the unhoisted state or the hoisted state. Specifically, it determines whether the target hanging part is located in the hoisting area according to the horizontal coordinates and depth coordinates in the target three-dimensional coordinates, or determines whether the target hanging part is located in the hoisting area according to the x-coordinates and y-coordinates in the target two-dimensional coordinates, and then determines whether the target hanging part leaves the ground or the bracket holding the target hanging part according to the height coordinates.

[0073] In the present application, the server obtains the depth coordinates and height coordinates in the target three-dimensional coordinates of the target hanging part by detecting the target two-dimensional coordinates and target area of ​​the target hanging part. In this way, the target three-dimensional coordinates of the target hanging part can be used to detect whether the target hanging part is located in the hoisting area and whether it is off the ground, thereby improving the accuracy of detecting the hoisting state of the hanging part. Based on the prior art, the present application adds the hanging part area parameter and realizes the conversion from two-dimensional coordinates to three-dimensional coordinates.

[0074] In addition, this application does not rely on the detection frame to determine whether the target hanging part is in the hoisted state, which avoids the problem of low fit between the detection frame and the target hanging part and improves the accuracy of detection. Moreover, for old factories, this application does not need to provide additional signal support and can also achieve accurate detection of the hoisting state, reducing production costs.

[0075] As an optional implementation, determining the current hoisting state of the target hanger according to the target three-dimensional coordinates includes: determining that the target hanger is located in the hoisting area according to the lateral coordinates and depth coordinates in the target three-dimensional coordinates, and obtaining the height coordinates of the target hanger; and determining that the target hanger is currently in a hoisted state when it is determined that the height coordinate exceeds a first height threshold.

[0076] In the embodiment of the present application, the server determines whether the target hanging part is located in the hanging area according to the lateral coordinates and the depth coordinates in the target three-dimensional coordinates. Optionally, the server can also determine whether the target hanging part is located in the hanging area according to the x-coordinate and the y-coordinate in the target two-dimensional coordinates. If the server determines that the target hanging part is not located in the hanging area, it indicates that the target hanging part is currently in a non-hanging state, and personnel are allowed to enter the hanging area.

[0077] If the server determines that the target hanging part is in the hoisting area, it is necessary to further determine the height coordinates of the target hanging part. If the server determines that the height coordinates of the target hanging part are greater than the first height threshold, it indicates that the current height of the target hanging part is high and the target hanging part is currently in the hoisted state. Personnel are not allowed to enter the hoisting area, and a warning signal can be given, such as controlling a warning light to light up or controlling a warning bell to sound.

[0078] The first height threshold may be the height of the target hanging part from the ground, or the height of the target hanging part from a bracket, where the bracket is used to carry the target hanging part to be hoisted.

[0079] In the present application, the server can determine the hoisting state of the target hoisting part according to whether the target hoisting part is located in the hoisting area, and can further determine the hoisting state according to the height of the target hoisting part, and improve the accuracy of determining the hoisting state by combining the hoisting area and the hoisting height. When it is determined that the target hoisting part is in the hoisted state, an alarm is issued to avoid safety accidents.

[0080] As an optional implementation, after obtaining the height coordinates of the target hanging object, the method also includes: when it is determined that the height coordinates are greater than a second height threshold and less than a first height threshold, obtaining a target positive slope of the target hanging object, wherein the target hanging object is a horizontally arranged strip-shaped object, and the target positive slope is used to indicate the degree of inclination of the target hanging object; when the target positive slope satisfies a preset slope range, it is determined that the target hanging object is currently in a hoisted state.

[0081] In an embodiment of the present application, if the server determines that the height coordinate of the target hanging component is greater than the second height threshold and less than the first height threshold, it indicates that the current height of the target hanging component is not very high. The server is not sure whether the target hanging component is in a temporary placement state or in a hoisted state. At this time, the positive slope of the target hanging component can be used to judge the hoisting state of the target hanging component.

[0082] The target hanging piece is a strip-shaped object arranged horizontally. When the target hanging piece is in the hoisted state, the target hanging piece is in a horizontal state or in a state where the right side is higher and the left side is lower. When the target hanging piece is in a state where the right side is higher and the left side is lower, the slope of the target hanging piece is a positive slope. When the target hanging piece is in a state where the left side is higher and the right side is lower, the slope of the target hanging piece is a negative slope. Therefore, the positive slope of the target hanging piece can be used to determine whether the target hanging piece is currently in the hoisted state. Among them, the server can use the Hough transform algorithm to obtain the upper edge of the target hanging piece and calculate its positive slope.

[0083] Specifically, the server obtains the target positive slope of the target hanging part. If the server determines that the target positive slope satisfies the preset slope range, it is determined that the target hanging part is currently in the hoisted state. If the server determines that the target positive slope does not satisfy the preset slope range, it is determined that the target hanging part is currently in the unhoisted state. The preset slope range is the normal tilt angle range when the target hanging part is in the hoisted state. The tilt angle range can be set according to the actual hoisting scene, and this application does not make specific restrictions.

[0084] In the present application, when the server cannot determine whether the target hanging part is in a temporary placement state or in a hoisted state, the target positive slope of the target hanging part can be used to judge the hoisting state of the target hanging part, thereby improving the accuracy of hoisting state determination.

[0085] As an optional implementation, obtaining the target area of ​​the target hanging part in the two-dimensional image through semantic segmentation technology includes: inputting the target image into a semantic segmentation model to obtain two-dimensional matrix elements output by the semantic segmentation model, wherein each two-dimensional matrix element is used to indicate a pixel point in the two-dimensional image, and each pixel point has a corresponding category label; determining the target pixel point of the target hanging part according to the category label; and taking the number of pixels of the target pixel point as the target area of ​​the target hanging part.

[0086] In an embodiment of the present application, the server inputs the target image in the detection frame into the semantic segmentation model, and the semantic segmentation model recognizes the content of the target image to obtain multiple two-dimensional matrix elements, each of which indicates a pixel in the two-dimensional image, and each pixel has a corresponding category label, that is, the semantic segmentation model outputs a two-dimensional matrix with a category label. The server can determine multiple target pixels of the target hanging part according to the category label, and then use the number of pixels of the target pixel as the target area of ​​the target hanging part.

[0087] Exemplarily, the semantic segmentation model recognizes that the target image includes a hanging background and a target hanging object, and sets the category label of the hanging background to 1 and the category label of the target hanging object to 2. Then the server determines the number of pixels with a category label of 2. If the server determines that the number of pixels is 300, the target area of ​​the target hanging object is 300.

[0088] Exemplarily, the semantic segmentation model can be a DeepLabv3+ model, and the backbone in the DeepLabv3+ model can be selected according to the complexity of the actual lifting scene.

[0089] In the present application, the server can identify the category label of the target hanger through a semantic segmentation model, and then determine the target area of ​​the target hanger through the number of pixel points of the target hanger. The method of determining the target area in the present application is simple, which improves the efficiency of determining the target area, thereby improving the efficiency of determining the lifting status of the target hanger.

[0090] As an optional implementation, obtaining the target two-dimensional coordinates of the target hanger in the two-dimensional image through semantic segmentation technology includes: determining a detection frame of the target hanger based on the two-dimensional image; determining the first two-dimensional coordinates of the detection frame in the two-dimensional image, and the second two-dimensional coordinates of the target hanger in the detection frame; and taking the sum of the first two-dimensional coordinates and the second two-dimensional coordinates as the target two-dimensional coordinates of the target hanger in the two-dimensional image.

[0091] In an embodiment of the present application, the server can determine the target two-dimensional coordinates of the target hanger on the two-dimensional image through the detection box. Specifically, the server determines the first two-dimensional coordinates (x_box, y_box) of the detection box in the two-dimensional image, and the second two-dimensional coordinates (x_sub, y_sub) of the target hanger in the detection box, and takes the sum of the first two-dimensional coordinates and the second two-dimensional coordinates (x_box+x_sub, y_box+y_sub) as the target two-dimensional coordinates of the target hanger in the two-dimensional image.

[0092] Exemplarily, the specific process of determining the first two-dimensional coordinates of the detection frame in the two-dimensional image is as follows: the horizontal direction of the two-dimensional image is taken as the x-axis, the vertical direction of the two-dimensional image is taken as the y-axis, a plane rectangular coordinate system is established, and the server uses the upper left corner vertex of the detection frame as the first two-dimensional coordinate of the detection frame. The server can also use the coordinates of the center or center of gravity of the target hanging part as the second two-dimensional coordinates of the target hanging part in the detection frame.

[0093] Since the detection frame cannot be closely attached to the target hanging part to be detected, there may be a position deviation between the detection frame and the target hanging part. Therefore, it is not accurate enough to directly use the coordinates of the detection frame in the two-dimensional image as the coordinates of the target hanging part in the two-dimensional image. In the present application, the first two-dimensional coordinates of the detection frame in the two-dimensional image and the second two-dimensional coordinates of the target hanging part in the detection frame can be determined, so that the sum of the first two-dimensional coordinates and the second two-dimensional coordinates is used as the target two-dimensional coordinates of the target hanging part in the two-dimensional image, which can improve the accuracy of the target two-dimensional coordinates in the target hanging part.

[0094] As an optional implementation, before determining the target three-dimensional coordinates of the target hanger in the actual space through a perspective transformation scheme, the method also includes: determining the sample height coordinates and sample depth coordinates of the sample hanger in the actual space; obtaining a sample two-dimensional image corresponding to the sample height coordinates and the sample depth coordinates; obtaining the sample area and sample two-dimensional coordinates of the sample hanger in the sample two-dimensional image through a target detection model and semantic segmentation technology; establishing a corresponding relationship between the sample height coordinates, the sample depth coordinates and the sample area, the sample two-dimensional coordinates, and saving the corresponding relationship in a database.

[0095] In an embodiment of the present application, the server selects a period when the factory is shut down for maintenance, and moves the target hanging parts at different heights and different depths of field (y and z coordinates in the actual space) to obtain the sample height coordinate y and sample depth coordinate z of the sample hanging parts in the actual space. Each time the server obtains a (y, z) of the sample hanging parts, it obtains the sample two-dimensional image corresponding to the sample height coordinate and the sample depth coordinate, and then obtains the detection frame of the sample hanging parts through the target detection model, and obtains the sample area (A) and sample two-dimensional coordinates (y_d) of the sample hanging parts in the sample two-dimensional image through the semantic segmentation technology, thereby obtaining the corresponding relationship between (y, z) and (A, y_d), and the server saves the corresponding relationship in the database.

[0096] The server can also obtain the corresponding horizontal coordinates of the sample hanger in the three-dimensional image based on the x-coordinate of the sample hanger in the two-dimensional image through a perspective transformation scheme, and also save the corresponding relationship between the x-coordinate in the two-dimensional image and the horizontal coordinate in the three-dimensional image in the database.

[0097] Therefore, the database stores the corresponding relationship between the x-coordinate, y-coordinate, and area in the two-dimensional coordinates and the horizontal axis coordinate, depth coordinate, and height coordinate in the three-dimensional coordinates.

[0098] The area of ​​the hanging piece in the three-dimensional image is related to the depth of field of the image and the height of the hanging piece. When the height of the hanging piece is constant, the deeper the depth of field, the smaller the area; when the depth of field is constant, the higher the height, the smaller the area. Therefore, the application can infer its actual height from the ground based on the depth of field and area.

[0099] As an optional implementation, before determining that the target hanging part is included in the two-dimensional image, the server first detects whether there is a person in the hanging area. If the server determines that there is no person in the hanging area, it continues to detect the next two-dimensional image. If the server detects that there is a person in the hanging area, it analyzes the hanging state of the target hanging part in the two-dimensional image.

[0100] Since the proportion of a person in a two-dimensional image is relatively small, the coordinates of the person detection frame can be used as the coordinates of the person. Specifically, the relationship between the person detection frame and the hoisting area is used to determine whether there is a person in the hoisting area.

[0101] As an optional implementation, the present application needs to preprocess the two-dimensional image before detecting whether there is a person in the two-dimensional image, including modifying the two-dimensional image size, adjusting the brightness and saturation, etc., to improve the accuracy of identifying the two-dimensional image content.

[0102] Optionally, the embodiment of the present application also provides a processing flow chart of a method for determining a hoisting state, such as Figure 3 As shown, the specific steps are as follows.

[0103] Step 3001: Acquire a two-dimensional image sent by a photographing device;

[0104] Step 3002: Use the YOLO model to determine whether there is a person in the hoisting area. If not, return to step 3001; if yes, execute step 3003;

[0105] Step 3003: Obtain a detection frame of the target hanging part through the DeepLabv3+ model;

[0106] Step 3004: obtaining the two-dimensional coordinates and target area of ​​the target hanging part according to the detection frame;

[0107] Step 3005: determining the target three-dimensional coordinates of the target hanging part according to the corresponding relationship stored in the database, the target three-dimensional coordinates including the horizontal coordinates, the depth coordinates and the height coordinates;

[0108] Step 3006: Determine whether the target hanging part is located in the hanging area according to the horizontal coordinate and the depth coordinate. If it is located in the hanging area, execute step 3007; if it is determined that it is not located in the hanging area, return to step 3001;

[0109] Step 3007: Determine whether the height coordinate is greater than the first height threshold, if so, execute step 3008; if not, execute step 3009;

[0110] Step 3008: Determine that the current state is hoisted, and issue a warning message;

[0111] Step 3009: Determine whether it is greater than the second height threshold, if so, execute step 3010; if not, execute step 3011.

[0112] Step 3010: Use a positive slope to determine the lifting state.

[0113] Step 3011: Determine whether the current area is not hoisted.

[0114] Based on the same technical concept, the embodiment of the present application also provides a device for determining the hoisting state, such as Figure 4 As shown, the device comprises:

[0115] An acquisition module 401 acquires a two-dimensional image including a hoisting scene sent by a shooting device;

[0116] The obtaining module 402 is used to obtain the target area and target two-dimensional coordinates of the target hanging part in the two-dimensional image by using a semantic segmentation technology when it is determined that the target hanging part is included in the two-dimensional image, wherein the target hanging part is a target hanging arm or a target hanging object;

[0117] A first determination module 403 is used to determine the target three-dimensional coordinates of the target hanging object in the actual space through a perspective transformation scheme, wherein the target two-dimensional coordinates and the target area are used to determine the depth coordinates and the height coordinates of the target hanging object in the target three-dimensional coordinates;

[0118] The second determining module 404 is used to determine the current hoisting state of the target hanging component according to the target three-dimensional coordinates.

[0119] Optionally, the device is also used for:

[0120] Input the two-dimensional image into the target detection model to obtain the target detection model output detection result;

[0121] In the case where the detection result is a detection frame of the target hanging part, determining that the target hanging part is included in the two-dimensional image, wherein the detection frame includes a target image of the target hanging part;

[0122] Cut the target image out of the two-dimensional image.

[0123] Optionally, the obtaining module 402 is used to:

[0124] Input the target image into the semantic segmentation model to obtain a two-dimensional matrix element output by the semantic segmentation model, wherein each two-dimensional matrix element is used to indicate a pixel point in the two-dimensional image, and each pixel point has a corresponding category label;

[0125] Determine the target pixel point of the target hanging part according to the category label;

[0126] The number of pixels of the target pixel point is taken as the target area of ​​the target hanging part.

[0127] Optionally, the obtaining module 402 is used to:

[0128] Determine a first two-dimensional coordinate of the detection frame in the two-dimensional image, and a second two-dimensional coordinate of the target hanging part in the detection frame;

[0129] The sum of the first two-dimensional coordinate and the second two-dimensional coordinate is used as the target two-dimensional coordinate of the target hanging part in the two-dimensional image.

[0130] Optionally, the device is also used for:

[0131] Determine the sample height coordinate and the sample depth coordinate of the sample hanging part in the actual space;

[0132] Obtain a sample two-dimensional image corresponding to the sample height coordinate and the sample depth coordinate;

[0133] Through the target detection model and semantic segmentation technology, the sample area and sample two-dimensional coordinates of the sample hanging part in the sample two-dimensional image are obtained;

[0134] The corresponding relationship among the sample height coordinate, the sample depth coordinate, the sample area and the sample two-dimensional coordinate is established, and the corresponding relationship is saved in the database.

[0135] Optionally, the second determining module 404 is used to:

[0136] When it is determined that the target hanging part is located in the hanging area according to the lateral coordinates and the depth coordinates in the target three-dimensional coordinates, the height coordinates of the target hanging part are obtained;

[0137] When it is determined that the height coordinate exceeds the first height threshold, it is determined that the target hanging component is currently in a hoisted state.

[0138] Optionally, the second determining module 404 is used to:

[0139] When it is determined that the height coordinate is greater than the second height threshold and less than the first height threshold, a target positive slope of the target hanging object is obtained, wherein the target hanging object is a strip-shaped object arranged in a horizontal direction, and the target positive slope is used to indicate the degree of inclination of the target hanging object;

[0140] When the target positive slope meets the preset slope range, it is determined that the target hanging component is currently in the hoisted state.

[0141] According to another aspect of the embodiments of the present application, the present application provides an electronic device, such as Figure 5 As shown, it includes a memory 503, a processor 501, a communication interface 502 and a communication bus 504. The memory 503 stores a computer program that can be run on the processor 501. The memory 503 and the processor 501 communicate through the communication interface 502 and the communication bus 504. When the processor 501 executes the computer program, the steps of the above method are implemented.

[0142] The memory and processor in the above electronic device communicate via a communication bus and a communication interface. The communication bus may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc.

[0143] The memory may include a random access memory (RAM) or a non-volatile memory, such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0144] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0145] According to another aspect of the embodiments of the present application, a computer-readable medium having a non-volatile program code executable by a processor is provided.

[0146] Optionally, in an embodiment of the present application, a computer-readable medium is configured to store program code for the processor to execute the above method.

[0147] Optionally, the specific examples in this embodiment may refer to the examples described in the above embodiments, and this embodiment will not be described in detail here.

[0148] When the embodiments of the present application are specifically implemented, reference may be made to the above-mentioned embodiments, which have corresponding technical effects.

[0149] It is understood that the embodiments described herein may be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit may be implemented in one or more application specific integrated circuits (ASIC), digital signal processors (DSP), digital signal processing devices (DSPD), programmable logic devices (PLD), field programmable gate arrays (FPGA), general purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described in the present application, or a combination thereof.

[0150] For software implementation, the technology described herein can be implemented by a unit that performs the functions described herein. The software code can be stored in a memory and executed by a processor. The memory can be implemented in the processor or outside the processor.

[0151] Those of ordinary skill in the art will appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professional and technical personnel can use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0152] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0153] In the embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the modules is only a logical function division. There may be other division methods in actual implementation, such as multiple modules or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0154] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0155] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.

[0156] If the function is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk or an optical disk. It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply that there is any such actual relationship or order between these entities or operations. Moreover, the term "include", "include" or any other variant thereof is intended to cover non-exclusive inclusion, so that the process, method, article or device including a series of elements includes not only those elements, but also includes other elements that are not explicitly listed, or also includes elements inherent to such a process, method, article or device. Without more constraints, an element defined by the phrase "comprising a..." does not exclude the existence of other identical elements in the process, method, article or apparatus comprising the element.

[0157] The above description is only a specific implementation of the present application, so that those skilled in the art can understand or implement the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein can be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest range consistent with the principles and novel features applied for herein.

Claims

1. A method for determining a hoisting state, characterized in that: The method comprises: Acquire a two-dimensional image including a hoisting scene sent by a shooting device; When it is determined that the two-dimensional image includes a target hanging part, obtaining a target area and a target two-dimensional coordinate of the target hanging part in the two-dimensional image by using a semantic segmentation technology, wherein the target hanging part is a target hanging arm or a target hanging object; Determine the target three-dimensional coordinates of the target hanging object in the actual space by a perspective transformation scheme, wherein the target two-dimensional coordinates and the target area are used to determine the depth coordinates and the height coordinates of the target hanging object in the target three-dimensional coordinates; Determine the current hoisting state of the target hanging component according to the target three-dimensional coordinates; Before obtaining the target area and target two-dimensional coordinates of the target hanging part in the two-dimensional image by using the semantic segmentation technology, the method further includes: Inputting the two-dimensional image into a target detection model to obtain a detection result output by the target detection model; In a case where the detection result is a detection frame of the target hanging part, determining that the target hanging part is included in the two-dimensional image, wherein the detection frame includes a target image of the target hanging part; The target image is segmented from the two-dimensional image.

2. The method according to claim 1, characterized in that: The obtaining of the target area of ​​the target hanging part in the two-dimensional image by using the semantic segmentation technology includes: Inputting the target image into a semantic segmentation model to obtain two-dimensional matrix elements output by the semantic segmentation model, wherein each of the two-dimensional matrix elements is used to indicate a pixel point in the two-dimensional image, and each of the pixel points has a corresponding category label; Determine a target pixel point of the target hanging part according to the category label; The number of pixels of the target pixel point is used as the target area of ​​the target hanging part.

3. The method according to claim 1, characterized in that The obtaining of the target two-dimensional coordinates of the target hanging part in the two-dimensional image by using the semantic segmentation technology includes: Determine a first two-dimensional coordinate of the detection frame in the two-dimensional image, and a second two-dimensional coordinate of the target hanging part in the detection frame; The sum of the first two-dimensional coordinate and the second two-dimensional coordinate is used as the target two-dimensional coordinate of the target hanging part in the two-dimensional image.

4. The method according to claim 1, characterized in that: Before determining the target three-dimensional coordinates of the target hanging part in the actual space by the perspective transformation scheme, the method further includes: Determine the sample height coordinate and the sample depth coordinate of the sample hanging part in the actual space; Acquire a sample two-dimensional image corresponding to the sample height coordinate and the sample depth coordinate by a photographing device; Obtaining the sample area and sample two-dimensional coordinates of the sample hanging part in the sample two-dimensional image by using the target detection model and the semantic segmentation technology; A corresponding relationship between the sample height coordinate, the sample depth of field coordinate, the sample area, and the sample two-dimensional coordinate is established, and the corresponding relationship is stored in a database.

5. The method according to claim 1, characterized in that Determining the current hoisting state of the target hanging component according to the target three-dimensional coordinates includes: When it is determined that the target hanging component is located in the hanging area according to the lateral coordinates and the depth coordinates in the target three-dimensional coordinates, obtaining the height coordinates of the target hanging component; When it is determined that the height coordinate exceeds the first height threshold, it is determined that the target hanging component is currently in a hoisted state.

6. The method according to claim 5, characterized in that After obtaining the height coordinate of the target hanging part, the method further includes: When it is determined that the height coordinate is greater than a second height threshold and less than the first height threshold, acquiring a target positive slope of the target hanging object, wherein the target hanging object is a strip-shaped object arranged in a horizontal direction, and the target positive slope is used to indicate a degree of inclination of the target hanging object; When the target positive slope satisfies a preset slope range, it is determined that the target hanging component is currently in a hoisted state.

7. A device for determining a hoisting state, characterized in that: The device comprises: An acquisition module, which acquires a two-dimensional image including a hoisting scene sent by a shooting device; an obtaining module, for obtaining a target area and a target two-dimensional coordinate of the target hanging part in the two-dimensional image by using a semantic segmentation technology when it is determined that the target hanging part is included in the two-dimensional image, wherein the target hanging part is a target hanging arm or a target hanging object; A first determination module is used to determine the target three-dimensional coordinates of the target hanging object in the actual space through a perspective transformation scheme, wherein the target two-dimensional coordinates and the target area are used to determine the depth coordinates and height coordinates of the target hanging object in the target three-dimensional coordinates; A second determination module is used to determine the current hoisting state of the target hanging component according to the target three-dimensional coordinates; Wherein, the device is also used for: Inputting the two-dimensional image into a target detection model to obtain a detection result output by the target detection model; In a case where the detection result is a detection frame of the target hanging part, determining that the target hanging part is included in the two-dimensional image, wherein the detection frame includes a target image of the target hanging part; The target image is segmented from the two-dimensional image.

8. An electronic device, characterized in that: It includes a processor, a communication interface, a memory and a communication bus, wherein the processor, the communication interface and the memory communicate with each other through the communication bus; Memory, used to store computer programs; A processor, for implementing the method steps described in any one of claims 1 to 6 when executing a program stored in a memory.

9. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method steps described in any one of claims 1 to 6 are implemented.

Images