Method, device, electronic equipment and readable storage medium for judging violation of transport vehicle

Abstract

The application discloses a forklift truck violation judgment method and device, an electronic device and a readable storage medium. The method comprises the following steps: acquiring a target image; extracting position information and target vehicle attribute information of a cargo forklift truck in the target image; acquiring violation attribute information corresponding to the position information; and determining a violation judgment result of the cargo forklift truck according to the violation attribute information and the target vehicle attribute information. Therefore, the forklift truck violation judgment method provided by the application can reduce manual detection, reduce subjective judgment, improve violation judgment accuracy, and can judge different violation conditions corresponding to different positions in a site, can identify more comprehensive violation types, is not limited by the type of the cargo forklift truck, and can be applied in various scenes.

Description

Technical Field

[0001] This application relates to the field of logistics technology, specifically to a method, device, electronic device, and readable storage medium for judging violations of handling vehicles. Background Technology

[0002] In the logistics industry, forklifts are indispensable transportation tools. In actual production processes, to ensure the standardized operation of forklifts, it is necessary to supervise their operation.

[0003] Currently, forklift violation monitoring is typically conducted manually or through image detection. However, manual monitoring significantly increases labor costs and relies on the subjective judgment of staff, resulting in low accuracy. Image detection methods, on the other hand, can only detect simple violations and lack comprehensive functionality. Therefore, a forklift violation detection method is needed that is both cost-effective and capable of detecting various violations. Summary of the Invention

[0004] This application provides a method, device, electronic device, and readable storage medium for judging violations of handling vehicles, aiming to solve the technical problem that current methods cannot guarantee both low cost and detection of various violations.

[0005] Firstly, this application provides a method for determining violations by a handling vehicle, including:

[0006] Acquire the target image;

[0007] Extract the location information and target vehicle attribute information of the cargo handling vehicle in the target image;

[0008] Obtain the violation attribute information corresponding to the location information;

[0009] Based on the violation attribute information and the target vehicle attribute information, the violation judgment result of the cargo handling vehicle is determined.

[0010] In one possible implementation of this application, obtaining the violation attribute information corresponding to the location information includes:

[0011] Based on the corner positions of the first type of corner points in the location information, the vehicle body area of ​​the cargo handling vehicle in the target image is determined;

[0012] Based on the corner point position of the second type of corner point in the first type of corner point, determine the extended area of ​​the auxiliary components on the cargo handling vehicle in the target image;

[0013] Based on the vehicle body area and the extended area, determine the overall vehicle area of ​​the cargo handling vehicle in the target image;

[0014] If the overall area of ​​the vehicle at least partially overlaps with the preset target warning area in the target image, then the violation attribute information corresponding to the target warning area is used as the violation attribute information corresponding to the location information.

[0015] In one possible implementation of this application, determining the extended area of ​​the auxiliary component on the cargo handling vehicle in the target image based on the corner position of the second type of corner point among the first type of corner points includes:

[0016] Extract the corner point positions of the second type of corner points from the first type of corner points;

[0017] Based on the corner point positions of the second type of corner points, determine the extension direction of the auxiliary components on the cargo handling vehicle in the target image;

[0018] The extended area is determined based on the area of ​​the vehicle body region using a preset area conversion strategy.

[0019] The extension area of ​​the auxiliary component in the target image is determined based on the extension area and the extension direction.

[0020] In one possible implementation of this application, obtaining the violation attribute information corresponding to the location information includes:

[0021] Based on the corner positions of each type of corner point in the location information, predict the movement direction of the cargo handling vehicle in the target image;

[0022] Based on the direction of movement and the corner positions of each type of corner point, determine the target position of the cargo handling vehicle in the target image;

[0023] The location of the moving target is matched with a preset target warning area in the target image to determine whether the target warning area is the moving target area of ​​the cargo handling vehicle.

[0024] If the target warning area is the movement target area of ​​the cargo handling vehicle, then the violation attribute information corresponding to the target warning area is used as the violation attribute information corresponding to the location information.

[0025] In one possible implementation of this application, before obtaining the violation attribute information corresponding to the location information, the method further includes:

[0026] Extract the vehicle identity information and cargo information from the target vehicle attribute information;

[0027] If the cargo information and the vehicle identity information match the preset compliant cargo information, then the step of obtaining the violation attribute information corresponding to the location information is executed.

[0028] In one possible implementation of this application, the step of extracting the location information and target vehicle attribute information of the cargo handling vehicle in the target image includes:

[0029] The target image is input into a preset handling vehicle detection model to obtain the location information of the cargo handling vehicle and the target vehicle attribute information;

[0030] The preset transport vehicle detection model is trained through the following steps:

[0031] Acquire training data, wherein the training data includes training images and the actual positions of corner points of various types in the training images;

[0032] Extract the relative distribution features of each type of corner point in the training image;

[0033] Based on the relative distribution characteristics, the corner position features of each type of corner point in the training image are determined;

[0034] Based on the corner point location characteristics, the predicted locations of each type of corner point are obtained;

[0035] Based on the predicted and actual positions of the corner points of each type, the parameters in the initial transport vehicle detection model are adjusted to obtain the preset transport vehicle detection model.

[0036] In one possible implementation of this application, before determining the violation judgment result of the cargo handling vehicle based on the violation attribute information and the target vehicle attribute information, the method further includes:

[0037] Facial recognition is performed on the target image to determine the operator information of the cargo handling vehicle;

[0038] Obtain the target personnel information corresponding to the target vehicle attribute information of the cargo handling vehicle;

[0039] If the operator information matches the target personnel information, then the step of determining the violation judgment result of the cargo handling vehicle based on the violation attribute information and the target vehicle attribute information is executed.

[0040] Secondly, this application provides a device for judging violations of handling vehicles, comprising:

[0041] The first acquisition unit is used to acquire the target image;

[0042] The extraction unit is used to extract the location information and target vehicle attribute information of the cargo handling vehicle in the target image;

[0043] The second acquisition unit is used to acquire the violation attribute information corresponding to the location information;

[0044] The determining unit is used to determine the violation judgment result of the cargo handling vehicle based on the violation attribute information and the target vehicle attribute information.

[0045] In one possible implementation of this application, the second acquisition unit is further configured to:

[0046] Based on the corner positions of the first type of corner points in the location information, the vehicle body area of ​​the cargo handling vehicle in the target image is determined;

[0047] Based on the corner point position of the second type of corner point in the first type of corner point, determine the extended area of ​​the auxiliary components on the cargo handling vehicle in the target image;

[0048] Based on the vehicle body area and the extended area, determine the overall vehicle area of ​​the cargo handling vehicle in the target image;

[0049] If the overall area of ​​the vehicle at least partially overlaps with the preset target warning area in the target image, then the violation attribute information corresponding to the target warning area is used as the violation attribute information corresponding to the location information.

[0050] In one possible implementation of this application, the second acquisition unit is further configured to:

[0051] Extract the corner point positions of the second type of corner points from the first type of corner points;

[0052] Based on the corner point positions of the second type of corner points, determine the extension direction of the auxiliary components on the cargo handling vehicle in the target image;

[0053] The extended area is determined based on the area of ​​the vehicle body region using a preset area conversion strategy.

[0054] The extension area of ​​the auxiliary component in the target image is determined based on the extension area and the extension direction.

[0055] In one possible implementation of this application, the second acquisition unit is further configured to:

[0056] Based on the corner positions of each type of corner point in the location information, predict the movement direction of the cargo handling vehicle in the target image;

[0057] Based on the direction of movement and the corner positions of each type of corner point, determine the target position of the cargo handling vehicle in the target image;

[0058] The location of the moving target is matched with a preset target warning area in the target image to determine whether the target warning area is the moving target area of ​​the cargo handling vehicle.

[0059] If the target warning area is the movement target area of ​​the cargo handling vehicle, then the violation attribute information corresponding to the target warning area is used as the violation attribute information corresponding to the location information.

[0060] In one possible implementation of this application, the handling vehicle violation detection device further includes a cargo-loading detection unit, which is used for:

[0061] Extract the vehicle identity information and cargo information from the target vehicle attribute information;

[0062] If the cargo information and the vehicle identity information match the preset compliant cargo information, then the step of obtaining the violation attribute information corresponding to the location information is executed.

[0063] In one possible implementation of this application, the extraction unit is further configured to:

[0064] The target image is input into a preset handling vehicle detection model to obtain the location information of the cargo handling vehicle and the target vehicle attribute information.

[0065] In one possible implementation of this application, the extraction unit is further configured to:

[0066] Acquire training data, wherein the training data includes training images and the actual positions of corner points of various types in the training images;

[0067] Extract the relative distribution features of each type of corner point in the training image;

[0068] Based on the relative distribution characteristics, the corner position features of each type of corner point in the training image are determined;

[0069] Based on the corner point location characteristics, the predicted locations of each type of corner point are obtained;

[0070] Based on the predicted and actual positions of the corner points of each type, the parameters in the initial transport vehicle detection model are adjusted to obtain the preset transport vehicle detection model.

[0071] In one possible implementation of this application, the handling vehicle violation judgment device further includes a face recognition unit, which is used for:

[0072] Facial recognition is performed on the target image to determine the operator information of the cargo handling vehicle;

[0073] Obtain the target personnel information corresponding to the target vehicle attribute information of the cargo handling vehicle;

[0074] If the operator information matches the target personnel information, then the step of determining the violation judgment result of the cargo handling vehicle based on the violation attribute information and the target vehicle attribute information is executed.

[0075] Thirdly, this application also provides an electronic device, which includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor calls the computer program in the memory, it executes the steps in any of the methods for judging violations of transport vehicles provided in this application.

[0076] Fourthly, this application also provides a readable storage medium storing a computer program, which, when executed by a processor, implements the steps in any of the methods for judging violations of transport vehicles provided in this application.

[0077] In summary, the method for determining violations by transport vehicles provided in this application includes: acquiring a target image; extracting the location information and target vehicle attribute information of the transport vehicle in the target image; acquiring violation attribute information corresponding to the location information; and determining the violation determination result of the transport vehicle based on the violation attribute information and the target vehicle attribute information. Therefore, the method for determining violations by transport vehicles provided in this application eliminates the need for manual inspection, reducing labor costs and subjective judgment, and improving the accuracy of violation determination. Furthermore, it can determine different violations corresponding to different locations in the site, enabling the identification of more comprehensive violation types, and is not limited by the type of transport vehicle, thus allowing application in various scenarios. Attached Figure Description

[0078] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0079] Figure 1 This is a schematic diagram of the information acquisition model provided in the embodiments of this application;

[0080] Figure 2 This is a schematic diagram illustrating an application scenario of the handling vehicle violation judgment method provided in this application embodiment;

[0081] Figure 3 This is a flowchart illustrating a method for determining violations of handling vehicles provided in this application embodiment;

[0082] Figure 4 This is a flowchart illustrating a method for obtaining violation attribute information provided in an embodiment of this application;

[0083] Figure 5 This is a schematic image of a cargo handling vehicle provided in the embodiments of this application;

[0084] Figure 6 This is another flowchart illustrating the process of obtaining violation attribute information provided in the embodiments of this application;

[0085] Figure 7 This is a flowchart illustrating a process for determining whether a cargo handling vehicle is illegally carrying cargo, as provided in an embodiment of this application.

[0086] Figure 8 This is a schematic diagram of a model training process provided in the embodiments of this application;

[0087] Figure 9 This is a schematic diagram of a module structure provided in an embodiment of this application;

[0088] Figure 10 This is a schematic diagram of an embodiment of the handling vehicle violation judgment device provided in this application.

[0089] Figure 11 This is a schematic diagram of an embodiment of the electronic device provided in this application. Detailed Implementation

[0090] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0091] In the description of the embodiments of this application, it should be understood that the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Therefore, features defined with "first" and "second" may explicitly or implicitly include one or more of the stated features. In the description of the embodiments of this application, "multiple" means two or more, unless otherwise explicitly specified.

[0092] To enable any person skilled in the art to implement and use this application, the following description is provided. In this description, details are set forth for purposes of explanation. It should be understood that those skilled in the art will recognize that this application can be implemented without using these specific details. In other instances, well-known processes will not be described in detail to avoid obscuring the description of the embodiments of this application with unnecessary detail. Therefore, this application is not intended to be limited to the embodiments shown, but is consistent with the broadest scope of the principles and features disclosed in the embodiments of this application.

[0093] This application provides a method, apparatus, electronic device, and readable storage medium for determining violations by transport vehicles. The transport vehicle violation determination apparatus can be integrated into an electronic device, which can be a server or a terminal, etc.

[0094] The execution subject of the handling vehicle violation judgment method in this application embodiment can be the handling vehicle violation judgment device provided in this application embodiment, or different types of electronic devices such as server equipment, physical host, or user equipment (UE) that integrate the handling vehicle violation judgment device. The handling vehicle violation judgment device can be implemented in hardware or software. The UE can be a terminal device such as a smartphone, tablet computer, laptop computer, handheld computer, desktop computer, or personal digital assistant (PDA).

[0095] The electronic device can operate independently or in a cluster.

[0096] First, we introduce an information acquisition model that can be used to obtain information about freight handling vehicles, referencing... Figure 1 , Figure 1 The information acquisition model 100 in the middle includes:

[0097] The preprocessing layer 101 consists of a convolutional neural network 1011 (CNN) and a residual network 1012 (ResNet). The convolutional neural network 1011 is used to perform convolution processing on the input image to achieve functions such as pooling and noise reduction, while the residual network 1012 is used to improve the computation speed of the convolutional neural network.

[0098] The first feature extraction layer 102 can be composed of an hourglass network. The hourglass network can extract features at different scales through multiple upsampling and downsampling, and fuse the features at each scale to obtain a heatmap, which is then input into the second feature extraction layer 103.

[0099] The second feature extraction layer 103 can be composed of an hourglass network, which extracts features from the input heatmap and outputs the extracted features.

[0100] The location prediction layer 104 can be composed of fully connected layers (FC). It makes predictions based on the output of the second feature extraction layer 103 to obtain the location information of the cargo handling vehicle. The location prediction layer 104 can perform multi-class prediction.

[0101] The attribute prediction layer 105 can be composed of fully connected layers. It makes predictions based on the output of the second feature extraction layer 103 to obtain the target vehicle attribute information of the cargo handling vehicle. The attribute prediction layer 105 can perform multi-class prediction.

[0102] Both the location prediction layer 104 and the attribute prediction layer 105 can be prediction layers capable of multi-class classification prediction.

[0103] As can be seen, the information acquisition model 100 described above can simultaneously acquire the location information of the cargo handling vehicle and the target vehicle attribute information through a single model, reducing the number of models that need to be called when judging violations. Since the prediction of location information and target vehicle attribute information is based on the same feature (output of the second feature extraction layer 103), the amount of computation is greatly reduced, and the prediction difference caused by different extracted features when using multiple models is avoided.

[0104] See Figure 2 , Figure 2 This is a schematic diagram of a scenario for the handling vehicle violation judgment system provided in this application embodiment. The handling vehicle violation judgment system may include an electronic device 200, which integrates a handling vehicle violation judgment device.

[0105] In addition, such as Figure 2 As shown, the transport vehicle violation judgment system may also include a memory 201 for storing data, such as image data.

[0106] It should be noted that, Figure 2The schematic diagram of the handling vehicle violation judgment system shown is merely an example. The handling vehicle violation judgment system and scenario described in this application embodiment are for the purpose of more clearly illustrating the technical solutions of this application embodiment and do not constitute a limitation on the technical solutions provided in this application embodiment. As those skilled in the art will know, with the evolution of the handling vehicle violation judgment system and the emergence of new business scenarios, the technical solutions provided in the embodiments of this invention are also applicable to similar technical problems.

[0107] The following describes the method for judging violations of handling vehicles provided in this application. In this application, an electronic device is used as the execution subject. For simplicity and ease of description, the execution subject will be omitted in the subsequent method embodiments. The violation judgment of handling vehicles includes: acquiring a target image; extracting the location information and target vehicle attribute information of the goods handling vehicle in the target image; acquiring the violation attribute information corresponding to the location information; and determining the violation judgment result of the goods handling vehicle based on the violation attribute information and the target vehicle attribute information.

[0108] Reference Figure 3 , Figure 3 This is a flowchart illustrating a method for determining violations by a transport vehicle according to an embodiment of this application. It should be noted that although the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order. Furthermore, for ease of explanation, unless otherwise specified, the target image is assumed to contain only one transport vehicle; however, the number of transport vehicles in the target image should not be used as a limitation on this application. Specifically, the method for determining violations by a transport vehicle may include the following steps 301-304, wherein:

[0109] 301. Obtain the target image.

[0110] The target image is an image containing the cargo handling vehicle to be judged for violation. Cargo handling vehicles refer to various wheeled handling vehicles used for loading, unloading, stacking and short-distance transportation of goods. For example, forklifts used in the express logistics industry to handle packages are a type of cargo handling vehicle.

[0111] This application does not limit the method for acquiring the target image; the target image can be acquired using image acquisition devices such as cameras or video cameras. For example, when determining whether a forklift in a warehouse is in violation of regulations, a camera pre-installed in the warehouse can be used to take pictures of the forklift to obtain the target image.

[0112] In some embodiments, the image acquired by the image acquisition device can be preprocessed to obtain a target image with clear content. For example, the image acquired by the image acquisition device can be preprocessed with contrast enhancement, noise reduction, etc., to obtain the target image.

[0113] 302. Extract the location information and target vehicle attribute information of the cargo handling vehicle in the target image.

[0114] The location information includes the position of the cargo handling vehicle in the target image. For example, an image coordinate system can be established within the target image, and the coordinates of the cargo handling vehicle in this system can be used as its location information. This embodiment does not restrict the coordinate system parameters such as the origin and axis directions. Any point in the target image can be used as the origin, and any direction in the target image can be used as the positive direction of one coordinate axis. The positive directions of other coordinate axes are then determined based on this positive direction. For ease of understanding, the following example illustrates how to establish an image coordinate system in the target image: Assuming the coordinate axes in the image coordinate system are the X-axis and Y-axis, the midpoint of the target image can be used as the origin, any direction in the target image can be used as the positive direction of the X-axis, and the positive direction of the Y-axis can be determined based on the determined positive direction of the X-axis to establish the image coordinate system. After establishing the image coordinate system, the coordinates of the cargo handling vehicle within this system can be obtained to acquire its location information.

[0115] In some embodiments, the coordinates of the midpoint of the region corresponding to the cargo handling vehicle in the target image within the image coordinate system can be used as the location information of the cargo handling vehicle. For example, if the region corresponding to the cargo handling vehicle in the target image is a square region, and the coordinate range of the square region in the image coordinate system is x∈[4,8], y∈[4,8], then the coordinates (6,6) can be used as the location information of the cargo handling vehicle.

[0116] The target vehicle attribute information refers to the vehicle attribute information of the cargo handling vehicle in the target image. The vehicle attribute information may include one or more types of information. For example, the vehicle attribute information may include at least one of the cargo loading information and vehicle identity information of the cargo handling vehicle. The cargo loading information refers to the cargo type information of the cargo currently being handled by the cargo handling vehicle, as well as the information on whether the cargo handling vehicle is currently carrying cargo. The vehicle identity information refers to the vehicle type information of the cargo handling vehicle.

[0117] There are multiple ways to classify the aforementioned cargo type and vehicle type information. For example, cargo type information can be categorized by size, such as large cargo, medium cargo, small cargo, etc. Alternatively, cargo can be categorized by purpose, such as furniture, food, electronic products, etc. Vehicle type information can be categorized by carrying capacity, such as large handling vehicles, medium handling vehicles, small handling vehicles (or, when the handling vehicle is a forklift, large forklift, medium forklift, small forklift). Or, it can be categorized by the type of cargo being handled, such as for handling furniture, for handling food, for handling electronic products, etc.

[0118] In some embodiments, it can be achieved through Figure 1 The information acquisition model 100 extracts the location information and target vehicle attribute information of the cargo handling vehicle in the target image. Specifically, the target image can be input into the information acquisition model 100, processed by the preprocessing layer 101, and then the image features in the target image are extracted by the first feature extraction layer 102 and the second feature extraction layer 103. The location information and target vehicle attribute information of the cargo handling vehicle in the target image are then predicted by the location prediction layer 104 and the attribute prediction layer 105, respectively.

[0119] 303. Obtain the violation attribute information corresponding to the location information.

[0120] Violation attribute information is the basis for determining whether a cargo handling vehicle has violated regulations. It includes the vehicle attribute information that is deemed to be in violation; therefore, it can also be understood as a type of vehicle attribute information. When the target vehicle attribute information matches the violation attribute information corresponding to the location information, it indicates that the cargo handling vehicle in the target image has violated regulations. Violations can include illegal parking, illegal driving, etc. For example, the violation attribute information corresponding to the location information can be the violation attribute information corresponding to a vehicle restricted area that matches the location information. A vehicle restricted area refers to an area where there may be danger when a cargo handling vehicle enters. Matching the location information with a vehicle restricted area means that the area corresponding to the vehicle restricted area in the target image includes the image location corresponding to the location information. It is understandable that since different vehicle restricted areas can be associated with different violation attribute information, the corresponding violation attribute information can also be different when the location information matches different vehicle restricted areas.

[0121] To facilitate understanding, the following specific examples illustrate this: Assume the goods handling vehicle in the target image is a forklift. When the forklift's location information in the target image matches a "large forklift prohibited zone," meaning the forklift entered the "large forklift prohibited zone," the violation attribute information corresponding to the location information is the same as the violation attribute information corresponding to the "large forklift prohibited zone." Understandably, this violation attribute information includes the type information of a "large forklift." If the forklift entering the "large forklift prohibited zone" is indeed a large forklift, then the target vehicle attribute information matches the violation attribute information, and the forklift is determined to be in violation. Alternatively, when the forklift's location information in the target image matches a "cargo vehicle prohibited zone," meaning the forklift entered the "cargo vehicle prohibited zone," the violation attribute information corresponding to the location information is the same as the violation attribute information corresponding to the "cargo vehicle prohibited zone." Understandably, this violation attribute information includes the loading information that the forklift is currently loading goods. If the forklift entering the "cargo vehicle prohibited zone" is currently loading goods, then the target vehicle attribute information matches the violation attribute information, and the forklift is determined to be in violation.

[0122] In some embodiments, each vehicle restricted area can be pre-assigned to a violation attribute information based on the type of goods piled up in each restricted area. For example, the violation attribute information corresponding to each vehicle restricted area can be manually set according to the type of goods piled up. When the goods piled up in the vehicle restricted area are flammable and explosive, "large truck" can be set as the violation attribute information for that vehicle restricted area to prevent the goods piled up from exploding or burning when the large truck enters the vehicle restricted area and overturns.

[0123] It should be noted that the violation attribute information can contain one or more types of information. For example, the violation attribute information can include either cargo information or vehicle identification information, or both. For instance, when the restricted area is "Large Forklift Restricted Area" or "Cargo Vehicle Restricted Area," the corresponding violation attribute information is vehicle identification information and cargo information, respectively. As another example, when the target vehicle restricted area is "Large Forklift Restricted Area," the corresponding violation attribute information includes both vehicle identification information and cargo information.

[0124] 304. Based on the violation attribute information and the target vehicle attribute information, determine the violation judgment result of the cargo handling vehicle.

[0125] For example, violation attribute information can be matched with target vehicle attribute information to determine the violation judgment result of the goods handling vehicle. If the violation attribute information is the same as the target vehicle attribute information, the goods handling vehicle can be judged to be in violation. For ease of understanding, the following specific examples are given: Suppose the goods handling vehicle in the target image is a forklift. When the violation attribute information corresponding to the forklift's position information in the target image is the violation attribute information corresponding to "large forklift prohibited area," that is, the violation attribute information includes the type information of "large forklift," if the forklift in the target image is a large forklift, then the forklift can be judged to be in violation. As another example, if the violation attribute information corresponding to the forklift's position information in the target image is the violation attribute information corresponding to "cargo vehicle prohibited area," that is, the violation attribute information includes the cargo information "forklift is loading cargo," if the forklift is loading cargo, then the forklift can be judged to be in violation.

[0126] If a violation is determined to be by a cargo handling vehicle, an alert can be issued. For example, at least one type of alert can be sent via target terminals such as smartphones, tablets, video matrices, monitoring platforms, or vehicle-mounted devices, including audio alerts, text alerts, and flashing alerts, to notify staff of the violation.

[0127] In summary, the method for determining violations by transport vehicles provided in this application includes: acquiring a target image; extracting the location information and target vehicle attribute information of the transport vehicle in the target image; acquiring violation attribute information corresponding to the location information; and determining the violation determination result of the transport vehicle based on the violation attribute information and the target vehicle attribute information. Therefore, the method for determining violations by transport vehicles provided in this application eliminates the need for manual inspection, reducing labor costs and subjective judgment, and improving the accuracy of violation determination. Furthermore, it can determine different violations corresponding to different locations in the site, enabling the identification of more comprehensive violation types, and is not limited by the type of transport vehicle, thus being applicable to various scenarios.

[0128] In some embodiments, to improve the accuracy of location information extraction, the position of the vehicle body of the freight transport vehicle in the target image can be extracted to obtain location information. The reason is as follows: In addition to the vehicle body, a freight transport vehicle also includes a series of functional components. For example, a forklift, besides its main body which includes the driving wheels, control chip, engine, and tires, also includes functional components such as forks. These functional components are difficult to accurately identify from the image due to their indistinct feature points and similar colors to the background. Therefore, extracting the position of the vehicle body yields a more accurate result. In this case, to obtain violation attribute information, the vehicle body in the target image can be detected, and then extended according to a preset ratio to obtain the functional components in the target image, thereby determining the entire freight transport vehicle in the target image. The following provides a method for obtaining violation attribute information in this situation, see reference. Figure 4 At this point, obtaining the violation attribute information corresponding to the location information includes:

[0129] 401. Based on the corner positions of the first type of corner points in the location information, determine the vehicle body area of ​​the cargo handling vehicle in the target image.

[0130] The first type of corner point refers to the corner points of the vehicle body in the target image. For ease of understanding, let's take... Figure 5 For example, to illustrate the first type of corner point, refer to... Figure 5 The forklift 500 consists of a vehicle body 501 and forks 502, therefore in Figure 5 In the diagram, the first type of corner point refers to corner point 5011, corner point 5012, corner point 5013 and corner point 5014, and the vehicle body area refers to the area enclosed by connecting corner points 5011, corner point 5012, corner point 5013 and corner point 5014 in sequence.

[0131] In some embodiments, an image coordinate system can be established first in the target image, and then the coordinates of the first type of corner points in the image coordinate system can be used as the corner point positions of the first type of corner points.

[0132] The embodiments of this application can be accessed through... Figure 1 The preset information acquisition model 100 obtains the location information. At this time, the corner points of the vehicle body in the training image can be labeled during training, and the initial information acquisition model 100 can be trained with the labeled training image to obtain the preset information acquisition model 100. The corner point position of each first type of corner point can be extracted through the position prediction layer 104.

[0133] 402. Based on the corner point position of the second type of corner point in the first type of corner point, determine the extension area of ​​the auxiliary components on the cargo handling vehicle in the target image.

[0134] In some embodiments, the second type of corner point may refer to the corner point among the first type of corner points that is closest to a functional component of the cargo handling vehicle in the target image. (Continue to refer to...) Figure 5 To explain, in Figure 5 Among corner points 5011, 5012, 5013, and 5014, corner points 5011 and 5012 are closest to the forklift 502. Therefore, in Figure 5 In the middle, the second type of corner point refers to corner point 5011 and corner point 5012.

[0135] If passed Figure 1 The preset information acquisition model 100 obtains the location information of the cargo handling vehicle in the target image. During the initial training of the information acquisition model 100, different labels are assigned to various types of corner points within the first type of corner points in the training image. Then, when processing the target image, the preset information acquisition model 100 can determine the corner point positions of the second type of corner points in the target image. The training image is... Figure 5 For example, during the training phase, corner points 5011 and 5012 can be labeled as second-class corner points, and corner points 5013 and 5014 can be labeled as third-class corner points. Third-class corner points refer to all corner points in the first-class corner points except for the second-class corner points. Then, the initial information acquisition model 100 is trained using the labeled training images to obtain the preset information acquisition model 100. The position prediction layer 104 in the model can realize the function of multi-class classification. Through the position prediction layer 104, the corner point positions of each type of corner point in the target image can be extracted, that is, the corner point positions of each second-class corner point and the corner point positions of each third-class corner point.

[0136] In other embodiments, each corner point included in the first type of corner points in the training image may be labeled with a different label during the training phase. The training image is... Figure 5 For example, if corner points 5011, 5012, 5013, and 5014 are first-class corner points, then during the training phase, corner points 5011, 5012, 5013, and 5014 can be labeled as first-class, second-class, third-class, and fourth-class corner points, respectively. During step 402, the first-class and second-class corner points are designated as second-class corner points. In this embodiment, when adding new functional components to a cargo handling vehicle, if it is necessary to determine whether the cargo handling vehicle violates regulations after adding the functional components, only the definition logic of the second-class corner points needs to be changed, which is very flexible. For example, in... Figure 5 When adding another fork 503 to the forklift, the first sub-class corner point, the second sub-class corner point, and the third sub-class corner point can be used as the second class corner point.

[0137] The definitions of auxiliary components and functional components are the same, and will not be repeated below.

[0138] The extended region refers to the area corresponding to the functional components of the cargo handling vehicle in the target image, in order to... Figure 5 For example, if Figure 5 If the only functional component in the design is the fork 502, then the extended area refers to the area corresponding to the fork 502. If... Figure 5 If the functional components in the model include both fork 502 and fork 503, then the extended area refers to the area corresponding to fork 502.

[0139] In some embodiments, the extension area can be determined based on the area of ​​the vehicle body region in the target image and the direction in which the functional components extend outward from the vehicle body. The step "determining the extension area of ​​the auxiliary components on the cargo handling vehicle in the target image based on the corner point positions of the second type of corner points among the first type of corner points" can be implemented through the following steps:

[0140] (1) Extract the corner point position of the second type of corner point from the first type of corner point.

[0141] (2) Determine the extension direction of the auxiliary components on the cargo handling vehicle in the target image based on the corner position of the second type of corner.

[0142] The direction of extension refers to the direction in which a functional component extends outward from the vehicle body in the target image. Figure 5 Here's an example to illustrate a method for determining the direction of extension:

[0143] (2.1) Assuming that corner points 5011 and 5012 are second-type corner points, the direction of the edge 5015 of the vehicle body 501 in the target image is first determined according to the relative position between corner points 5011 and 5012. The edge 5015 is the edge obtained by connecting corner points 5011 and 5012.

[0144] (2.2) Based on the direction of edge 5015 in the target image, determine the direction of the normal line pointing from edge 5015 to the outside of vehicle body 501 in the target image, that is, the direction of normal line 5016.

[0145] (2.3) Take the direction of normal 5016 as the extension direction.

[0146] (3) The extended area is determined based on the area of ​​the vehicle body area by using a preset area conversion strategy.

[0147] The extended area refers to the area of ​​the newly added region after extending the vehicle body area in the target image. Since the purpose of the extension is to obtain the area corresponding to the functional component in the target image, the extended area is the area of ​​the area corresponding to the functional component in the target image.

[0148] The area conversion strategy is a preset strategy for calculating the extended area. In some embodiments, the area conversion strategy can be a formula for calculating the extended area. For example, formula (1) can be used as the area conversion strategy:

[0149] S1 = S0 * a Equation (1)

[0150] Where S1 is the extended area, S0 is the area of ​​the vehicle body region, and a is the preset area conversion coefficient.

[0151] Since the proportion of the projected area between the functional components and the vehicle body on the horizontal plane is similar for any type of cargo handling vehicle, and the image acquisition equipment is usually set at the top of the site, the acquired target image can be approximated as a top view. Therefore, the proportion of the projected area between the functional components and the vehicle body on the horizontal plane of multiple cargo handling vehicles can be calculated in advance, and the average of each proportion of the projected area is taken as 'a' in formula (1). The extended area can be calculated according to formula (1).

[0152] (4) Determine the extension area of ​​the auxiliary component in the target image based on the extension area and the extension direction.

[0153] In some embodiments, the vehicle body area can be extended along the extension direction until the extended area reaches the extension area to obtain the extended area.

[0154] In addition to the methods described above for obtaining the extended area, the extended area can also be set to a fixed value, and the vehicle body area can be extended according to the extension direction and the extended area to obtain the extended area.

[0155] In other embodiments, the corner position of the second type of corner point can be used as the starting point of the extension, and the extension endpoint can be determined according to the extension area in the extension direction. The area enclosed by the second type of corner point and the extension endpoint can be used as the extension area.

[0156] It should be noted that the method for obtaining the extended region described above should not be considered a limitation of the embodiments of this application. For example, the method for obtaining the extension direction can also be modified. For instance, when obtaining the extension direction as described above, instead of using the normal direction as the extension direction, the normal direction can be rotated by a preset angle, and the resulting direction can be used as the extension direction.

[0157] 403. Determine the overall vehicle area of ​​the cargo handling vehicle in the target image based on the vehicle body area and the extended area.

[0158] The overall vehicle region refers to the total area corresponding to the vehicle body and functional components of the freight transport vehicle in the target image. The overall vehicle region is obtained by overlaying the vehicle body region and its extended regions. Figure 5 For example, if the only functional component is the fork 502, then the overall vehicle area refers to the area between the vehicle body 501 and the fork 502. Figure 5 The corresponding area within. If the functional component includes both fork 502 and fork 503, then the overall vehicle area refers to the area where the vehicle body 501, fork 502, and fork 503 are located. Figure 5 The corresponding area in the middle.

[0159] 404. If the overall area of ​​the vehicle at least partially overlaps with the preset target warning area in the target image, then the violation attribute information corresponding to the target warning area shall be used as the violation attribute information corresponding to the location information.

[0160] The target warning area can be understood as any one of the vehicle restricted areas described above, corresponding to the area in the target image.

[0161] If the overall area of ​​the vehicle at least partially overlaps with the preset target warning area in the target image, it indicates that the cargo handling vehicle has entered the target vehicle restricted area corresponding to the target warning area. Therefore, it is necessary to determine whether the cargo handling vehicle is not allowed to enter the target vehicle restricted area. The violation attribute information corresponding to the target warning area, i.e. the violation attribute information corresponding to the target vehicle restricted area, can be used as the violation attribute information corresponding to the location information. The target vehicle attribute information of the cargo handling vehicle is matched with the violation attribute information corresponding to the target warning area to determine whether the cargo handling vehicle has violated the rules.

[0162] Furthermore, a threshold can be preset to determine the size of the overlapping area between the overall vehicle area and the target warning area. When the overlapping area is less than the threshold, it indicates that the cargo handling vehicle may have only entered the target vehicle restricted area due to poor operator control. Since the vehicle body entering the target vehicle restricted area is very small, even if the target vehicle attribute information of the cargo handling vehicle matches the violation attribute information corresponding to the target vehicle restricted area, no safety problem will occur, and it can be considered that the cargo handling vehicle has not entered the target vehicle restricted area. Conversely, when the overlapping area is greater than or equal to the threshold, it indicates that most of the cargo handling vehicle has entered the target vehicle restricted area. At this time, it can no longer be assumed that the cause is due to poor operator control. It is necessary to use the violation attribute information corresponding to the target warning area, i.e., the violation attribute information corresponding to the target vehicle restricted area, as the violation attribute information corresponding to the location information, and match the target vehicle attribute information of the cargo handling vehicle with the violation attribute information corresponding to the target warning area to determine whether the cargo handling vehicle in the target image is in violation.

[0163] In some embodiments, the movement direction of the cargo handling vehicle in the target image can be predicted based on the corner point position, and the violation attribute information corresponding to the vehicle restricted area that the cargo handling vehicle may enter can be used as the violation attribute information corresponding to the location information. (Reference) Figure 6 At this point, obtaining the violation attribute information corresponding to the location information includes:

[0164] 601. Based on the corner positions of each type of corner point in the location information, predict the movement direction of the cargo handling vehicle in the target image.

[0165] In step 601, the corner point positions include the positions of various types of corner points. For example, it may include the corner point positions of the second type of corner point and the corner point positions of the third type of corner point, or it may include the corner point positions of the first subtype of corner point, the second subtype of corner point, the third subtype of corner point, and the fourth subtype of corner point. The descriptions of the second type of corner point, the third type of corner point, the first subtype of corner point, the second subtype of corner point, the third subtype of corner point, and the fourth subtype of corner point can be found above, and will not be repeated here.

[0166] The direction of movement in step 601 refers to the potential direction of movement of the cargo handling vehicle, that is, the possible direction of movement of the cargo handling vehicle in the next step.

[0167] For example, the direction of motion can be obtained by the same method as obtaining the direction of extension, for example, the direction of motion can be obtained according to the method in steps (2.1)-(2.3) above.

[0168] 602. Determine the target position of the cargo handling vehicle in the target image based on the direction of movement and the corner positions of each type of corner point.

[0169] In some embodiments, the corner positions of each type of corner point in the target image can be used as the starting point, and the direction of movement can be used as the direction of the movement trajectory to obtain the movement trajectory of each type of corner point. Each point on the movement trajectory can be used as the position of the moving target.

[0170] 603. Match the location of the moving target with a preset target warning area in the target image to determine whether the target warning area is the moving target area of ​​the cargo handling vehicle.

[0171] 604. If the target warning area is the movement target area of ​​the cargo handling vehicle, then the violation attribute information corresponding to the target warning area shall be used as the violation attribute information corresponding to the location information.

[0172] In some embodiments, it can be determined whether the location range of the target warning area in the target image includes at least one of the moving target locations. If the location range of the target warning area in the target image includes at least one of the moving target locations, it indicates that the target warning area is the moving target area of ​​the cargo handling vehicle in the target image, that is, the cargo handling vehicle in the target image may enter the target vehicle restricted area. At this time, it is necessary to match the target vehicle attribute information with the violation attribute information corresponding to the target warning area, that is, the violation attribute information corresponding to the target vehicle restricted area, to determine whether the cargo handling vehicle in the target image may violate the rules.

[0173] Similarly, a threshold can be set to determine whether the reason the target warning area is a moving target area is due to poor operator control. See step 404 for details, which will not be elaborated further.

[0174] In some embodiments, before obtaining violation attribute information, it can be determined whether the cargo handling vehicle in the target image is illegally carrying cargo. (Reference) Figure 7 At this point, before obtaining the violation attribute information corresponding to the location information, the method further includes:

[0175] 701. Extract the vehicle identity information and cargo information from the target vehicle attribute information.

[0176] For details on vehicle identity information and cargo information, please refer to the description of the target vehicle attribute information in step 302.

[0177] Referring to Table 1, which shows the specific details after extracting vehicle identity information and cargo information from the target vehicle attribute information for a forklift:

[0178] Types of forklifts Whether it carries cargo Types of goods small forklift yes Large cargo

[0179] Table 1

[0180] Among them, the forklift type corresponds to the vehicle identity information in the target vehicle attribute information, and whether it is carrying cargo and the type of cargo correspond to the cargo information in the target vehicle attribute information.

[0181] Referring to Table 2, which shows the details after extracting vehicle identity information and cargo information from the target vehicle attribute information (forklift), another type of cargo handling vehicle is shown:

[0182] Types of forklifts Whether it carries cargo Types of goods Used for moving furniture yes furniture

[0183] Table 2

[0184] Similarly, the forklift type corresponds to the vehicle identity information in the target vehicle attribute information, and whether it is carrying cargo and the type of cargo correspond to the cargo information in the target vehicle attribute information.

[0185] 702. If the cargo information and the vehicle identity information match the preset compliant cargo information, then the step of obtaining the violation attribute information corresponding to the location information is executed.

[0186] Preset compliant cargo loading information refers to the cargo loading information corresponding to the vehicle identity information in the target vehicle's attribute information when loading cargo in compliance with regulations. For example, preset compliant cargo loading information could refer to the type of goods corresponding to the vehicle identity information in the target vehicle's attribute information when loading cargo in compliance with regulations. Different types of cargo handling vehicles have different cargo loading capacities. If the cargo currently being transported by a cargo handling vehicle exceeds its corresponding cargo loading capacity, dangers such as rollover may occur. Therefore, it is necessary to pre-limit the types of cargo that different types of cargo handling vehicles can transport, i.e., to set corresponding preset compliant cargo loading information for different types of cargo handling vehicles.

[0187] For example, different preset compliant loading information can be set according to the type of goods handling vehicle. For instance, "small goods" can be used as the preset compliant loading information for small forklifts with relatively weak loading capacity; that is, when a small forklift is handling small goods, the forklift is deemed compliant. Alternatively, "small goods," "medium goods," and "large goods" can be used as the preset compliant loading information for large forklifts with relatively strong loading capacity; that is, regardless of the type of goods handled, a large forklift is deemed compliant. In this example, if the forklift's loading information and vehicle identification information are as shown in Table 1, then the forklift is deemed non-compliant. Or, "furniture" can be used as the preset compliant loading information for forklifts whose type is "used for handling furniture"; that is, when a forklift used for handling furniture handles furniture, the forklift is deemed compliant. In this example, if the forklift's loading information and vehicle identification information are as shown in Table 2, then the forklift is deemed compliant.

[0188] If the cargo handling vehicle is determined to be compliant, the step of obtaining the violation attribute information corresponding to the location information can be executed. If the cargo handling vehicle is determined to be in violation, an alert message can be issued to trigger an alarm.

[0189] In some embodiments, it can be achieved through Figure 1 The information acquisition model 100 obtains the location information and target vehicle attribute information of the cargo handling vehicle in the cargo target image. At this time, the target image can be input. Figure 1 The system uses a pre-defined information acquisition model 100 to obtain location information and target vehicle attribute information. (Reference) Figure 8 The preset information acquisition model 100 can be trained in the following ways:

[0190] 801. Obtain training data, wherein the training data includes training images and the actual positions of corner points of various types in the training images.

[0191] For example, a large number of cargo handling vehicle images can be obtained from a preset database, and data augmentation processing can be performed on a portion of these images. This application does not restrict the format of any of the images; for example, training images and target images can be in formats such as XML or JPG. In some embodiments, a portion of the obtained cargo handling vehicle images can be randomly selected, and data augmentation processing such as flipping, random cropping, color dithering, translation, scaling, contrast adjustment, and noise perturbation can be performed on the selected images. Then, the augmented images are normalized to obtain training images. Next, various types of corner points are labeled in the training images using methods such as manual annotation or machine annotation. Specific annotation methods can refer to the method described in step 402, and will not be elaborated further.

[0192] 802. Extract the relative distribution features of each type of corner point in the training image.

[0193] The relative distribution features contain information about the relative positional relationships between corner points of different types in the training image. For example, this can be achieved through... Figure 1 The first feature extraction layer 102 in the initial information acquisition model 100 extracts the relative distribution features of each type of corner point in the training image. For example, when the first feature extraction layer 102 is composed of an hourglass network, the training image processed by the preprocessing layer 101 can be upsampled and downsampled multiple times to extract features of each type of corner point at different scales in the training image. Then, the features at different scales are fused to obtain the relative distribution features of each type of corner point in the training image. The relative distribution features can be represented in the form of a heat map.

[0194] A heatmap is a feature map that can characterize the distribution of objects.

[0195] 803. Based on the relative distribution characteristics, determine the corner position characteristics of each type of corner point in the training image.

[0196] Corner location features contain the location information of various types of corner points in the training image. For example, this can be achieved through... Figure 1The second feature extraction layer 103 in the initial information acquisition model 100 obtains the corner position features of each type of corner point in the training image. For example, when the second feature extraction layer 103 is composed of an hourglass network, the relative distribution features output by the first feature extraction layer 102 can be upsampled and downsampled multiple times to extract features of each type of corner point at different scales in the relative distribution features. Then, the features at different scales are fused to obtain the corner position features of each type of corner point in the training image. Similarly, the corner position features can also be represented in the form of a heatmap.

[0197] Understandably, since the relative positional relationships between different types of corner points in the relative distribution features are taken into account when determining the corner point position features, the corner point position information contained in the obtained corner point position features is more accurate than the corner point position information obtained by other methods.

[0198] 804. Based on the corner point location characteristics, predict the predicted positions of each type of corner point.

[0199] For example, it can be done by Figure 1 The initial information acquisition model 100 uses a location prediction layer 104 to predict the locations of corner points of various types in the training image. Assuming the corner point types include a first type of corner point, and the first type of corner point further includes a first sub-type corner point, a second sub-type corner point, a third sub-type corner point, and a fourth sub-type corner point, then if... Figure 5 The predicted locations obtained from the training images can be found in Table 3.

[0200]

[0201] Table 3

[0202] Wherein, x1, y1, x2, y2, x3, y3, x4, y4 are the coordinates of the first subclass corner point 5011, the second subclass corner point 5012, the third subclass corner point 5013, and the fourth subclass corner point 5014 respectively in the preset image coordinate system within the training image. In this embodiment, the origin and coordinate axis range of the image coordinate system established in the training image are not limited.

[0203] 805. Based on the predicted and actual positions of the corner points of each type, adjust the parameters in the initial transport vehicle detection model to obtain the preset transport vehicle detection model.

[0204] In this embodiment of the application, the transport vehicle detection model can be understood as... Figure 1 Information acquisition model 100 in the middle.

[0205] Before performing steps 801-805, training preparation can be carried out through the following steps:

[0206] (1) Build a training environment for the transport vehicle detection model. The training environment refers to the configuration environment of the model.

[0207] (2) Construct a transport vehicle detection model and define the loss function of the transport vehicle detection model. The loss function can be a cross-entropy loss function, a squared loss function, etc.

[0208] (3) Modify the training parameter configuration. Training parameters may include training step size, training rate, etc.

[0209] (4) Import the pre-trained weights of the transport vehicle detection model to obtain the initial transport vehicle detection model.

[0210] After completing the above training preparation steps, the labeled training images can be input into the initial transport vehicle detection model for training.

[0211] In addition, the training images in step 801 can be divided into a training image set and a test image set. The initial transport vehicle detection model can be trained using the training images in the training image set, and the trained transport vehicle detection model can be verified using the training images in the test image set. If the trained transport vehicle detection model meets the preset training termination condition, then the trained transport vehicle detection model can be used as the preset transport vehicle detection model.

[0212] In some embodiments, it can also be determined whether the operator of the cargo handling vehicle corresponds to the personnel on the shift schedule, in order to determine whether the cargo handling vehicle is used by a designated person. For example, this can be determined by the following methods:

[0213] (1) Perform facial recognition on the target image to determine the operator information of the cargo handling vehicle. The operator information includes the personnel information of the personnel currently operating the cargo handling vehicle.

[0214] (2) Obtain the target personnel information corresponding to the target vehicle attribute information of the cargo handling vehicle. The target personnel information may be the personnel information of the compliant operators of the cargo handling vehicle pre-determined in the shift schedule.

[0215] (3) Match the operator information with the target personnel information. If the operator information and the target personnel information are successfully matched, it means that the cargo handling vehicle is for dedicated use by a specific person.

[0216] This application also provides a module structure for executing a method to determine violations by a transport vehicle, as described in the embodiments below. Figure 9 , Figure 9The module structure includes a transport vehicle detection module 901, an attribute detection module 902, an image acquisition module 903, and a terminal module 904. The transport vehicle detection module 901 includes a detection dataset creation module 9011 and a detection model training module 9012. The detection dataset creation module 9011 is used to acquire and label training images, and can also be used to divide the training images into training image sets and test image sets. The detection model training module 9012 is used for training preparation, constructing an initial transport vehicle detection model, and training the initial transport vehicle detection model using the training image set and test image set output by the detection dataset creation module 9011 to obtain a preset transport vehicle detection model.

[0217] The attribute detection module 902 is used to process the target image through a preset handling vehicle detection model in order to determine the violation judgment result of the goods handling vehicle.

[0218] The image acquisition module 903 is used to acquire the target image.

[0219] Terminal module 904 is used to issue a warning message when a cargo handling vehicle violates regulations.

[0220] To better implement the method for judging violations of transport vehicles in the embodiments of this application, this application also provides a device for judging violations of transport vehicles, such as... Figure 10 The diagram shown is a structural schematic of one embodiment of the handling vehicle violation judgment device in this application. The handling vehicle violation judgment device 1000 includes:

[0221] The first acquisition unit 1001 is used to acquire the target image;

[0222] Extraction unit 1002 is used to extract the location information and target vehicle attribute information of the cargo handling vehicle in the target image;

[0223] The second acquisition unit 1003 is used to acquire the violation attribute information corresponding to the location information;

[0224] The determining unit 1004 is used to determine the violation judgment result of the cargo handling vehicle based on the violation attribute information and the target vehicle attribute information.

[0225] In one possible implementation of this application, the second acquisition unit 1003 is further configured to:

[0226] Based on the corner positions of the first type of corner points in the location information, the vehicle body area of ​​the cargo handling vehicle in the target image is determined;

[0227] Based on the corner point position of the second type of corner point in the first type of corner point, determine the extended area of ​​the auxiliary components on the cargo handling vehicle in the target image;

[0228] Based on the vehicle body area and the extended area, determine the overall vehicle area of ​​the cargo handling vehicle in the target image;

[0229] If the overall area of ​​the vehicle at least partially overlaps with the preset target warning area in the target image, then the violation attribute information corresponding to the target warning area is used as the violation attribute information corresponding to the location information.

[0230] In one possible implementation of this application, the second acquisition unit 1003 is further configured to:

[0231] Extract the corner point positions of the second type of corner points from the first type of corner points;

[0232] Based on the corner point positions of the second type of corner points, determine the extension direction of the auxiliary components on the cargo handling vehicle in the target image;

[0233] The extended area is determined based on the area of ​​the vehicle body region using a preset area conversion strategy.

[0234] The extension area of ​​the auxiliary component in the target image is determined based on the extension area and the extension direction.

[0235] In one possible implementation of this application, the second acquisition unit 1003 is further configured to:

[0236] Based on the corner positions of each type of corner point in the location information, predict the movement direction of the cargo handling vehicle in the target image;

[0237] Based on the direction of movement and the corner positions of each type of corner point, determine the target position of the cargo handling vehicle in the target image;

[0238] The location of the moving target is matched with a preset target warning area in the target image to determine whether the target warning area is the moving target area of ​​the cargo handling vehicle.

[0239] If the target warning area is the movement target area of ​​the cargo handling vehicle, then the violation attribute information corresponding to the target warning area is used as the violation attribute information corresponding to the location information.

[0240] In one possible implementation of this application, the handling vehicle violation judgment device 1000 further includes a cargo judgment unit 1005, which is used for:

[0241] Extract the vehicle identity information and cargo information from the target vehicle attribute information;

[0242] If the cargo information and the vehicle identity information match the preset compliant cargo information, then the step of obtaining the violation attribute information corresponding to the location information is executed.

[0243] In one possible implementation of this application, the extraction unit 1002 is further configured to:

[0244] The target image is input into a preset handling vehicle detection model to obtain the location information of the cargo handling vehicle and the target vehicle attribute information.

[0245] In one possible implementation of this application, the extraction unit 1002 is further configured to:

[0246] Acquire training data, wherein the training data includes training images and the actual positions of corner points of various types in the training images;

[0247] Extract the relative distribution features of each type of corner point in the training image;

[0248] Based on the relative distribution characteristics, the corner position features of each type of corner point in the training image are determined;

[0249] Based on the corner point location characteristics, the predicted locations of each type of corner point are obtained;

[0250] Based on the predicted and actual positions of the corner points of each type, the parameters in the initial transport vehicle detection model are adjusted to obtain the preset transport vehicle detection model.

[0251] In one possible implementation of this application, the transport vehicle violation judgment device 1000 further includes a face recognition unit 1006, which is used for:

[0252] Facial recognition is performed on the target image to determine the operator information of the cargo handling vehicle;

[0253] Obtain the target personnel information corresponding to the target vehicle attribute information of the cargo handling vehicle;

[0254] If the operator information matches the target personnel information, then the step of determining the violation judgment result of the cargo handling vehicle based on the violation attribute information and the target vehicle attribute information is executed.

[0255] In practice, each of the above units can be implemented as an independent entity or can be arbitrarily combined to be implemented as the same or several entities. For the specific implementation of each of the above units, please refer to the previous method embodiments, which will not be repeated here.

[0256] Since the transport vehicle violation judgment device can execute the steps in the transport vehicle violation judgment method in any embodiment, it can achieve the beneficial effects that the transport vehicle violation judgment method in any embodiment of this application can achieve, as detailed in the preceding description, and will not be repeated here.

[0257] Furthermore, to better implement the method for judging violations of transport vehicles in this application, this application also provides an electronic device based on the method for judging violations of transport vehicles. (See attached document.) Figure 11 , Figure 11 This illustration shows a structural diagram of an electronic device according to an embodiment of this application. Specifically, the electronic device provided in this embodiment includes a processor 1101. The processor 1101 is used to execute the computer program stored in the memory 1102 to implement each step of the method for judging violations of transport vehicles in any embodiment; or, the processor 1101 is used to execute the computer program stored in the memory 1102 to implement, for example... Figure 10 The functions of each unit in the corresponding embodiment.

[0258] For example, a computer program may be divided into one or more modules / units, one or more of which are stored in memory 1102 and executed by processor 1101 to complete the embodiments of this application. One or more modules / units may be a series of computer program instruction segments capable of performing a specific function, which describe the execution process of the computer program in a computer device.

[0259] The electronic device may include, but is not limited to, processor 1101 and memory 1102. Those skilled in the art will understand that the illustrations are merely examples of an electronic device and do not constitute a limitation on the device. It may include more or fewer components than illustrated, or combine certain components, or use different components.

[0260] Processor 1101 can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor. The processor is the control center of the electronic device, connecting various parts of the electronic device through various interfaces and lines.

[0261] The memory 1102 can be used to store computer programs and / or modules. The processor 1101 implements various functions of the computer device by running or executing the computer programs and / or modules stored in the memory 1102 and by calling data stored in the memory 1102. The memory 1102 may mainly include a program storage area and a data storage area. The program storage area may store the operating system, application programs required for at least one function (such as sound playback function, image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device (such as audio data, video data, etc.). In addition, the memory may include high-speed random access memory, and may also include non-volatile memory, such as hard disk, RAM, plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, at least one disk storage device, flash memory device, or other volatile solid-state storage device.

[0262] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working process of the above-described handling vehicle violation judgment device, electronic equipment and its corresponding units can be referred to the description of the handling vehicle violation judgment method in any embodiment, and will not be repeated here.

[0263] Those skilled in the art will understand that all or part of the steps in the various methods of the above embodiments can be implemented by instructions, or by instructions controlling related hardware. These instructions can be stored in a readable storage medium and loaded and executed by a processor.

[0264] Therefore, embodiments of this application provide a readable storage medium storing a computer program. When the computer program is executed by a processor, it performs the steps in the method for judging violations of transport vehicles in any embodiment of this application. For specific operations, please refer to the description of the method for judging violations of transport vehicles in any embodiment, which will not be repeated here.

[0265] The readable storage medium may include: read-only memory (ROM), random access memory (RAM), magnetic disk or optical disk, etc.

[0266] Since the instructions stored in the readable storage medium can execute the steps in the method for judging violations of transport vehicles in any embodiment of this application, the beneficial effects that the method for judging violations of transport vehicles in any embodiment of this application can achieve can be realized, as detailed in the preceding description, and will not be repeated here.

[0267] The foregoing has provided a detailed description of a method, apparatus, storage medium, and electronic device for judging violations of transport vehicles, as provided in the embodiments of this application. Specific examples have been used to illustrate the principles and implementation methods of this application. The descriptions of the embodiments above are only for the purpose of helping to understand the method and core ideas of this application. At the same time, for those skilled in the art, there will be changes in the specific implementation methods and application scope based on the ideas of this application. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A method for judging violations by transport vehicles, characterized in that, include: Acquire the target image; Extract the location information and target vehicle attribute information of the cargo handling vehicle in the target image; Obtaining violation attribute information corresponding to the location information includes determining the vehicle body area of ​​the cargo handling vehicle in the target image based on the corner position of the first type of corner point in the location information; extracting the corner position of the second type of corner point in the first type of corner point; determining the extension direction of the auxiliary components on the cargo handling vehicle in the target image based on the corner position of the second type of corner point; and determining the extension area based on the area of ​​the vehicle body area through a preset area conversion strategy. The extension area of ​​the auxiliary component in the target image is determined based on the extension area and the extension direction; The violation attribute information is obtained based on the vehicle body area and the extended area; Based on the violation attribute information and the target vehicle attribute information, the violation judgment result of the cargo handling vehicle is determined.

2. The method for judging violations by transport vehicles according to claim 1, characterized in that, The step of obtaining the violation attribute information based on the vehicle body area and the extended area includes: Based on the vehicle body area and the extended area, determine the overall vehicle area of ​​the cargo handling vehicle in the target image; If the overall area of ​​the vehicle at least partially overlaps with the preset target warning area in the target image, then the violation attribute information corresponding to the target warning area is used as the violation attribute information corresponding to the location information.

3. The method for judging violations by transport vehicles according to claim 1, characterized in that, The step of obtaining the violation attribute information corresponding to the location information includes: Based on the corner positions of each type of corner in the location information, the movement direction of the cargo handling vehicle in the target image is predicted; wherein, the corner positions include the corner positions of the second type of corner and the corner positions of the third type of corner; Based on the direction of movement and the corner positions of each type of corner point, the target position of the cargo handling vehicle in the target image is determined; wherein, the corner positions of each type of corner point in the target image are taken as the starting point, the direction of movement is taken as the direction of the movement trajectory, and the movement trajectory of each type of corner point is obtained, and each point on the movement trajectory is the target position of the vehicle; The location of the moving target is matched with a preset target warning area in the target image to determine whether the target warning area is the moving target area of ​​the cargo handling vehicle. If the target warning area is the movement target area of ​​the cargo handling vehicle, then the violation attribute information corresponding to the target warning area is used as the violation attribute information corresponding to the location information.

4. The method for judging violations by transport vehicles according to claim 1, characterized in that, Before obtaining the violation attribute information corresponding to the location information, the method further includes: Extract the vehicle identity information and cargo information from the target vehicle attribute information; If the cargo information and the vehicle identity information match the preset compliant cargo information, then the step of obtaining the violation attribute information corresponding to the location information is executed.

5. The method for judging violations by transport vehicles according to claim 1, characterized in that, The step of extracting the location information and target vehicle attribute information of the cargo handling vehicle in the target image includes: The target image is input into a preset handling vehicle detection model to obtain the location information of the cargo handling vehicle and the target vehicle attribute information; The preset transport vehicle detection model is trained through the following steps: Acquire training data, wherein the training data includes training images and the actual positions of corner points of various types in the training images; Extract the relative distribution features of each type of corner point in the training image; Based on the relative distribution characteristics, the corner position features of each type of corner point in the training image are determined; Based on the corner point location characteristics, the predicted locations of each type of corner point are obtained; Based on the predicted and actual positions of the corner points of each type, the parameters in the initial transport vehicle detection model are adjusted to obtain the preset transport vehicle detection model.

6. The method for judging violations of handling vehicles according to any one of claims 1-5, characterized in that, Before determining the violation judgment result of the cargo handling vehicle based on the violation attribute information and the target vehicle attribute information, the method further includes: Facial recognition is performed on the target image to determine the operator information of the cargo handling vehicle; Obtain the target personnel information corresponding to the target vehicle attribute information of the cargo handling vehicle; If the operator information matches the target personnel information, then the step of determining the violation judgment result of the cargo handling vehicle based on the violation attribute information and the target vehicle attribute information is executed.

7. A device for detecting violations by transport vehicles, characterized in that, include: The first acquisition unit is used to acquire the target image; The extraction unit is used to extract the location information and target vehicle attribute information of the cargo handling vehicle in the target image; The second acquisition unit is used to acquire violation attribute information corresponding to the location information; wherein, based on the corner positions of the first type of corner points in the location information, the vehicle body area of ​​the cargo handling vehicle in the target image is determined; the corner positions of the second type of corner points in the first type of corner points are extracted; based on the corner positions of the second type of corner points, the extension direction of the auxiliary components on the cargo handling vehicle in the target image is determined; based on the area of ​​the vehicle body area using a preset area conversion strategy, the extension area is determined; based on the extension area and the extension direction, the extension area of ​​the auxiliary components in the target image is determined; and based on the vehicle body area and the extension area, the violation attribute information is acquired. The determining unit is used to determine the violation judgment result of the cargo handling vehicle based on the violation attribute information and the target vehicle attribute information.

8. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps in the method for determining violations of transport vehicles as described in any one of claims 1 to 6.

9. A readable storage medium, characterized in that, The readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the handling vehicle violation judgment method according to any one of claims 1 to 6.

Images