Monitoring equipment early warning method and system based on improved image recognition model

Abstract

The invention provides a monitoring equipment early warning method and system based on an improved image recognition model. The method comprises the following steps: acquiring effective image information of target traction equipment by using monitoring equipment; according to the effective image information, obtaining monitoring state information of the target traction equipment by using an improved image recognition model; when the monitoring state information indicates that the loading of the target traction equipment is abnormal, generating early warning information and sending the early warning information to the user equipment; wherein the improved image recognition model comprises an image segmentation model and a detection scoring model. The method does not depend on the computing power of the traction equipment, reduces the hardware requirements for the transportation device, and reduces the overall monitoring and early warning cost. The image recognition model can quickly process image information, discover abnormity in time and give an early warning, and does not depend on the state of the transportation device, so that the real-time performance and reliability of monitoring are enhanced.

Description

Technical Field

[0001] The present application relates to the technical field of computer vision and image recognition, and in particular to a monitoring device warning method and system based on an improved image recognition model. Background Art

[0002] With the advancement of intelligent manufacturing, the requirements for the automation and intelligence of the goods loading, unloading and transportation processes in automated logistics and intelligent factories are getting higher and higher. In this process, the monitoring of abnormal situations during the transportation of goods is essential.

[0003] In response to the monitoring requirements for abnormal situations during the transportation of goods, related technologies provide a trailer cargo loss monitoring system, which includes a human-machine interaction system installed on the tractor. The human-machine interaction system is communicatively connected to sensors installed on the bottom plates of multiple trailers carrying goods. During the driving process of the tractor, the human-machine interaction system receives the monitoring results from the sensors, and the monitoring results reflect whether there is any cargo loss at the monitored position. In this way, the human-machine interaction system can timely send a reminder to the driver to check and confirm whether the goods have fallen from the trailer. It monitors the cargo loss situation of the trailer through sensors and transmits the monitoring results to the human-machine interaction system, so that the tractor driver can timely obtain relevant information during the driving process and avoid the situation where the cargo loss of the trailer fails to be detected in time.

[0004] However, its monitoring process depends on the computing power of the tractor itself, and the communication data volume and data processing volume are relatively large. That is to say, related technologies mainly rely on the transportation device itself to monitor abnormal situations during the transportation of goods, and there are defects of relatively high costs. Further, effective monitoring cannot be achieved when the transportation device has an abnormal failure.

[0005] Therefore, there is an urgent need to provide a monitoring device warning method and system based on an improved image recognition model to enhance the decoding success rate. Summary of the Invention

[0006] In view of the above technical problems, the present application provides a monitoring device warning method and system based on an improved image recognition model.

[0007] The object of the present application is achieved by the following technical solutions: In the first aspect, the present application provides a monitoring device warning method based on an improved image recognition model, and the method includes the steps of:

[0008] S1, using a monitoring device to obtain effective image information of a target towing device;

[0009] S2, according to the effective image information, using an improved image recognition model to obtain the monitoring status information of the target towing device;

[0010] S3. When the monitored status information indicates that the load of the target towing device is abnormal, generate a warning message and send it to the user device;

[0011] Among them, the improved image recognition model includes an image segmentation model and a detection scoring model; the image segmentation model is used to extract target features in the effective image information, and the detection scoring model is used to obtain monitored status information based on the target features.

[0012] The beneficial effects of this technical solution are as follows: It does not rely on the computing power of the towing device itself, reduces the hardware requirements for the transportation device, and thus reduces the overall monitoring and warning costs. The image recognition model can quickly process image information, discover abnormalities and give warnings in a timely manner, and does not rely on the status of the transportation device itself. Compared with the situation in related technologies where the transportation device itself is relied on for monitoring, with a large amount of data processing and low efficiency, it enhances the real-time performance and reliability of monitoring.

[0013] In some possible implementation manners, the target towing device includes a tractor and a trailer, and step S2 includes:

[0014] According to the effective image information, use the image segmentation model to extract the body structure features of the tractor and the load area features of the trailer as the target features in the effective image information;

[0015] Obtain an abnormality judgment strategy according to the body structure features; according to the abnormality judgment strategy and the load area features, use the detection scoring model to obtain an abnormality detection score and use it as the monitored status information;

[0016] Step S3 includes: When the abnormality detection score is not higher than the first threshold, it is considered that the load of the target towing device is abnormal, generate a first warning message and send it to the user device.

[0017] The beneficial effects of this technical solution are as follows: Dynamically select the abnormality judgment strategy according to the state of the tractor body, avoiding misjudgments caused by one-size-fits-all rules. Process features in stages, independently extract the features of the tractor body and the load of the trailer, and avoid the failure of feature extraction caused by occlusion (such as goods covering the license plate). Distinguish the severity of abnormalities through multiple thresholds, and avoid frequent false alarms from consuming operation and maintenance resources.

[0018] In some possible implementation manners, the monitoring device includes a plurality of first collection devices arranged on the running route of the target towing device and a plurality of second collection devices arranged in a plurality of cargo loading areas on the traveling route;

[0019] Step S1 includes: Use the first collection device to obtain the first image information of the target towing device on the traveling route, and use the second collection device to obtain the second image information of the target towing device in the cargo loading area;

[0020] Step S3 further includes: when the anomaly detection score is between the first threshold and the second threshold, obtaining preset loading information according to the second image information; the second threshold is greater than the first threshold;

[0021] Obtaining predicted loading area information of the trailer according to the preset loading information and the current position information of the target towing device;

[0022] Judging whether the trailer is loaded incorrectly based on the predicted loading area information and the first image information; when it is judged that the trailer is loaded incorrectly, generating a second warning message and sending it to the user device.

[0023] The beneficial effects of this technical solution are as follows: By setting multiple acquisition devices on the running route of the towing device and the cargo loading area, comprehensive monitoring of the towing device at different positions and stages is realized. The multi-point monitoring method can timely detect abnormal conditions of the towing device during travel and loading, improving the comprehensiveness and accuracy of monitoring. Two thresholds are set, and different levels of warning messages are generated according to the anomaly detection score. The dual warning mechanism can more flexibly respond to abnormal conditions of different degrees. For example, when the abnormal condition is relatively serious, a first warning message is generated in time to remind the user to take measures immediately; when the abnormal condition is in a critical state, the loading information is further analyzed to judge whether there is an incorrect loading, and a second warning message is generated, enabling the user to check the loading situation in time. Through real-time monitoring and warning of the load state of the towing device, abnormal conditions can be detected and processed in time, reducing transportation accidents caused by cargo dropping, improper loading, etc., improving the safety of the transportation process, and reducing transportation risks and losses.

[0024] In some possible implementation manners, the second image information is two-dimensional code image information; the method for obtaining the second image information of the target towing device in the cargo loading area includes: using a second acquisition device to obtain the second image information from an electronic two-dimensional code screen arranged at the tail or both sides of the trailer.

[0025] The beneficial effects of this technical solution are as follows: By using two-dimensional code image information for monitoring, the hardware dependence on the towing device itself is reduced, and the hardware cost of the monitoring system is lowered. The cost of equipping the trailer with an electronic two-dimensional code screen is relatively low, which is convenient for integration and promotion.

[0026] In some possible implementation manners, obtaining the anomaly judgment strategy according to the ontology structure characteristics; obtaining the anomaly detection score according to the anomaly judgment strategy and the load area characteristics by using the detection scoring model, and using it as the monitoring status information, includes:

[0027] According to the type and value of the ontology structure characteristics, matching the anomaly judgment strategy and the weight coefficient from the preset strategy library through a rule engine;

[0028] Input the load area features into the anomaly judgment strategy, and calculate the anomaly detection score according to the deduction items and weight coefficients; use the anomaly detection score as the monitoring status information.

[0029] The beneficial effect of this technical solution is that through the rule engine, according to the predefined rules and logics, the extracted features are evaluated and scored. The flexibility and interpretability of the rule engine enable the anomaly judgment strategy to be easily adjusted and optimized, and can quickly adapt to different application scenarios and requirements.

[0030] In some possible implementation manners, the obtaining the predicted load area information of the trailer according to the preset loading information and the current position information of the target towing device includes:

[0031] Obtain the loading plan of the target towing device from the preset loading information, where the loading plan includes the loading sequence and placement requirements of the goods in different loading areas;

[0032] According to the loading plan and the current position information, take the previous loading area reached by the target towing device as the target loading area;

[0033] According to the goods information of the target loading area in the loading plan, predict the types, quantities, and arrangement modes of the goods in the load area of the trailer, and generate the predicted load area information.

[0034] The beneficial effect of this technical solution is that by obtaining the current position information of the towing device in real time, it can dynamically judge the loading area reached by the towing device, and generate the predicted load area information only when the anomaly detection score is between the first threshold and the second threshold, reducing data processing. By reducing the amount of data processing, computing resources can be utilized more efficiently, and energy consumption can be reduced. It can predict the loading and unloading conditions of the trailer in advance before entering the next loading area, help the user discover potential loading problems in advance, remind the user to instruct the loading personnel to check and handle problems in the next loading area, and reduce transportation accidents caused by improper loading.

[0035] In some possible implementation manners, the judging whether the trailer is loaded incorrectly based on the predicted load area information and the first image information includes:

[0036] Obtain the loading similarity between the predicted load area information and the first image information. When the value of the loading similarity is less than the preset similarity, it is considered that the loading is incorrect; otherwise, it is considered that the loading is normal, start counting and increment the count by one. When the count is greater than the preset number, clear the count, and send the second threshold anomaly information to the user device, where the second threshold anomaly information is used to prompt to reduce the value of the second threshold.

[0037] The beneficial effects of this technical solution are as follows: When the number of statistics exceeds the preset number, the user is prompted to lower the second threshold, which helps the user adjust the parameters according to the actual situation and improves the adaptability and accuracy of the method. The warning is triggered only when the similarity is lower than the threshold, reducing unnecessary data processing and improving the execution efficiency of the method.

[0038] Second, this application also provides a monitoring device warning device based on an improved image recognition model, including:

[0039] An information acquisition module, configured to use the monitoring device to acquire effective image information of the target traction device;

[0040] A model processing module, configured to use the improved image recognition model to obtain the monitoring status information of the target traction device according to the effective image information;

[0041] An information sending module, configured to generate a warning message and send it to the user device when the monitoring status information indicates that the load of the target traction device appears abnormal;

[0042] Wherein, the improved image recognition model includes an image segmentation model and a detection scoring model; the image segmentation model is used to extract target features in the effective image information, and the detection scoring model is used to obtain monitoring status information according to the target features.

[0043] Third, this application also provides a warning system, including the monitoring device warning device described in the second aspect and a monitoring device for acquiring effective image information of the target traction device.

[0044] In some possible implementation manners, the monitoring device includes a plurality of first collection devices arranged on the running route of the target traction device and a plurality of second collection devices arranged in a plurality of cargo loading areas on the traveling route. Description of the Drawings

[0045] The following further describes this application in conjunction with the drawings and embodiments.

[0046] Figure 1 is a schematic flowchart of a monitoring device warning method provided by an embodiment of this application;

[0047] Figure 2 is a schematic flowchart of a process for obtaining monitoring status information provided by an embodiment of this application;

[0048] Figure 3 is a partial schematic flowchart of a process for generating a warning message provided by an embodiment of this application;

[0049] Figure 4 is a module schematic diagram of a monitoring device warning device provided by an embodiment of this application;

[0050] Figure 5 It is a schematic diagram of a module of an early warning system provided by an embodiment of the present application;

[0051] Figure 6 It is a schematic flow chart of another method for warning a monitoring device provided by an embodiment of the present application. Detailed implementation manners

[0052] First, a brief description is given to the technical field and related terms of the embodiments of the present application, so as to facilitate the understanding of those skilled in the art.

[0053] A monitoring camera array refers to a camera system composed of multiple cameras, which provides more comprehensive and detailed scene information by capturing images from multiple perspectives simultaneously. These cameras are installed on a plane, for example, to ensure that they can capture the scenes of each image information acquisition point on the driving route of the target towing device simultaneously.

[0054] A two-dimensional code is a black-and-white image distributed in a plane (in two-dimensional directions) according to certain rules with specific geometric figures, which can store a large amount of information and can be quickly read by a scanning device, so as to realize the automatic collection and input of information.

[0055] In recent years, image recognition technology has been widely used in the field of industrial monitoring and early warning, but the existing technology still has deficiencies in dealing with complex scenes. On the one hand, because the monitoring process depends on the computing power of the tractor itself, the communication data volume and data processing volume are relatively large, and effective monitoring cannot be achieved when the transportation device has an abnormal fault. Further, to solve the above problems, the related technology mainly reduces the dimension of information source analysis and processing, reduces the data communication volume and processing volume, that is, relies on a single image for analysis and lacks the motivation to consider multi-source information.

[0056] To address the above challenges, this application proposes a monitoring device warning method and system based on an improved image recognition model. The first acquisition device is used to obtain the first image information of the target towing device on the travel route, and the second acquisition device is used to obtain the second image information from the electronic QR code screen at the tail or on both sides of the trailer. Both are used as valid (image) information. Further, an image segmentation model based on a deep neural network is used to accurately extract the body structure features of the tractor and the load area features of the trailer from the first image information. At the same time, according to the type and value of the body structure features, the anomaly judgment strategy and weight coefficient are matched from the preset policy library through the rule engine. The load area features are input into this strategy, and the anomaly detection score is calculated through the deduction items and weight coefficients as the monitoring status information. That is to say, this monitoring device warning method and system no longer rely on the computing power of the tractor itself, but perform data analysis through external image acquisition devices and deep neural network models, thereby reducing the system cost and avoiding the monitoring failure caused by the failure of the transportation device. Secondly, it reduces the communication data volume and data processing volume, improves the processing efficiency, and enhances the real-time performance and reliability. By combining the first image information and the second image information, the load status of the target towing device can be analyzed more comprehensively, reducing the possibility of misjudgment and missed judgment. The user in this application refers to the dispatcher or manager responsible for monitoring and managing the transportation process, while the loader refers to the staff responsible for loading the goods in the goods loading area (using equipment). They directly participate in the goods loading process to ensure that the goods are correctly loaded according to the preset loading plan.

[0057] Next, in combination with the accompanying drawings and specific implementation manners, the technical solutions of the embodiments of this application and how the technical solutions of the embodiments of this application solve the above technical problems will be specifically described. It should be noted that any combination of the following-described embodiments or technical features can form a new embodiment, and the same or similar concepts or processes may not be repeated in some embodiments. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments.

[0058] Method embodiment.

[0059] See Figure 1 , Figure 1 which is a schematic flowchart of a monitoring device warning method provided by an embodiment of this application. This embodiment provides a monitoring device warning method based on an improved image recognition model. The method includes:

[0060] S101, using a monitoring device to obtain valid image information of a target towing device;

[0061] It can be understood that the monitoring device is an external acquisition device relative to the target towing device, and the acquisition of effective image information does not depend on the sensors of the towing vehicle itself. By collecting images through external devices, the dependence on the computing power and sensors of the towing vehicle itself is avoided, and the hardware requirements for the target towing device are reduced.

[0062] S102. According to the effective image information, use an improved image recognition model to obtain the monitoring status information of the target towing device;

[0063] Among them, the improved image recognition model includes an image segmentation model and a detection scoring model; the image segmentation model is used to extract target features in the effective image information, and the detection scoring model is used to obtain monitoring status information according to the target features.

[0064] It can be understood that in the improved image recognition model, the image segmentation model is used to perform pixel-level semantic segmentation on the effective image information to extract key features. The key features are, for example, the body structure features of the towing vehicle (such as body color and pattern features, body size features, and license plate number features), and the load area features of the trailer (such as the projection area features of the goods on the towing vehicle and the shape features of the goods in the load area of the trailer). Based on the segmentation result, use the detection scoring model to obtain the monitoring status information, and the monitoring status information can include scoring data. The image segmentation model can accurately locate the key area, and the detection scoring model quantifies the degree of abnormality to achieve subsequent early warning.

[0065] Among them, the extracted features are evaluated by the detection scoring model to generate an anomaly detection score, and whether the load is abnormal is judged according to the score.

[0066] S103. When the monitoring status information indicates that the load of the target towing device is abnormal, generate a warning message and send it to the user device. Among them, the user device is, for example, a mobile phone, a tablet, a desktop computer, or an industrial control computer, and the warning message can be displayed to the user in the form of a text message, voice, or pop-up window.

[0067] The technical solution provided in this embodiment does not depend on the computing power of the towing device itself, reduces the hardware requirements for the transportation device, and thus reduces the overall monitoring and warning costs. The image recognition model can quickly process image information, detect abnormalities in a timely manner and give warnings, and does not depend on the state of the transportation device itself. Compared with the situation in the related art where a large amount of data needs to be processed and the efficiency is low when relying on the transportation device itself for monitoring, the real-time performance and reliability of the monitoring are enhanced.

[0068] See Figure 2 , Figure 2 which is a schematic flowchart of a process for obtaining monitoring status information provided by an embodiment of the present application.

[0069] In some embodiments, the target towing device includes a towing vehicle and a trailer, and step S102 includes:

[0070] S201, according to the effective image information, extract the body structure features of the towing vehicle and the load area features of the trailer through an image segmentation model and use them as the target features in the effective image information;

[0071] Extract the features of the components of the towing vehicle through the image segmentation model as the body structure features; perform semantic segmentation on the trailer area to extract cargo-related features, including load contours, stacking parameters, etc. Take the towing vehicle body and the trailer load as independent segmentation targets to avoid interference between the two features; the target features in the effective image information are used as structured outputs, that is, convert the segmentation results into quantifiable parameters (such as coordinates, area, angle), rather than the original pixel data, to reduce the subsequent calculation complexity.

[0072] S202, obtain an anomaly judgment strategy according to the body structure features; according to the anomaly judgment strategy and the load area features, use the detection scoring model to obtain an anomaly detection score and use it as the monitoring status information.

[0073] Step S103 includes: when the anomaly detection score is not higher than the first threshold, it is considered that the load of the target towing device is abnormal, generate a first warning message and send it to the user device. The anomaly detection score is, for example, on a percentile scale, such as 70, 80, 90; the value range of the first threshold is, for example, (60, 70], such as taking 65; the value range of the second threshold mentioned below is, for example, (70, 75], such as taking 73.

[0074] Thus, dynamically select the anomaly judgment strategy through the status of the towing vehicle body to avoid misjudgment caused by one-size-fits-all rules. Perform phased feature processing, independently extract the features of the towing vehicle body and the trailer load, and avoid feature extraction failure caused by occlusion (such as the cargo occluding the license plate).

[0075] In some embodiments, the monitoring device includes a plurality of first collection devices arranged on the running route of the target towing device and a plurality of second collection devices arranged in a plurality of cargo loading areas on the traveling route;

[0076] Step S101 includes: using the first collection device to obtain the first image information of the target towing device on the traveling route, and using the second collection device to obtain the second image information of the target towing device in the cargo loading area;

[0077] The first collection device is arranged on the running route of the target towing device and is used to obtain the first image information of the towing device during the traveling process. The first collection device is, for example, an image collection device such as a monitoring camera array, which can capture the appearance of the towing device running to the camera point, the cargo loading situation, etc. in real time.

[0078] The second acquisition device includes a camera, which is arranged in a plurality of cargo loading areas on the travel route and is used to obtain the second image information of the towing device in the loading area. It specifically collects images for the cargo loading area. It can be considered that there are a plurality of cargo loading areas between the starting point and the ending point of the driving route, and the cargo is loaded into the trailer in sequence according to the preset loading information in the preset loading plan.

[0079] See Figure 3 , Figure 3 which is a partial schematic flowchart of a method for generating a warning message provided by an embodiment of the present application.

[0080] Step S103 further includes:

[0081] S301, when the anomaly detection score is between the first threshold and the second threshold, obtain the preset loading information according to the second image information; the second threshold is greater than the first threshold;

[0082] S302, obtain the predicted loading area information of the trailer according to the preset loading information and the current position information of the target towing device;

[0083] S303, judge whether the trailer is loaded incorrectly based on the predicted loading area information and the first image information; when it is judged that the trailer is loaded incorrectly, generate a second warning message and send it to the user device.

[0084] When the anomaly detection score is not higher than the first threshold, it is considered that a serious anomaly has occurred to the load (such as dropping), and a first warning message is generated and sent to the user device. When the score is between the first threshold and the second threshold, considering that during the transportation of the tractor and the trailer, loading problems (such as incorrect quantity or type of loaded goods) are relatively common abnormal situations. Loading problems are different from cargo dropping, but they also affect the stability of the goods during transportation. The preset loading information is obtained using the second image information, and combined with the current position information, the predicted loading area information of the trailer is obtained. By comparing the predicted loading area information with the actual first image information, it is judged whether the trailer is loaded incorrectly (in the most recent previous loading area). If it is judged that the loading is incorrect, a second warning message is generated and sent to the user device.

[0085] Thus, by setting up multiple acquisition devices along the running route of the towing equipment and in the cargo loading area, comprehensive monitoring of the towing equipment at different positions and stages is achieved. The multi-point monitoring method can promptly detect abnormal conditions during the travel and loading processes of the towing equipment, improving the comprehensiveness and accuracy of monitoring. Two thresholds are set, and different levels of warning information are generated based on the abnormal detection score. The dual warning mechanism can more flexibly respond to abnormal conditions of different degrees. For example, when the abnormal condition is relatively serious, the first warning information is generated in a timely manner to remind the user to take immediate measures; when the abnormal condition is in a critical state, the loading information is further analyzed to determine whether there is a loading error, and the second warning information is generated, enabling the user to promptly check the loading situation. Through the real-time monitoring and warning of the load state of the towing equipment, abnormal conditions can be detected and handled in a timely manner, reducing transportation accidents caused by cargo dropping, improper loading, etc., improving the safety of the transportation process, and reducing transportation risks and losses.

[0086] In some embodiments, the second image information is two-dimensional code image information; the method for obtaining the second image information of the target towing equipment in the cargo loading area includes:

[0087] Using a second acquisition device to obtain the second image information from an electronic two-dimensional code screen arranged at the tail or both sides of the trailer.

[0088] The two-dimensional code displayed on the electronic two-dimensional code screen includes cargo loading information, etc. The obtained two-dimensional code image information is decoded to obtain the cargo loading situation after passing through the corresponding cargo loading area. The electronic two-dimensional code screen is, for example, an electronic ink screen, including a screen and a communication module. The communication module is communicatively connected to a local server to obtain preset loading information sent by a warehousing management system running on the local server. The preset loading information is encoded into a two-dimensional code and displayed on the screen of the electronic two-dimensional code screen.

[0089] Thus, by using two-dimensional code image information for monitoring, the hardware dependence on the towing equipment itself is reduced, and the hardware cost of the monitoring system is lowered. The cost of equipping the trailer with an electronic two-dimensional code screen is relatively low, which is convenient for integration and promotion.

[0090] In some embodiments, obtaining an abnormal judgment strategy according to the ontology structure characteristics; according to the abnormal judgment strategy and the load area characteristics, using the detection scoring model to obtain an abnormal detection score and serving as monitoring status information includes:

[0091] According to the type and value of the ontology structure characteristics, an abnormal judgment strategy and a weight coefficient are matched from a preset strategy library through a rule engine;

[0092] Inputting the load area characteristics into the abnormal judgment strategy, calculating the abnormal detection score according to the deduction items and the weight coefficient; using the abnormal detection score as the monitoring status information.

[0093] The detection scoring model, as a rule-based scoring model, is used to evaluate the degree of abnormality of the load area status. By matching rules and calculating scores, it can quickly determine whether the load is abnormal, which is suitable for real-time monitoring and quick decision-making. For example, it is a decision tree model that divides the feature space of the object to be scored into multiple decision regions and evaluates according to the corresponding rules in each region.

[0094] Among them, the ontology structure features include the shape of the vehicle head, the contour of the vehicle body, color, license plate number, etc. The types and values of the above features will be used as the basis for matching the abnormal judgment strategy. From the preset strategy library, according to the types and values of the ontology structure features, the abnormal judgment strategy and the corresponding weight coefficient that best match the current tractor are matched. It can be considered that the preset strategy library stores abnormal judgment strategies for multiple tractors, and each strategy corresponds to a specific range of ontology structure features and weight coefficients.

[0095] The deduction items refer to the situations that may indicate abnormalities, such as irregular cargo shape, size exceeding the range, position deviation, etc. The weight coefficients will be adjusted according to the changes in the ontology structure features and the load area features to reflect the importance of different features in the current situation.

[0096] By integrating all deduction items and weight coefficients, the abnormal detection score is calculated. This score will be used as the monitoring status information to determine whether the load is abnormal.

[0097] Thus, through the rule engine, according to the predefined rules and logics, the extracted features are evaluated and scored. The flexibility and interpretability of the rule engine enable the abnormal judgment strategy to be easily adjusted and optimized, and can quickly adapt to different application scenarios and requirements.

[0098] Among them, the formula corresponding to the detection scoring model can be:

[0099]

[0100] Among them, S is the abnormal detection score, representing the final monitoring status information for determining whether the load is abnormal. S0 is the initial score, representing the basic score in the case of no abnormalities found, usually set to a fixed value, such as 100 points. D i is the i-th deduction item, representing the deduction value for the i-th possible abnormal situation, which is determined according to the actual detection results. If this abnormal situation occurs, D i takes the corresponding deduction value, otherwise it is 0. Wi is the i-th weight coefficient, representing the importance of the i-th deduction item in the overall scoring, which is dynamically adjusted according to the ontology structure features and the load area features. n is the total number of deduction items, representing that there are n possible abnormal situations to be evaluated.

[0101] In some embodiments, considering that the transportation tasks are pre - determined according to the production plan and order requirements, the transportation route of each tractor is usually fixed. Therefore, the transportation tasks performed by the same tractor over a period of time are repetitive and unchanged. It can be considered that the pre - set policy library stores the anomaly judgment policies corresponding to multiple tractors (in terms of their body structure characteristics). The advantage is that in practical applications, when the production plan and order requirements change, the data in the pre - set policy library can be uniformly updated to adjust the anomaly judgment policies of each tractor in real - time.

[0102] In some embodiments, the first threshold is: T1 = S0*(1 - α). α is the risk coefficient in the case of severe anomalies, for example, 0.3. The second threshold is the product of the first threshold and the threshold correlation coefficient. The threshold correlation coefficient is greater than 1, for example, 1.03, 1.05.

[0103] In some embodiments, obtaining the predicted loading area information of the trailer according to the pre - set loading information and the current position information of the target towing device includes:

[0104] Obtain the loading plan of the target towing device from the pre - set loading information. The loading plan includes the loading sequence and placement requirements of the goods in different loading areas;

[0105] According to the loading plan and the current position information, take the previous loading area reached by the target towing device as the target loading area;

[0106] According to the goods information of the target loading area in the loading plan, predict the types, quantities, and arrangement patterns of the goods in the load - carrying area of the trailer to generate the predicted loading area information.

[0107] As an example, extract the loading plan of the target towing device from the pre - set loading information. The loading plan details the loading sequence and placement requirements of the goods in different loading areas, including the numbers and locations of the loading areas, as well as the types, quantities, and arrangement patterns of the goods in each loading area, and the loading sequence. Obtain the current position information of the target towing device through GPS. The current position information includes the specific coordinates and travel trajectory of the towing device in the travel route.

[0108] Pre - define the geographical scope or coordinates of each loading area, compare the current position (travel trajectory) of the towing device with the positions of each loading area, and find the matching previous loading area (i.e., the loading area where the goods were last loaded).

[0109] Predict the types, quantities, and arrangement patterns of goods in the loading area of the trailer based on the goods information of the previous loading area in the loading plan. Extract the goods information including the types, quantities, and arrangement patterns of the previous loading area from the loading plan, and then generate loading area prediction information to describe the expected loading situation of the trailer in this area. Integrate the predicted types, quantities, and arrangement patterns of goods into the loading area prediction information.

[0110] Thus, by obtaining the current position information of the towing device in real time, it is possible to dynamically determine the loading area reached by the towing device, and generate the loading area prediction information only when the anomaly detection score is between the first threshold and the second threshold, reducing data processing. By reducing the amount of data processing, computing resources can be utilized more efficiently, reducing energy consumption. It is possible to predict the loading and unloading situation of the trailer in advance before entering the next loading area, helping users to discover potential loading problems in advance, reminding users to instruct the loading personnel to check and handle problems in the next loading area, and reducing transportation accidents caused by improper loading.

[0111] In some embodiments, determining whether the trailer is loaded incorrectly based on the loading area prediction information and the first image information includes:

[0112] Obtain the loading similarity between the loading area prediction information and the first image information. When the value of the loading similarity is less than the preset similarity, it is considered that the loading is incorrect; otherwise, it is considered that the loading is normal, start counting and increment the count by one. When the count is greater than the preset number, clear the count and send second threshold anomaly information to the user device, where the second threshold anomaly information is used to prompt to reduce the value of the second threshold. The preset number is, for example, 1, 2, 3.

[0113] Among them, the way to obtain the loading similarity can be to calculate the similarity by comparing the consistency of the image information of the loading area prediction information and the first image information at the semantic level, that is, to perform semantic annotation on the objects in the image, and then compare the consistency of the prediction information and the actual image at the semantic level.

[0114] Thus, when the count exceeds the preset number, prompting the user to reduce the second threshold helps the user to adjust the parameters according to the actual situation, improving the adaptability and accuracy of the method. Triggering an alarm only when the similarity is lower than the threshold reduces unnecessary data processing and improves the execution efficiency of the method.

[0115] As an example, take 0.04 for the case where the goods size exceeds the range, and take 0.02 for the case where the goods position deviates. The weight coefficient W i can be dynamically adjusted according to the actual situation. For example, if a certain tractor is more sensitive to the deviation of the goods position (more likely to have the deviation of the goods position), the weight coefficient of this deduction item can be increased.

[0116] The image segmentation model is used to extract the target device body structure features of the tractor head and the load area features of the trailer from the first image information, and it can be a convolutional neural network architecture based on deep learning with a U-Net structure. During training, a large amount of image data of tractors in different scenarios is collected, including normal driving, cargo dropping, cargo shifting, etc., as training data. In this embodiment, parameters such as the learning rate (e.g., 0.001), the number of iterations (e.g., 500), and the batch size (e.g., 32) are not limited.

[0117] As an example, a monitoring device warning method based on an improved image recognition model is provided, which is applied to a warning system. The warning system includes a monitoring device for obtaining effective image information of a target towing device. The monitoring device includes a plurality of first collection devices arranged on the running route of the target towing device and a plurality of second collection devices arranged in a plurality of cargo loading areas on the traveling route. The method includes:

[0118] Using the first collection device to obtain the first image information of the target towing device on the traveling route, and using the second collection device to obtain the second image information from the electronic two-dimensional code screen arranged at the tail or both sides of the trailer. The second image information is two-dimensional code image information; the target towing device includes a tractor and a trailer;

[0119] According to the effective image information, the image segmentation model extracts the body structure features of the tractor and the load area features of the trailer from the first image information and uses them as the target features in the effective image information; according to the type and value of the body structure features, the rule engine matches the anomaly judgment strategy and the weight coefficient from the preset policy library; inputs the load area features into the anomaly judgment strategy, and calculates the anomaly detection score according to the deduction items and the weight coefficient; uses the anomaly detection score as the monitoring status information;

[0120] When the anomaly detection score is not higher than the first threshold, it is considered that the load of the target towing device is abnormal, and a first warning message is generated and sent to the user device; when the anomaly detection score is between the first threshold and the second threshold, the preset loading information is obtained according to the second image information; the second threshold is greater than the first threshold; the loading plan of the target towing device is obtained from the preset loading information, and the loading plan includes the loading sequence and placement requirements of the goods in different loading areas; according to the loading plan and the current location information, the previous loading area reached by the target towing device is taken as the target loading area; according to the goods information of the target loading area in the loading plan, the types, quantities and arrangement methods of the goods in the load area of the trailer are predicted, and a loading area prediction information is generated; the loading similarity between the loading area prediction information and the first image information is obtained, and when the value of the loading similarity is less than the preset similarity, it is considered that the loading is incorrect. When it is judged that the trailer is loaded incorrectly, a second warning message is generated and sent to the user device; otherwise, it is considered that the loading is normal, the statistics are started and the statistical count is incremented by one. When the statistical count is greater than the preset count, the statistical count is cleared, and a second threshold anomaly information is sent to the user device. The second threshold anomaly information is used to prompt to reduce the value of the second threshold.

[0121] Among them, the improved image recognition model includes an image segmentation model based on a deep neural network and a detection scoring model driven by a rule engine; the image segmentation model is used to extract the target features in the effective image information, and the detection scoring model is used to obtain the monitoring status information according to the target features.

[0122] See Figure 6 , as another example, a monitoring device warning method based on an improved image recognition model is provided, including:

[0123] Obtain the first image information, and use the first acquisition device to obtain the first image information of the target towing device on the travel route; the first acquisition device is a monitoring camera array;

[0124] Obtain the second image information, and use the second acquisition device to obtain the second image information from the electronic two-dimensional code screen set at the tail or both sides of the trailer; the second image information is two-dimensional code image information; the second acquisition device is a two-dimensional code scanning device set at the top of the loading area;

[0125] Obtain the body structure feature and obtain the load area feature. Through the image segmentation model, the body structure feature of the tractor and the load area feature of the trailer are extracted according to the first image information;

[0126] Obtain the anomaly judgment strategy and the weight coefficient. According to the type and value of the body structure feature, the anomaly judgment strategy and the weight coefficient are matched from the preset strategy library through the rule engine; the preset strategy library can be stored in relational databases such as MySQL and PostgreSQL;

[0127] Calculate the abnormal detection score. Input the load area features into the abnormal judgment strategy, and calculate the abnormal detection score according to the deduction items and weight coefficients. Use the abnormal detection score as the monitoring status information.

[0128] Compare the score with the first threshold. When the abnormal detection score is not higher than the first threshold, it is considered that the load of the target towing device is abnormal, and a first warning message is generated and sent to the user device. When it is higher, determine whether it is between the first threshold and the second threshold.

[0129] If so, obtain the preset loading information according to the second image information. If not, do not perform any actions.

[0130] Obtain the loading plan. Obtain the loading plan of the target towing device from the preset loading information. The loading plan includes the loading sequence and placement requirements of the goods in different loading areas.

[0131] Obtain the loading area prediction information. According to the goods information in the target loading area in the loading plan, predict the types, quantities, and arrangement methods of the goods in the load area of the trailer to obtain the loading area prediction information. Among them, according to the loading plan and the current location information, the previous loading area reached by the target towing device can be used as the target loading area.

[0132] Obtain the loading similarity. The loading similarity is the similarity between the loading area prediction information and the first image information.

[0133] Judge that the value of the loading similarity is less than the preset similarity. If it is less, generate a second warning message and send it to the user device.

[0134] If not, start counting and increment the count by one.

[0135] When the count is greater than the preset number, if so, generate a second threshold abnormal information and send it to the user device. If not, do nothing.

[0136] The technical solution provided by this example first uses the first acquisition device and the second acquisition device to obtain effective image information; then extracts the body structure features of the tractor and the load area features of the trailer from the first image information through an image segmentation model as target features; then matches the corresponding anomaly judgment strategy and weight coefficient from the preset strategy library according to the type and value of the body structure features; then inputs the load area features into this anomaly judgment strategy, calculates the anomaly detection score through the detection scoring model according to the deduction items and the weight coefficient, and uses it as the monitoring status information; finally, when the anomaly detection score is not higher than the first threshold, it is considered that the load is abnormal, generates the first warning information and sends it to the user device; when the score is between the first threshold and the second threshold, the preset loading information is obtained by using the second image information, and combined with the current position information of the target traction device, the predicted loading area information of the trailer is obtained, and then based on this predicted information and the first image information, it is judged whether the trailer is loaded incorrectly. If so, the second warning information is generated and sent to the user device.

[0137] Thus, by setting multiple acquisition devices on the travel route of the traction device and the cargo loading area, the comprehensive monitoring of the traction device at different positions and stages is realized, improving the comprehensiveness and accuracy of the monitoring. Using an image segmentation model based on a deep neural network can accurately extract the feature information of the tractor and the trailer, providing a more accurate data basis for subsequent anomaly detection and improving the accuracy of anomaly detection. Through the weight coefficient, the anomaly detection score can more flexibly reflect the importance of different features in the current situation, improving the accuracy and adaptability of the score. By setting two thresholds and generating warning information of different levels according to the anomaly detection score, the dual warning mechanism can more flexibly respond to different degrees of abnormal situations and timely remind users to take measures. By combining the preset loading information, pick-up information and current position information, the predicted loading area information of the trailer is obtained, and by comparing the predicted information with the actual image information, it is intelligently judged whether the trailer is loaded incorrectly, providing a more accurate decision-making basis for users. Through the real-time monitoring and warning of the load status of the traction device, abnormal situations can be discovered and processed in time, reducing transportation accidents caused by cargo dropping, improper loading, etc., and improving the safety of the transportation process.

[0138] Device embodiment.

[0139] The embodiment of the present application also provides a monitoring device warning device based on an improved image recognition model. Its specific implementation is consistent with the implementation and the achieved technical effects described in the above method embodiments, and some contents will not be elaborated here. The monitoring device warning device, the improved image recognition model, and the preset policy library can be deployed on a cloud server, or on a local server, or deployed through an edge cloud collaboration architecture. Under the edge cloud collaboration architecture, the edge device can be an industrial computer or a high-performance embedded device, and the cloud server can be a virtual machine provided by a cloud service provider, enabling preliminary image processing and anomaly detection on the edge device, and transmitting key data to the cloud server for further analysis and decision-making, that is, the edge device is responsible for the preliminary processing of real-time data to ensure low-latency response; the cloud server is responsible for the in-depth analysis of complex data to provide more accurate decision support.

[0140] See Figure 4 , Figure 4 which is a schematic diagram of the modules of a monitoring device warning device provided by an embodiment of the present application. The device includes:

[0141] An information acquisition module, configured to use the monitoring device to acquire valid image information of the target traction device;

[0142] A model processing module, configured to use the improved image recognition model to obtain the monitoring status information of the target traction device according to the valid image information;

[0143] An information sending module, configured to generate a warning message and send it to the user device when the monitoring status information indicates that the load of the target traction device appears abnormal;

[0144] Among them, the improved image recognition model includes an image segmentation model and a detection scoring model; the image segmentation model is used to extract the target features in the valid image information, and the detection scoring model is used to obtain the monitoring status information according to the target features.

[0145] In some embodiments, the target traction device includes a tractor and a trailer. The model processing module includes:

[0146] A feature extraction unit, configured to extract the body structure features of the tractor and the load area features of the trailer from the valid image information through the image segmentation model and use them as the target features in the valid image information;

[0147] A scoring acquisition unit, configured to obtain an anomaly judgment strategy according to the body structure features; according to the anomaly judgment strategy and the load area features, use the detection scoring model to obtain an anomaly detection score and use it as the monitoring status information;

[0148] The information sending module includes: a first abnormality determination unit, which is used to consider that the load of the target towing device is abnormal when the abnormality detection score is not higher than the first threshold, generate a first warning message and send it to the user device.

[0149] In some embodiments, the monitoring device includes a plurality of first collection devices arranged on the running route of the target towing device and a plurality of second collection devices arranged in a plurality of cargo loading areas on the traveling route; the information acquisition module includes:

[0150] A first information acquisition unit, which is used to acquire first image information of the target towing device on the traveling route by using the first collection device;

[0151] A second information acquisition unit, which is used to acquire second image information of the target towing device in the cargo loading area by using the second collection device;

[0152] The information sending module further includes:

[0153] A preset loading information acquisition unit, which is used to acquire preset loading information according to the second image information when the abnormality detection score is between the first threshold and the second threshold; the second threshold is greater than the first threshold;

[0154] A loading area prediction information acquisition unit, which is used to acquire loading area prediction information of the trailer according to the preset loading information and the current position information of the target towing device;

[0155] A second abnormality determination unit, which is used to judge whether the trailer is loaded incorrectly based on the loading area prediction information and the first image information; when it is judged that the trailer is loaded incorrectly, generate a second warning message and send it to the user device.

[0156] In some embodiments, the second image information is two-dimensional code image information; the method for acquiring the second image information of the target towing device in the cargo loading area includes: using the second collection device to acquire the second image information from an electronic two-dimensional code screen arranged at the tail or both sides of the trailer.

[0157] In some embodiments, the score acquisition unit includes:

[0158] A matching subunit, which is used to match an abnormality determination strategy and a weight coefficient from a preset policy library through a rule engine according to the type and value of the body structure characteristics;

[0159] A scoring subunit, which is used to input the load area characteristics into the abnormality determination strategy, calculate the abnormality detection score according to the deduction items and the weight coefficient; use the abnormality detection score as the monitoring status information.

[0160] In some embodiments, the loading area prediction information acquisition unit includes:

[0161] A planned acquisition subunit, configured to acquire a loading plan of a target towing device from preset loading information, where the loading plan includes the loading sequence and placement requirements of goods in different loading areas;

[0162] An area confirmation subunit, configured to use the loading plan and current location information to determine the previous loading area reached by the target towing device as the target loading area;

[0163] An information generation subunit, configured to predict the types, quantities, and arrangement modes of goods in the load area of the trailer according to the goods information of the target loading area in the loading plan, and generate loading area prediction information.

[0164] In some embodiments, the second anomaly determination unit is configured to obtain the loading similarity between the loading area prediction information and the first image information. When the value of the loading similarity is less than a preset similarity, it is considered that the loading is incorrect; otherwise, it is considered that the loading is normal, start counting and increment the count by one. When the count is greater than a preset number, clear the count and send second threshold anomaly information to the user device, where the second threshold anomaly information is used to prompt to reduce the value of the second threshold.

[0165] System embodiments.

[0166] See Figure 5 , Figure 5 is a schematic diagram of the modules of an early warning system provided by an embodiment of the present application.

[0167] An embodiment of the present application provides an early warning system, and its specific implementation manners are consistent with the implementation manners and achieved technical effects recorded in the above method implementation manners, and some contents will not be elaborated.

[0168] The system includes the monitoring device and early warning device described in the device embodiments, and a monitoring device for acquiring valid image information of the target towing device.

[0169] In some embodiments, the monitoring device includes a plurality of first collection devices arranged on the running route of the target towing device and a plurality of second collection devices arranged in a plurality of goods loading areas on the traveling route.

[0170] It should be noted that, in the embodiments of the present application, "at least one" means one or more, and "a plurality" means two or more. "And / or" describes the relationship between associated objects and indicates that there can be three relationships. For example, A and / or B can mean: A exists alone, A and B exist simultaneously, or B exists alone, where A and B can be singular or plural. The character " / " generally indicates an "or" relationship between the associated objects before and after. "At least one (item)" or a similar expression thereof refers to any combination of these items, including any combination of a single item or plural items. For example, at least one (item) of a, b, or c can mean: a, b, c, a and b, a and c, b and c, or a, b, and c, where a, b, and c can be single or multiple. It should be noted that "at least one (item)" can also be interpreted as "one item or multiple items".

[0171] The terms "first", "second", etc. in the description, claims, and the above-mentioned drawings of the present application are configured to distinguish similar objects and do not necessarily need to be configured to describe a specific order or sequence. It should be understood that the data used in this way can be interchanged under appropriate circumstances so that the embodiments of the present application described here can be implemented in an order other than those illustrated or described here. In addition, the terms "comprising" and "having" and any variations thereof are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or device that includes a series of steps or units does not necessarily have to be limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or are inherent to these processes, methods, products, or devices.

[0172] The present application is described from the perspectives of usage purpose, effectiveness, progress, and novelty, etc., and has met the functional improvement and usage requirements emphasized by the Patent Law. The above description and the accompanying drawings of the present application are only preferred embodiments of the present application and do not limit the present application thereto. Therefore, all those that are similar or identical to the structure, device, features, etc. of the present application, that is, all equivalent replacements or modifications made according to the scope of the patent application of the present application, shall fall within the scope of the patent application protection of the present application.

Claims

1. A monitoring device warning method based on an improved image recognition model, characterized in that, The method includes the steps of: S1, obtaining effective image information of a target towing device by using a monitoring device; S2, obtaining monitoring status information of the target towing device by using an improved image recognition model according to the effective image information; S3, when the monitoring status information indicates that the load of the target towing device is abnormal, generating a warning message and sending it to a user device; wherein, the improved image recognition model includes an image segmentation model based on a deep neural network and a detection scoring model driven by a rule engine; the image segmentation model is used to extract target features in the effective image information, and the detection scoring model is used to obtain monitoring status information according to the target features.

2. The warning method of the monitoring device according to claim 1, characterized in that The target towing device includes a tractor and a trailer. Step S2 includes: extracting the body structure features of the tractor and the load area features of the trailer as the target features in the effective image information through the image segmentation model according to the effective image information; obtaining an abnormal judgment strategy according to the body structure features; obtaining an abnormal detection score by using the detection scoring model according to the abnormal judgment strategy and the load area features, and using it as the monitoring status information; Step S3 includes: when the abnormal detection score is not higher than a first threshold, it is considered that the load of the target towing device is abnormal, generating a first warning message and sending it to the user device.

3. The warning method for a monitoring device according to claim 2, wherein The monitoring device includes a plurality of first collection devices arranged on the running route of the target towing device and a plurality of second collection devices arranged in a plurality of cargo loading areas on the traveling route; Step S1 includes: obtaining first image information of the target towing device on the traveling route by using the first collection device, and obtaining second image information of the target towing device in the cargo loading area by using the second collection device; Step S3 further includes: when the abnormal detection score is between a first threshold and a second threshold, obtaining preset loading information according to the second image information; the second threshold is greater than the first threshold; obtaining predicted loading area information of the trailer according to the preset loading information and the current position information of the target towing device; judging whether the trailer is loaded incorrectly based on the predicted loading area information and the first image information; when it is judged that the trailer is loaded incorrectly, generating a second warning message and sending it to the user device.

4. The warning method for a monitoring device according to claim 3, wherein The second image information is two-dimensional code image information; the method for obtaining the second image information of the target towing device in the cargo loading area includes: obtaining the second image information from an electronic two-dimensional code screen arranged at the tail or both sides of the trailer by using the second collection device.

5. The warning method for a monitoring device according to claim 3, wherein The obtaining the predicted loading area information of the trailer according to the preset loading information and the current position information of the target towing device includes: obtaining a loading plan of the target towing device from the preset loading information, where the loading plan includes the loading sequence and placement requirements of the goods in different loading areas; taking the previous loading area reached by the target towing device as the target loading area according to the loading plan and the current position information; predicting the types, quantities and arrangement modes of the goods in the load area of the trailer according to the goods information of the target loading area in the loading plan, and generating predicted loading area information.

6. The warning method for a monitoring device according to claim 3, wherein, Determining whether the trailer is loaded incorrectly based on the predicted loading area information and the first image information includes: Obtaining the loading similarity between the predicted loading area information and the first image information. When the value of the loading similarity is less than a preset similarity, it is considered that the loading is incorrect; otherwise, it is considered that the loading is normal, start counting and increment the count by one. When the count is greater than the preset number, clear the count and send second threshold abnormal information to the user device, where the second threshold abnormal information is used to prompt to reduce the value of the second threshold.

7. The monitoring device warning method according to claim 2, wherein, Obtaining an abnormal judgment strategy according to the ontology structure characteristics; According to the abnormal judgment strategy and the load area characteristics, using the detection scoring model to obtain an abnormal detection score and use it as monitoring status information, including: According to the type and value of the ontology structure characteristics, match the abnormal judgment strategy and the weight coefficient from the preset strategy library through the rule engine; Input the load area characteristics into the abnormal judgment strategy, calculate the abnormal detection score according to the deduction items and the weight coefficient; use the abnormal detection score as the monitoring status information.

8. A monitoring device warning device based on an improved image recognition model, characterized in that, Including: An information acquisition module, configured to use a monitoring device to acquire effective image information of a target towing device; A model processing module, configured to use an improved image recognition model to obtain monitoring status information of the target towing device according to the effective image information; An information sending module, configured to generate a warning message and send it to the user device when the monitoring status information indicates that the load of the target towing device is abnormal; Wherein, the improved image recognition model includes an image segmentation model and a detection scoring model; the image segmentation model is used to extract target features in the effective image information, and the detection scoring model is used to obtain monitoring status information according to the target features.

9. An early warning system, characterized in that, Including the monitoring device warning device according to claim 8, and a monitoring device for acquiring effective image information of a target towing device.

10. The warning system according to claim 9, characterized in that, The monitoring device includes a plurality of first collection devices arranged on the running route of the target towing device and a plurality of second collection devices arranged in a plurality of cargo loading areas on the traveling route.

Images