Industrial vehicle cargo status dynamic recognition system based on visual end-side large model

Abstract

The present invention relates to the field of warehousing and distribution technology, and specifically to a dynamic identification system for the cargo status of industrial vehicles based on a visual end-side large model. The system can achieve the following steps through the mutual cooperation between multiple modules: through ADAS, obtain the speed data, vibration data and target cargo image of the target industrial vehicle in the current time period; divide the current time period into equal parts, and determine the abnormal speed change degree and vehicle operation status index corresponding to each current sub-time period; perform a fusion operation on all target cargo images in the current time period; according to the current fused image, obtain the current predicted cargo quantity and the current predicted cargo completeness through the end-side large model that has been pre-trained; determine the current cargo abnormality index; and perform status identification on the cargo of the target industrial vehicle in the current time period. The present invention realizes the identification of the cargo status of industrial vehicles and improves the accuracy of identifying the cargo status of industrial vehicles.

Description

Technical Field

[0001] The present invention relates to the technical field of warehousing and distribution, and in particular to a dynamic identification system for cargo status of industrial vehicles based on a large visual terminal model. Background Art

[0002] In the process of warehousing and distribution, it is very important to identify the state of goods in industrial vehicles used for transportation. At present, the method commonly used to identify the state of objects is to analyze the collected object images to achieve object state recognition.

[0003] However, when analyzing the collected vehicle cargo images to realize industrial vehicle cargo status recognition, the following technical problems often occur: Since the goods in industrial vehicles often block each other, some of the goods may not be clearly displayed in the image. Therefore, if only the vehicle cargo image is considered, the accuracy of identifying the status of the industrial vehicle cargo may be poor, resulting in poor warehousing and distribution effects. Summary of the invention

[0004] In order to solve the technical problem of poor accuracy in identifying the status of cargo on industrial vehicles, the present invention proposes a dynamic identification system for the status of cargo on industrial vehicles based on a large model on the visual end side.

[0005] In a first aspect, the present invention provides a system for dynamically identifying the state of cargo on an industrial vehicle based on a large model on a visual end side, the system comprising: A data acquisition module is used to acquire speed data, vibration data and target cargo images of a target industrial vehicle at each data acquisition moment in a current time period through ADAS; The equal division determination module is used to divide the current time period into equal parts to obtain the current sub-time period, and determine the abnormal speed change degree corresponding to each current sub-time period according to all speed data in each current sub-time period and its corresponding Z score; A vehicle operation status indicator determination module is used to determine the vehicle operation status indicator corresponding to each current sub-time period according to the abnormal speed change degree corresponding to each current sub-time period and all vibration data in each current sub-time period; An image fusion operation module is used to perform a fusion operation on all target cargo images within the current time period to obtain a current fused image; The indicator prediction module is used to obtain the current predicted cargo quantity and the current predicted cargo completeness based on the current fused image through the pre-trained large model on the end side; A current cargo abnormality index determination module, used to determine the current cargo abnormality index according to the difference between the current predicted cargo quantity and the current expected cargo quantity obtained in advance, and the current predicted cargo completeness; The state recognition module is used to identify the state of the cargo of the target industrial vehicle in the current time period according to the current vehicle operation state representative indicators corresponding to all current sub-time periods and the current cargo abnormality indicators.

[0006] In combination with the first aspect, in a possible implementation, determining the degree of abnormal speed change corresponding to each current sub-time period according to all speed data in each current sub-time period and its corresponding Z score includes: The average value of all speed data in each current sub-time period is determined as the speed representative factor corresponding to each current sub-time period; The accumulated value of the absolute values ​​of the Z scores corresponding to all the speed data in each current sub-time period is determined as the initial speed change factor corresponding to each current sub-time period; The speed abnormal variation degree corresponding to each current sub-time period is determined according to the speed representative factor and the initial speed variation factor corresponding to each current sub-time period, wherein both the speed representative factor and the initial speed variation factor are positively correlated with the speed abnormal variation degree.

[0007] In combination with the first aspect, in a possible implementation, determining the abnormal speed change degree corresponding to each current sub-time period according to the speed representative factor and the initial speed change factor corresponding to each current sub-time period includes: The product of the speed representative factor corresponding to each current sub-time period and the initial speed variation factor is determined as the speed abnormal variation degree corresponding to each current sub-time period.

[0008] In combination with the first aspect above, in a possible implementation, determining the vehicle running status index corresponding to each current sub-time period according to the abnormal speed change degree corresponding to each current sub-time period and all vibration data in each current sub-time period includes: The mean value of all vibration data in each current sub-time period is determined as the vibration representative factor corresponding to each current sub-time period; According to the abnormal speed variation degree and the vibration representative factor corresponding to each current sub-time period, the vehicle running state index corresponding to each current sub-time period is determined.

[0009] In combination with the first aspect, in a possible implementation, determining the vehicle running state index corresponding to each current sub-time period according to the abnormal speed change degree and the vibration representative factor corresponding to each current sub-time period includes: The product of the abnormal speed variation degree and the vibration representative factor corresponding to each current sub-time period is normalized to obtain the vehicle operation status index corresponding to each current sub-time period.

[0010] In combination with the first aspect above, in a possible implementation, determining the current cargo abnormality index according to the difference between the current predicted cargo quantity and the currently expected cargo quantity acquired in advance, and the current predicted cargo completeness, includes: The absolute value of the difference between the current forecasted cargo quantity and the current expected cargo quantity is determined as the current cargo quantity difference; Determine the current cargo abnormality index based on the current cargo quantity difference and the current predicted cargo completeness.

[0011] In combination with the first aspect above, in a possible implementation manner, determining the current cargo abnormality index according to the current cargo quantity difference and the current predicted cargo completeness includes: According to the current predicted cargo integrity, the current cargo damage rate is determined, wherein the current predicted cargo integrity and the current cargo damage rate are negatively correlated; Determine the current cargo abnormality index based on the difference between the current cargo damage rate and the current cargo quantity.

[0012] In combination with the first aspect above, in a possible implementation, determining the current cargo abnormality index according to the difference between the current cargo damage rate and the current cargo quantity includes: The weighted sum of the current cargo damage rate and the current cargo quantity difference is determined as the current cargo abnormality index.

[0013] In combination with the first aspect, in a possible implementation, the identifying the status of the cargo of the target industrial vehicle in the current time period according to the vehicle operation status indicators corresponding to all current sub-time periods and the current cargo abnormality indicator includes: The average of the vehicle operation status indicators corresponding to all current sub-time periods is determined as the representative indicator of the current vehicle operation status; Normalizing the product of the current vehicle running state representative index and the current cargo abnormality index to obtain a current cargo state index; If the current cargo status indicator is greater than a preset warning threshold, information indicating that the cargo status of the industrial vehicle is abnormal is sent.

[0014] In combination with the first aspect above, in a possible implementation, the training process of the large model on the client side includes: Build a large end-side model; Obtain historical cargo images of different industrial vehicles at each data collection moment in different historical time periods, and obtain the actual integrity and actual quantity of cargo of different industrial vehicles in different historical time periods; Perform a fusion operation on all historical cargo images of each industrial vehicle in each historical time period to obtain a historical fused image; The constructed end-side large model is trained using historical fusion images as training samples and the actual integrity and actual quantity of goods corresponding to the historical fusion images as training labels.

[0015] In a second aspect, the present invention provides a method for dynamically identifying the state of cargo on an industrial vehicle based on a large visual end-side model, which is implemented by a dynamic identification system for the state of cargo on an industrial vehicle based on a large visual end-side model. The method comprises: Through ADAS, the speed data, vibration data and target cargo image of the target industrial vehicle at each data collection moment in the current time period are obtained; The current time period is divided equally into current sub-time periods, and the degree of abnormal speed change corresponding to each current sub-time period is determined based on all speed data in each current sub-time period and its corresponding Z score; Determine the vehicle running status index corresponding to each current sub-time period according to the abnormal speed change degree corresponding to each current sub-time period and all vibration data in each current sub-time period; Perform a fusion operation on all target cargo images within the current time period to obtain a current fused image; Based on the current fused image, the current predicted cargo quantity and the current predicted cargo completeness are obtained through the pre-trained large model on the end side. Determine the current cargo abnormality indicator based on the difference between the current predicted cargo quantity and the current expected cargo quantity obtained in advance, and the current predicted cargo completeness; According to the vehicle operation status indicators corresponding to all current sub-time periods and the current cargo abnormality indicators, the status of the cargo of the target industrial vehicle in the current time period is identified.

[0016] In a third aspect, a server is provided, comprising a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, so that the device executes the above-mentioned method for dynamic identification of industrial vehicle cargo status based on the visual end-side large model.

[0017] In a fourth aspect, a computer program product is provided, which includes: a computer program code, which, when executed on a computer, enables the computer to execute the above-mentioned method for dynamic identification of the state of cargo of an industrial vehicle based on a large model of a visual end side.

[0018] In a fifth aspect, a computer-readable storage medium is provided, which stores a computer program code. When the computer program code runs on a computer, the computer executes the above-mentioned method for dynamic identification of industrial vehicle cargo status based on the visual end-side large model.

[0019] The present invention has the following beneficial effects: The industrial vehicle cargo status dynamic recognition system based on the visual end-side large model of the present invention realizes the industrial vehicle cargo status recognition, solves the technical problem of poor accuracy in recognizing the industrial vehicle cargo status, and improves the accuracy of recognizing the industrial vehicle cargo status. Compared with only analyzing the collected vehicle cargo images to realize industrial vehicle cargo status recognition, the present invention not only considers the target cargo image containing vehicle cargo information, but also comprehensively considers multiple other features related to the vehicle cargo status, such as speed data, vibration data, abnormal speed change degree, vehicle operation status index, current predicted cargo quantity and current predicted cargo completeness, etc., thereby improving the accuracy of recognizing the industrial vehicle cargo status, and further improving the warehousing and distribution effect. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] In order to more clearly illustrate the technical solutions and advantages in the embodiments of the present invention or the prior art, the drawings required for use in the embodiments or the prior art descriptions are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0021] Figure 1 It is a structural schematic diagram of the industrial vehicle cargo state dynamic recognition system based on the visual end-side large model of the present invention; Figure 2 It is a flow chart of a method for dynamically identifying the state of cargo on an industrial vehicle based on a large visual end-side model of the present invention; Figure 3 The figure is a schematic diagram of the structure of a computer device of the present invention. DETAILED DESCRIPTION

[0022] In order to further explain the technical means and effects adopted by the present invention to achieve the predetermined invention purpose, the specific implementation methods, structures, features and effects of the technical solutions proposed by the present invention are described in detail below in conjunction with the accompanying drawings and preferred embodiments. In the following description, different "one embodiment" or "another embodiment" does not necessarily refer to the same embodiment. In addition, specific features, structures or characteristics in one or more embodiments may be combined in any suitable form.

[0023] Unless defined otherwise, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs.

[0024] refer to Figure 1 , shows a schematic diagram of the structure of the industrial vehicle cargo state dynamic recognition system based on the visual end-side large model according to the present invention. The industrial vehicle cargo state dynamic recognition system based on the visual end-side large model includes: The data acquisition module 101 is used to acquire the speed data, vibration data and target cargo image of the target industrial vehicle at each data acquisition moment in the current time period through ADAS.

[0025] Among them, ADAS (Advanced Driving Assistance System) is a system that uses multiple sensors installed on the vehicle (such as millimeter wave radar, laser radar, single / binocular camera and satellite navigation) to sense the surrounding environment, collect data, identify, detect and track static and dynamic objects, and perform calculations and analysis in combination with navigation map data. The target industrial vehicle may be a vehicle to be identified in the state of goods during transportation. The compartment used to place goods in the industrial vehicle may be closed. The transportation process may be a process in which the vehicle needs to travel between two adjacent transportation sites. The transportation site may be a site for updating goods for the vehicle. The goods update may include adding or reducing goods to the vehicle. The current time period may be a preset time period with the current moment as the end moment during the current transportation process. The current transportation process may be the transportation process that the target vehicle is currently carrying out. The preset time period may be a pre-set time period, and its corresponding duration may be 5 minutes. The data collection moment may be the moment when data collection is performed. The duration between adjacent data collection moments may be less than the duration corresponding to the preset time period, for example, the duration between adjacent data collection moments may be 1 second. The speed data may represent the driving speed of the target industrial vehicle. The vibration data may represent the vibration of the target vehicle. The target cargo image may be an image collected during the current time period and used to represent the cargo placement inside the target industrial vehicle.

[0026] As an example, if the duration between adjacent data collection moments is 0.2 seconds, then in the current time period, the speed sensor included in the ADAS can be used to collect the driving speed of the target industrial vehicle every 0.2 seconds as speed data, and the vibration sensor included in the ADAS can be used to collect the vibration value of the target industrial vehicle every 0.2 seconds as vibration data, and the camera included in the ADAS can be used to collect the cargo placement image in the compartment of the target industrial vehicle every 0.2 seconds as the target cargo image.

[0027] It should be noted that in actual situations, the types of goods that need to be delivered are often different, and their corresponding sizes are often different, which may cause the goods in the industrial vehicle to be placed unevenly. Therefore, during the driving process of the vehicle, if the vehicle is driving too fast or the vibration is too large, it may cause some of the goods in the vehicle to tilt or fall to a lower place, resulting in abnormal state of the goods in the industrial vehicle. And the target cargo image can represent the cargo placement information to a certain extent, so obtaining the speed data, vibration data and target cargo image of the target industrial vehicle at each data collection time in the current time period can facilitate the subsequent analysis of the cargo state in the industrial vehicle.

[0028] The equal division determination module 102 is used to divide the current time period into equal parts to obtain current sub-time periods, and determine the abnormal speed variation degree corresponding to each current sub-time period based on all speed data in each current sub-time period and its corresponding Z score.

[0029] As an example, the equal division determination module 102 may be used to implement the following steps: The first step is to divide the current time period into equal parts to obtain the current sub-time period.

[0030] For example, the current time period may be equally divided into 10 sub-time periods, and each of the equally divided sub-time periods is recorded as the current sub-time period.

[0031] In the second step, the mean of all speed data in each current sub-time period is determined as the speed representative factor corresponding to each current sub-time period.

[0032] In the third step, the accumulated value of the absolute value of the Z score corresponding to all the speed data in each current sub-time period is determined as the initial speed change factor corresponding to each current sub-time period.

[0033] The Z-score, also known as the standard score, is calculated by dividing the difference between an observation or data point and the mean by the standard deviation.

[0034] For example, the formula for determining the initial speed change factor corresponding to the current sub-time period may be: ;in, is the initial speed change factor corresponding to the kth current sub-time period. k is the sequence number of the current sub-time period within the current time period. is the number of data collection moments in the kth current sub-time period. a is the sequence number of the data collection moment in the kth current sub-time period. It is the absolute value function. is the Z score corresponding to the speed data collected at the ath data collection time in the kth current sub-time period.

[0035] It should be noted that in actual situations, cargo in a vehicle is not likely to tilt or fall when the vehicle is driving steadily, but it is likely to tilt or fall when the vehicle is changing speed. The larger the absolute value of the Z score corresponding to the speed data, the farther the speed data is from the average value, and the greater the abnormal change. When the value is larger, it often indicates that the target vehicle's driving speed changes more significantly during the kth current sub-time period, which often indicates that the target vehicle's driving condition is more unstable during the kth current sub-time period.

[0036] The fourth step is to determine the abnormal speed change degree corresponding to each current sub-time period according to the speed representative factor and the initial speed change factor corresponding to each current sub-time period.

[0037] Among them, both the speed representative factor and the initial speed change factor can be positively correlated with the degree of abnormal speed change.

[0038] For example, the product of the speed representative factor corresponding to each current sub-time period and the initial speed variation factor may be determined as the speed abnormal variation degree corresponding to each current sub-time period.

[0039] For example, the formula for determining the degree of abnormal speed change corresponding to the current sub-time period can be: ;in, is the degree of abnormal speed change corresponding to the kth current sub-time period. k is the sequence number of the current sub-time period within the current time period. is the initial speed change factor corresponding to the kth current sub-time period. It is the speed representative factor corresponding to the kth current sub-time period.

[0040] It should be noted that when When the value is larger, it often indicates that the target vehicle's speed change degree is relatively larger in the kth current sub-time period, which often indicates that the target vehicle's driving condition is more unstable in the kth current sub-time period. The larger is the value, the faster the target vehicle is traveling in the kth current sub-time period. When it is larger, it often indicates that the driving condition of the target vehicle is more unstable and its driving speed is faster during the kth current sub-time period; it often indicates that the driving speed distribution change of the target vehicle is more abnormal during the kth current sub-time period, and it often indicates that the cargo in the target vehicle is more likely to tilt or fall during the kth current sub-time period.

[0041] The vehicle running state index determination module 103 is used to determine the vehicle running state index corresponding to each current sub-time period according to the abnormal speed variation degree corresponding to each current sub-time period and all vibration data in each current sub-time period.

[0042] As an example, the vehicle operating status indicator determination module 103 may be used to implement the following steps: In the first step, the mean value of all vibration data in each current sub-time period is determined as the vibration representative factor corresponding to each current sub-time period.

[0043] In the second step, the vehicle operation status index corresponding to each current sub-time period is determined according to the abnormal speed variation degree and the vibration representative factor corresponding to each current sub-time period.

[0044] For example, the product of the abnormal speed variation degree and the vibration representative factor corresponding to each current sub-time period may be normalized to obtain the vehicle running state index corresponding to each current sub-time period.

[0045] For example, the formula for determining the vehicle operating status index corresponding to the current sub-time period may be: ;in, is the vehicle operation status indicator corresponding to the kth current sub-time period. k is the sequence number of the current sub-time period within the current time period. is a normalization function. It is the degree of abnormal speed change corresponding to the kth current sub-time period. is the vibration representative factor corresponding to the kth current sub-time period.

[0046] It should be noted that when When the value is larger, it often indicates that the target vehicle's driving condition is more unstable and its driving speed is faster during the kth current sub-time period; it often indicates that the cargo in the target vehicle is more likely to tilt or fall during the kth current sub-time period. The larger is the value, the greater the vibration of the target vehicle is in the kth current sub-time period. When the value is larger, it often indicates that the running state of the target vehicle in the kth current sub-time period may be more abnormal, and it often indicates that the cargo in the target vehicle is more likely to tilt or fall in the kth current sub-time period.

[0047] The image fusion operation module 104 is used to perform a fusion operation on all target cargo images in the current time period to obtain a current fused image.

[0048] As an example, the OpenCV (Open Source Computer Vision) library can be used to perform a fusion operation on all target cargo images in the current time period, and the final fused image is used as the current fused image.

[0049] It is an existing technology to perform fusion operation on multiple images through OpenCV library, which can specifically include the following steps: The first step is to import the required libraries, such as numpy, cv2, matplotlib, etc.

[0050] The second step is to define a constant GOOD_POINTS_LIMITED to filter matching points.

[0051] The third step is to read several video image frames as reference images and stitching images.

[0052] The fourth step is to use the ORB (Oriented FAST and Rotated BRIEF, feature point detection) algorithm to detect and calculate the feature points and feature descriptors of the image.

[0053] The fifth step is to create a brute force matcher bf and use it to match the feature descriptors of the two images.

[0054] The sixth step is to sort the matching results and filter out good matching points with close distances.

[0055] Step 7: Use the cv2.drawMatches function to draw the matching result image.

[0056] Step 8: Extract the coordinates of the matching points for calculating the perspective transformation matrix.

[0057] Step 9. Use the cv2.findHomography function to calculate the perspective transformation matrix M.

[0058] The tenth step is to obtain the height and width of the original image and calculate the size of the stitched image.

[0059] In the eleventh step, use the cv2.warpPerspective function to perform perspective transformation on the stitched image.

[0060] In the twelfth step, use the cv2.warpAffine function to perform affine transformation on the reference image.

[0061] In the thirteenth step, several transformed video image frames are superimposed to obtain the final stitching result.

[0062] Step 14: Use the matplotlib library to draw the image and save the resulting image. The final image is the image after the fusion operation.

[0063] The indicator prediction module 105 is used to obtain the current predicted quantity of goods and the current predicted completeness of goods according to the current fused image through a pre-trained end-side large model.

[0064] The end-side big model refers to a large-scale deep learning model running locally on a device (such as a smartphone, IoT device, embedded system, etc.). The current predicted cargo quantity may be the cargo quantity in the target vehicle evaluated by the end-side big model. The current predicted cargo integrity may represent the integrity of the cargo in the target vehicle evaluated by the end-side big model.

[0065] It should be noted that the quantity and completeness of the cargo in the vehicle often remain unchanged in a short period of time during the transportation process. Since the duration corresponding to the current time period is often short, the current predicted quantity and completeness of the cargo obtained based on the current fused image can be used as the predicted quantity and completeness of the cargo of the target vehicle in the current time period, respectively.

[0066] As an example, the current fused image can be input into the end-side large model, and the end-side large model is used to evaluate the evaluation value of the target vehicle's cargo quantity and cargo integrity in the current time period, which are used as the current predicted cargo quantity and the current predicted cargo integrity, respectively.

[0067] Optionally, the training process of the large model on the client side may include the following steps: The first step is to build a large end-side model.

[0068] The second step is to obtain the historical cargo images of different industrial vehicles at each data collection moment in different historical time periods, and obtain the actual integrity and actual quantity of cargo of different industrial vehicles in different historical time periods.

[0069] The historical time period may be a time period during the transportation process. The duration corresponding to the historical time period may be equal to the duration corresponding to the current time period, and the historical time period may be before the current time period. The historical cargo image may be an image collected during the historical time period for representing the cargo placement inside the industrial vehicle.

[0070] The actual integrity of the cargo may be a manually evaluated value representing the actual integrity of the cargo in the vehicle, and the value range may be [0, 1]. The larger the value, the more complete the cargo in the vehicle. The more complete the cargo is, the more likely it is that the cargo is not damaged. The cargo quantity may represent the quantity of cargo in the vehicle. The actual cargo quantity may be the quantity of cargo actually placed in the industrial vehicle.

[0071] The third step is to fuse all historical cargo images of each industrial vehicle in each historical time period to obtain a historical fused image.

[0072] For example, the OpenCV library can be used to perform a fusion operation on all historical cargo images of each industrial vehicle in each historical time period, and each image finally fused at this time is used as a historical fused image.

[0073] In the fourth step, the constructed end-side big model is trained using the historical fusion images as training samples and the actual integrity and actual quantity of goods corresponding to the historical fusion images as training labels to obtain a trained end-side big model.

[0074] The current cargo abnormality index determining module 106 is used to determine the current cargo abnormality index according to the difference between the current predicted cargo quantity and the currently expected cargo quantity obtained in advance, and the current predicted cargo completeness.

[0075] The current expected quantity of goods may be the quantity of goods that should exist in the current time period if no pieces are lost during the transportation of the goods.

[0076] As an example, the current cargo abnormality indicator determination module 106 can be used to implement the following steps: In the first step, the absolute value of the difference between the current predicted cargo quantity and the current expected cargo quantity is determined as the current cargo quantity difference.

[0077] The second step is to determine the current cargo abnormality index based on the current cargo quantity difference and the current predicted cargo completeness, which may include the following sub-steps: The first sub-step is to determine the current cargo damage rate based on the current predicted cargo integrity.

[0078] Among them, the current predicted cargo integrity may be negatively correlated with the current cargo damage rate.

[0079] The second sub-step is to determine the current cargo abnormality index based on the difference between the current cargo damage rate and the current cargo quantity.

[0080] For example, the weighted sum of the current cargo damage rate and the current cargo quantity difference may be determined as the current cargo abnormality index.

[0081] For example, the formula for determining the current cargo abnormality index may be: ; Where D is the current cargo abnormality indicator. is a preset weight, and its value range can be [0, 1]. For example, its value can be 0.5. is a normalization function. is the current cargo quantity difference. p is the current predicted cargo completeness. is the current cargo damage rate.

[0082] It should be noted that in actual situations, the goods may fall due to unstable vehicle speed, and the goods may be damaged during the falling process. The larger the value is, the more damage the cargo in the target vehicle is in the current time period. The larger D is, the more likely it is that the quantity of goods in the target vehicle is inconsistent with the expected quantity of goods in the current time period, and the more likely it is that goods are lost or misplaced. Therefore, the larger D is, the more abnormal the state of the goods in the target vehicle is in the current time period, and the staff often need to be reminded to pay attention to the transportation status and make timely adjustments.

[0083] The state identification module 107 is used to identify the state of the cargo of the target industrial vehicle in the current time period according to the vehicle operation state indicators corresponding to all the current sub-time periods and the current cargo abnormality indicators.

[0084] As an example, the state identification module 107 may be used to implement the following steps: In the first step, the average of the vehicle operating status indicators corresponding to all current sub-time periods is determined as the representative indicator of the current vehicle operating status.

[0085] In the second step, the product of the above current vehicle operation status representative index and the above current cargo abnormality index is normalized to obtain the current cargo status index.

[0086] For example, the formula for determining the current cargo status indicator may be: ; Where H is the current cargo status indicator. is a normalized function. D is the current cargo abnormality index. W is the representative index of the current vehicle operation status.

[0087] It should be noted that when W is larger, it often indicates that the running state of the target vehicle in the current time period may be more abnormal, and it often indicates that the cargo in the target vehicle is more likely to tilt or fall in the current time period. When D is larger, it often indicates that the cargo state in the target vehicle in the current time period is more abnormal, and it is often necessary to remind the staff to pay attention to the transportation state and make timely adjustments. Therefore, when H is larger, it often indicates that the cargo state in the target vehicle in the current time period is more abnormal, and it is often necessary to remind the staff to pay attention to the transportation state and make timely adjustments.

[0088] In the third step, if the current cargo status indicator is greater than the preset warning threshold, information indicating that the cargo status of the industrial vehicle is abnormal is sent to remind the driver that the current cargo status is abnormal. At this time, the driver can stop the vehicle to check whether the cargo is misplaced or lost, and appropriately slow down the current speed. For example, the vehicle speed can be adjusted to , you can also adjust the vehicle's turning angle to . Wherein, H is the current cargo status indicator. v is the vehicle speed of the target vehicle when sending the information representing the abnormal status of the industrial vehicle cargo. It is the vehicle turning angle of the target vehicle when sending the information representing the abnormal state of the industrial vehicle cargo. The vehicle turning angle can be the maximum angle that the front wheels of the vehicle can reach when turning.

[0089] refer to Figure 2 Based on the same inventive concept as the above method embodiment, the present invention provides a method for dynamically identifying the state of cargo on an industrial vehicle based on a large visual end-side model, comprising the following steps: Step 201, obtaining speed data, vibration data and target cargo image of a target industrial vehicle at each data collection moment in a current time period through ADAS; Step 202, divide the current time period into equal parts to obtain current sub-time periods, and determine the degree of abnormal speed change corresponding to each current sub-time period based on all speed data in each current sub-time period and its corresponding Z score; Step 203, determining the vehicle running state index corresponding to each current sub-time period according to the abnormal speed change degree corresponding to each current sub-time period and all vibration data in each current sub-time period; Step 204, performing a fusion operation on all target cargo images within the current time period to obtain a current fused image; Step 205: According to the current fused image, the current predicted quantity of goods and the current predicted completeness of goods are obtained through the pre-trained large model on the end side; Step 206, determining a current cargo abnormality index according to the difference between the current predicted cargo quantity and the currently expected cargo quantity obtained in advance, and the current predicted cargo completeness; Step 207 , based on the vehicle operation status indicators corresponding to all current sub-time periods and the current cargo abnormality indicators, the cargo status of the target industrial vehicle in the current time period is identified.

[0090] Figure 3 is a schematic diagram of the structure of a computer device provided by an embodiment of the present invention. Figure 3As shown, the computer device 300 includes: a memory 301, a processor 302, and a computer program 303 stored in the memory 301 and running on the processor 302, wherein when the processor 302 executes the computer program 303, the computer device can execute the aforementioned method for dynamic identification of industrial vehicle cargo status based on a visual end-side large model.

[0091] Based on the same inventive concept as the above method embodiment, the present invention provides a server, including a memory and a processor. The memory is used to store executable program code, and the processor is used to call and run the executable program code from the memory, so that the device executes the above-mentioned method for dynamic identification of industrial vehicle cargo status based on a large visual end-side model.

[0092] Based on the same inventive concept as the above-mentioned method embodiment, the present invention provides a computer program product, which includes: computer program code, when the computer program code runs on a computer, enables the computer to execute the above-mentioned method for dynamic identification of industrial vehicle cargo status based on a visual end-side large model.

[0093] Based on the same inventive concept as the above-mentioned method embodiment, the present invention provides a computer-readable storage medium, which stores a computer program code. When the computer program code runs on a computer, the computer executes the above-mentioned method for dynamic identification of the cargo status of an industrial vehicle based on a large visual end-side model.

[0094] In summary, compared with only analyzing the collected vehicle cargo images to realize the industrial vehicle cargo status recognition, the present invention, in addition to considering the target cargo image containing vehicle cargo information, also comprehensively considers multiple other characteristics related to the vehicle cargo status, such as speed data, vibration data, abnormal speed change degree, vehicle operation status index, current predicted cargo quantity and current predicted cargo completeness, etc., thereby improving the accuracy of industrial vehicle cargo status recognition and further improving the warehousing and distribution effect.

[0095] The above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, a person skilled in the art should understand that the technical solutions described in the aforementioned embodiments can still be modified, or some of the technical features can be replaced by equivalents. These modifications or replacements do not deviate the essence of the corresponding technical solutions from the scope of the technical solutions of the embodiments of the present invention, and should all be included in the protection scope of the present invention.

Claims

1. A dynamic identification system for industrial vehicle cargo status based on a large visual end-side model, characterized in that: The system comprises: A data acquisition module is used to acquire speed data, vibration data and target cargo images of a target industrial vehicle at each data acquisition moment in a current time period through ADAS; The equal division determination module is used to divide the current time period into equal parts to obtain the current sub-time period, and determine the abnormal speed change degree corresponding to each current sub-time period according to all speed data in each current sub-time period and its corresponding Z score; A vehicle operation status indicator determination module is used to determine the vehicle operation status indicator corresponding to each current sub-time period according to the abnormal speed change degree corresponding to each current sub-time period and all vibration data in each current sub-time period; An image fusion operation module is used to perform a fusion operation on all target cargo images within the current time period to obtain a current fused image; The indicator prediction module is used to obtain the current predicted cargo quantity and the current predicted cargo completeness based on the current fused image through the pre-trained large model on the end side; A current cargo abnormality index determination module, used to determine the current cargo abnormality index according to the difference between the current predicted cargo quantity and the current expected cargo quantity obtained in advance, and the current predicted cargo completeness; The state recognition module is used to identify the state of the cargo of the target industrial vehicle in the current time period according to the vehicle operation state indicators corresponding to all current sub-time periods and the current cargo abnormality indicators.

2. According to claim 1, a dynamic identification system for industrial vehicle cargo status based on a visual end-side large model is characterized in that: Determining the degree of abnormal speed change corresponding to each current sub-time period according to all speed data in each current sub-time period and its corresponding Z score includes: The average value of all speed data in each current sub-time period is determined as the speed representative factor corresponding to each current sub-time period; The accumulated value of the absolute values ​​of the Z scores corresponding to all the speed data in each current sub-time period is determined as the initial speed change factor corresponding to each current sub-time period; The speed abnormal variation degree corresponding to each current sub-time period is determined according to the speed representative factor and the initial speed variation factor corresponding to each current sub-time period, wherein both the speed representative factor and the initial speed variation factor are positively correlated with the speed abnormal variation degree.

3. According to claim 2, a dynamic identification system for cargo status of industrial vehicles based on a large model of a visual terminal, characterized in that: The determining, according to the speed representative factor and the initial speed variation factor corresponding to each current sub-time period, the degree of abnormal speed variation corresponding to each current sub-time period includes: The product of the speed representative factor corresponding to each current sub-time period and the initial speed variation factor is determined as the speed abnormal variation degree corresponding to each current sub-time period.

4. According to claim 1, a dynamic identification system for industrial vehicle cargo status based on a visual end-side large model is characterized in that: The determining of the vehicle running status index corresponding to each current sub-time period according to the abnormal speed variation degree corresponding to each current sub-time period and all vibration data in each current sub-time period includes: The mean value of all vibration data in each current sub-time period is determined as the vibration representative factor corresponding to each current sub-time period; According to the abnormal speed variation degree and the vibration representative factor corresponding to each current sub-time period, the vehicle running state index corresponding to each current sub-time period is determined.

5. The industrial vehicle cargo status dynamic recognition system based on the visual end-side large model according to claim 4 is characterized in that: Determining the vehicle running state index corresponding to each current sub-time period according to the abnormal speed change degree and the vibration representative factor corresponding to each current sub-time period includes: The product of the abnormal speed variation degree and the vibration representative factor corresponding to each current sub-time period is normalized to obtain the vehicle operation status index corresponding to each current sub-time period.

6. The industrial vehicle cargo status dynamic recognition system based on the visual end-side large model according to claim 1 is characterized in that: Determining the current cargo abnormality index according to the difference between the current predicted cargo quantity and the currently expected cargo quantity obtained in advance, and the current predicted cargo completeness, includes: The absolute value of the difference between the current forecasted cargo quantity and the current expected cargo quantity is determined as the current cargo quantity difference; Determine the current cargo abnormality index based on the current cargo quantity difference and the current predicted cargo completeness.

7. The industrial vehicle cargo status dynamic recognition system based on the visual end-side large model according to claim 6 is characterized in that: Determining the current cargo abnormality index according to the current cargo quantity difference and the current predicted cargo completeness includes: According to the current predicted cargo integrity, the current cargo damage rate is determined, wherein the current predicted cargo integrity and the current cargo damage rate are negatively correlated; Determine the current cargo abnormality index based on the difference between the current cargo damage rate and the current cargo quantity.

8. The industrial vehicle cargo status dynamic recognition system based on the visual end-side large model according to claim 7 is characterized in that: Determining the current cargo abnormality index according to the current cargo damage rate and the current cargo quantity difference includes: The weighted sum of the current cargo damage rate and the current cargo quantity difference is determined as the current cargo abnormality index.

9. The industrial vehicle cargo status dynamic recognition system based on the visual end-side large model according to claim 1 is characterized in that: The method of identifying the state of the cargo of the target industrial vehicle in the current time period according to the vehicle operation state indicators corresponding to all the current sub-time periods and the current cargo abnormality indicators includes: The average of the vehicle operation status indicators corresponding to all current sub-time periods is determined as the representative indicator of the current vehicle operation status; Normalizing the product of the current vehicle running state representative index and the current cargo abnormality index to obtain a current cargo state index; If the current cargo status indicator is greater than a preset warning threshold, information indicating that the cargo status of the industrial vehicle is abnormal is sent.

10. The industrial vehicle cargo status dynamic recognition system based on the visual end-side large model according to claim 1 is characterized in that: The training process of the large model on the client side includes: Build a large end-side model; Obtain historical cargo images of different industrial vehicles at each data collection moment in different historical time periods, and obtain the actual integrity and actual quantity of cargo of different industrial vehicles in different historical time periods; Perform a fusion operation on all historical cargo images of each industrial vehicle in each historical time period to obtain a historical fused image; The constructed end-side large model is trained using historical fusion images as training samples and the actual integrity and actual quantity of goods corresponding to the historical fusion images as training labels.

Images