Semantic line-based vehicle cargo overrun detection method and device, and electronic equipment

Abstract

The invention discloses a semantic line-based vehicle cargo overrun detection method and device and electronic equipment, and relates to the field of data processing. The method comprises the following steps: acquiring an initial image of a transport vehicle; identifying the initial image based on a preset vehicle positioning model to obtain an initial coordinate of the transport vehicle; carrying out frame expansion and cutting on the initial image according to the initial coordinates to obtain a vehicle sub-graph of the transport vehicle; performing instance segmentation on the vehicle sub-graph, identifying a cargo area in the hopper area, and outputting cargo segmentation areas; performing semantic line detection on the vehicle sub-graph to obtain an upper edge position of a vehicle hopper area; comparing the cargo segmentation area with the upper edge position, and judging whether the cargo exceeds the upper edge position or not; if it is determined that the goods exceed the upper edge position, the transport vehicle is determined to be an overrun vehicle; and if it is determined that the goods do not exceed the upper edge position, determining that the transport vehicle is a normal vehicle. By implementing the technical scheme provided by the invention, the accuracy of vehicle overrun detection is improved.

Description

Technical Field

[0001] The present application relates to the field of data processing, and in particular to a method, device and electronic equipment for detecting overweight vehicle cargo based on semantic lines. Background Art

[0002] With the growth of global trade and the rapid development of the transportation industry, road transport has become a crucial mode of freight transportation. The safety and compliance of transport vehicles are crucial to road traffic safety. In particular, the presence of overweight vehicles not only increases road traffic risks but can also cause damage to road infrastructure. Therefore, efficiently and accurately detecting overweight vehicles has become a crucial task in road traffic management and freight transport regulation.

[0003] The existing vehicle overload detection technologies mainly include methods based on traditional edge detection and methods based on calibration comparison. Although these methods can achieve the determination of vehicle overload to a certain extent, they still have many shortcomings. For example, traditional edge detection methods are sensitive to image noise and are easily affected by noise, resulting in edge detection errors. In addition, this method lacks understanding of high-level semantic information and cannot accurately distinguish between the real edge of an object and the texture edge. In addition, the method based on calibration comparison requires calibration of each camera point, which is cumbersome to operate. The applicability of this method is limited by the consistency of the vehicle's parking position and vehicle type. When the vehicle types are diverse, the recognition accuracy may be reduced. Therefore, the methods for detecting whether vehicle cargo is overloaded in the relevant technology have the problem of low accuracy.

[0004] Therefore, there is an urgent need for a method, device and electronic equipment for detecting overweight vehicle cargo based on semantic lines. Summary of the Invention

[0005] The present application provides a method, device and electronic equipment for detecting over-gauge cargo on vehicles based on semantic lines, thereby improving the accuracy of over-gauge detection on vehicles.

[0006] In a first aspect of the present application, a semantic line-based vehicle cargo overload detection method is provided, the method comprising: acquiring an initial image of a transport vehicle; identifying the initial image based on a preset vehicle positioning model to obtain initial coordinates of the transport vehicle; expanding and cropping the initial image according to the initial coordinates to obtain a vehicle sub-image of the transport vehicle, the vehicle sub-image including a cargo area of the transport vehicle; performing instance segmentation on the vehicle sub-image to identify a cargo area within the cargo area and outputting a cargo segmentation area; performing semantic line detection on the vehicle sub-image to obtain an upper edge position of the cargo area; comparing the cargo segmentation area with the upper edge position to determine whether the cargo exceeds the upper edge position; if it is determined that the cargo exceeds the upper edge position, determining that the transport vehicle is an overloaded vehicle; if it is determined that the cargo does not exceed the upper edge position, determining that the transport vehicle is a normal vehicle.

[0007] By adopting the above technical solution, an initial image of a transport vehicle is acquired, the vehicle's position is identified using a preset vehicle positioning model, and the vehicle image is cropped to obtain a vehicle sub-image containing the cargo area. Then, instance segmentation is performed on the vehicle sub-image to identify the cargo area within the cargo area. Semantic line detection is also performed to determine the location of the cargo's upper edge. Finally, the cargo area is compared with the cargo's upper edge to determine whether the cargo exceeds the limit, thereby confirming whether the vehicle is overweight. This step enables automatic detection of overweight cargo on transport vehicles. Using computer vision technology, the vehicle's position, cargo area, and cargo's edge are automatically identified, improving detection efficiency and accuracy. By comparing the positional relationship between the cargo area and the cargo's upper edge, it is possible to objectively determine whether the cargo exceeds the cargo area, ultimately achieving automatic identification of overweight vehicles. This helps to promptly detect and address overweight behavior, thereby improving road transport safety. This method utilizes instance segmentation of the vehicle and cargo and comparative analysis of the cargo's edge. By leveraging deep learning models such as instance segmentation and semantic line detection, it eliminates the need for external camera parameters or complex calibration processes, significantly improving the method's ease of use and adaptability to various scenarios, thereby enhancing detection accuracy.

[0008] Optionally, instance segmentation is performed on the vehicle sub-image, the cargo area within the truck bed area is identified, and the cargo segmentation area is output, specifically including: inputting the vehicle sub-image into a preset instance segmentation model for feature extraction to obtain a feature map of the vehicle sub-image; decoding the feature map to obtain segmentation results of each instance in the vehicle sub-image; based on the segmentation results, identifying the instance located in the truck bed area, and outputting the instance located in the truck bed area as the cargo segmentation area.

[0009] By adopting the above technical solution, the vehicle sub-image is input into the preset instance segmentation model, and the feature extraction of the vehicle sub-image is performed to obtain the feature map of the vehicle sub-image; then the feature map is decoded to further obtain the accurate segmentation results of each instance (i.e., object) in the vehicle sub-image. Based on the segmentation results, the cargo instances located in the truck bed area can be accurately identified and output as the cargo segmentation area. This cargo recognition method based on instance segmentation can effectively deal with complex situations such as irregular cargo shapes and scattered stacking, and obtain the precise outline of the cargo area. The preset instance segmentation model uses deep learning technology to automatically learn the appearance characteristics of cargo from the vehicle sub-image, making it more adaptable.

[0010] Optionally, performing semantic line detection on the vehicle sub-image to obtain the upper edge position of the vehicle bucket area specifically includes: inputting the vehicle sub-image into a preset semantic line detection model for feature extraction to obtain an edge feature map of the vehicle sub-image; decoding the edge feature map to obtain the semantic line in the vehicle sub-image; determining a target semantic line located above and parallel to the vehicle bucket area based on the position of the semantic line in the vehicle sub-image, and determining the target semantic line as the upper edge position of the vehicle bucket area.

[0011] By adopting the above technical solution and utilizing a preset semantic line detection model, edge features are extracted from the vehicle sub-image, resulting in an edge feature map of the image. Decoding is then performed to obtain the semantic lines within the vehicle sub-image, i.e., the edge contours of the objects. By analyzing the position of the semantic lines within the image, a target semantic line located above and parallel to the vehicle bed area is identified and used as the upper edge of the bed. This semantic line detection method fully utilizes the edge information in the image, accurately locating the bed edge while also addressing challenges such as tilted and curved bed deformations. The preset semantic line detection model utilizes a deep learning algorithm, trained with extensive annotated data, to automatically extract semantic line features from complex backgrounds, demonstrating strong robustness and generalization capabilities. This method, combined with prior knowledge of the vehicle scene and leveraging the geometric constraint that the upper edge of the bed is parallel to the bed, further improves the accuracy of bed edge location.

[0012] Optionally, comparing the cargo segmentation area with the upper edge position to determine whether the cargo exceeds the upper edge position specifically includes: determining whether the contour point set of the cargo segmentation area is contained in the vehicle bucket area; if it is determined that a target contour point exists outside the vehicle bucket area, determining that the cargo exceeds the upper edge position, the target contour point being any one of the multiple contour points included in the contour point set; if it is determined that the target contour point does not exist outside the vehicle bucket area, determining that the cargo does not exceed the upper edge position.

[0013] By employing this technical solution, the cargo's presence beyond the upper edge of the bucket is determined by determining whether the contour points of the cargo segmentation area are completely contained within the bucket area. If any contour point lies outside the bucket area, the cargo has exceeded the upper edge; otherwise, the cargo is considered within the limit. This method, based on the cargo contour points, can accommodate situations where the cargo partially exceeds the bucket, demonstrating good adaptability. Combining the cargo segmentation results with the bucket edge position allows for full utilization of the recognition results from both modules, enabling cross-validation and improved reliability of over-limit determination.

[0014] Optionally, before obtaining the initial image of the transport vehicle, the method further includes: obtaining a picture dataset of multiple vehicles; using annotation software to annotate the picture dataset with target frames to obtain a labeled training dataset; and using the training dataset to train a YOLOv8 target detection model to obtain the preset vehicle positioning model.

[0015] By employing this technical solution, before acquiring transport vehicle images, a vehicle image dataset is first obtained and labeled with object boxes using annotation software to generate a fully labeled training dataset. This dataset is then used to train the YOLOv8 object detection model to generate a pre-defined vehicle positioning model. This training method leverages artificial intelligence technology. Through extensive data training, the model automatically learns the appearance characteristics of vehicles, developing robust vehicle positioning capabilities. Utilizing the proven YOLOv8 algorithm, high positioning accuracy is achieved while maintaining high positioning speed. Manually annotating and training the target boxes effectively reduces the model's false positive and missed detection rates. The trained vehicle positioning model can be used in subsequent vehicle and cargo overload detection, significantly improving detection efficiency and accuracy while reducing labor costs.

[0016] Optionally, after using annotation software to perform target frame annotation on the image dataset to obtain a labeled training dataset, the method further includes: cropping the corresponding vehicle area according to the target frame coordinates of the training dataset to obtain a first vehicle sub-image dataset; using the annotation software to perform instance segmentation annotation of the cargo area on the first vehicle sub-image dataset to obtain a cargo segmentation training dataset; and using the cargo segmentation training dataset to train a YOLOv8 instance segmentation model to obtain a preset instance segmentation model.

[0017] By adopting the above technical solution, after obtaining a training dataset for vehicle positioning, the vehicle area is first cropped according to the target box coordinates to obtain the first vehicle sub-image dataset. Then, annotation software is used to perform instance segmentation annotation of the cargo area in this dataset to obtain the cargo segmentation training dataset. Finally, this dataset is used to train the YOLOv8 instance segmentation model to obtain the preset instance segmentation model. This streamlined training method fully utilizes the vehicle positioning results and reduces background interference through cropping, making subsequent cargo segmentation training more accurate and efficient. The cargo segmentation annotation accurately identifies the cargo outline, providing high-quality training samples for the instance segmentation model. Training with the YOLOv8 instance segmentation algorithm enables the model to possess excellent cargo segmentation capabilities, capable of handling cargo recognition tasks of different vehicle types, angles, and lighting conditions, with good generalization performance and strong adaptability.

[0018] Optionally, after the image dataset is annotated with target frames using annotation software to obtain an annotated training dataset, the method further includes: cropping the corresponding vehicle area according to the target frame coordinates of the training dataset to obtain a second vehicle sub-image dataset; annotating the semantic line segments of the vehicle bucket edge line of the second vehicle sub-image dataset using the annotation software to obtain a vehicle bucket edge training dataset; and training a DeepHough semantic line detection model using the vehicle bucket edge training dataset to obtain a preset semantic line detection model.

[0019] By adopting the above technical solution, after obtaining a training dataset for vehicle positioning, the vehicle area is first cropped according to the target frame coordinates to obtain a second vehicle sub-image dataset. This dataset is then annotated with semantic line segments of the vehicle bed edge lines using annotation software to obtain a vehicle bed edge training dataset. Finally, this dataset is used to train the DeepHough semantic line detection model to obtain a preset semantic line detection model. This targeted training method, by cropping the vehicle area, eliminates the influence of background on vehicle bed edge detection, focusing training on learning vehicle bed edge features. Semantic line segment annotation can accurately identify the edge line of the upper edge of the vehicle bed, providing high-quality training samples for the semantic line detection model. Using a specialized DeepHough semantic line detection algorithm for training enables the model to learn the geometric features and semantic information of the vehicle bed edge, accurately extracting the upper edge of the vehicle bed from complex backgrounds, and achieve high positioning accuracy and strong anti-interference capabilities.

[0020] In a second aspect of the present application, a vehicle cargo over-limit detection device based on semantic lines is provided, which includes an image acquisition module, a vehicle positioning module, a sub-image cropping module, a cargo segmentation module, a bucket edge detection module and an over-limit judgment module, wherein: the image acquisition module is used to acquire an initial image of a transport vehicle; the vehicle positioning module is used to identify the initial image based on a preset vehicle positioning model to obtain the initial coordinates of the transport vehicle; the sub-image cropping module is used to expand and crop the initial image according to the initial coordinates to obtain a vehicle sub-image of the transport vehicle, wherein the vehicle sub-image includes the bucket area of the transport vehicle; the cargo segmentation module is used to identify the initial image based on a preset vehicle positioning model to obtain the initial coordinates of the transport vehicle; the sub-image cropping module is used to expand and crop the initial image according to the initial coordinates to obtain a vehicle sub-image of the transport vehicle, wherein the vehicle sub-image includes the bucket area of the transport vehicle; The block is used to perform instance segmentation on the vehicle sub-graph, identify the cargo area in the truck bed area, and output the cargo segmentation area; the truck bed edge detection module is used to perform semantic line detection on the vehicle sub-graph to obtain the upper edge position of the truck bed area; the over-limit judgment module is used to compare the cargo segmentation area with the upper edge position to determine whether the cargo exceeds the upper edge position; the over-limit judgment module is further used to determine that the transport vehicle is an over-limit vehicle if it is determined that the cargo exceeds the upper edge position; the over-limit judgment module is further used to determine that the transport vehicle is a normal vehicle if it is determined that the cargo does not exceed the upper edge position.

[0021] In the third aspect of the present application, an electronic device is provided, including a processor, a memory, a user interface and a network interface, the memory is used to store instructions, the user interface and the network interface are both used to communicate with other devices, and the processor is used to execute the instructions stored in the memory so that the electronic device performs any of the methods described above.

[0022] In a fourth aspect of the present application, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores instructions. When the instructions are executed, any one of the methods described above is executed.

[0023] In summary, one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. By acquiring an initial image of a transport vehicle, the vehicle's position is identified using a pre-set vehicle positioning model. The vehicle image is cropped to obtain a vehicle sub-image containing the cargo area. Then, instance segmentation is performed on the vehicle sub-image to identify the cargo area within the cargo area. Semantic line detection is also performed to determine the location of the cargo's upper edge. Finally, the cargo area is compared with the cargo's upper edge to determine whether the cargo exceeds the limit, thereby confirming whether the vehicle is overweight. This step enables automatic detection of overweight cargo on transport vehicles. Using computer vision technology, the vehicle's position, cargo area, and cargo's edge are automatically identified, improving detection efficiency and accuracy. By comparing the positional relationship between the cargo area and the cargo's upper edge, it is possible to objectively determine whether the cargo exceeds the cargo area. Ultimately, automatic identification of overweight vehicles is achieved, facilitating timely detection and action, and improving road transport safety. This method utilizes instance segmentation of the vehicle and cargo and comparative analysis of the cargo's edge. By leveraging deep learning models, such as instance segmentation and semantic line detection, it eliminates the need for external camera parameters or complex calibration processes. This significantly improves the method's ease of use and adaptability to diverse scenarios, enhancing detection accuracy. BRIEF DESCRIPTION OF THE DRAWINGS

[0024] Figure 1 1 is a flow chart of a method for detecting over-gauge vehicle cargo based on semantic lines disclosed in an embodiment of the present application; Figure 2 This is a module diagram of a vehicle cargo overload detection device based on semantic lines disclosed in an embodiment of the present application; Figure 3 This is a structural diagram of an electronic device disclosed in an embodiment of the present application.

[0025] Explanation of the accompanying drawings: 201, image acquisition module; 202, vehicle positioning module; 203, sub-image cropping module; 204, cargo segmentation module; 205, vehicle bed edge detection module; 206, over-limit judgment module; 300, electronic device; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. DETAILED DESCRIPTION

[0026] In order to enable those skilled in the art to better understand the technical solutions in this specification, the technical solutions in the embodiments of this specification will be clearly and completely described below in conjunction with the drawings in the embodiments of this specification. Obviously, the described embodiments are only part of the embodiments of this application, not all of the embodiments.

[0027] In the description of the embodiments of this application, words such as "for example" or "for instance" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "for example" or "for instance" in the embodiments of this application should not be construed as being preferred or advantageous over other embodiments or designs. Rather, the use of words such as "for example" or "for instance" is intended to present the relevant concepts in a concrete manner.

[0028] In the description of the embodiments of the present application, the term "multiple" means two or more. For example, multiple systems refer to two or more systems, and multiple screen terminals refer to two or more screen terminals. In addition, the terms "first" and "second" are used for descriptive purposes only and are not to be understood as indicating or implying relative importance or implicitly indicating the indicated technical features. Thus, the features defined as "first" and "second" may explicitly or implicitly include one or more of the features. The terms "including", "comprising", "having" and their variations all mean "including but not limited to", unless otherwise specifically emphasized.

[0029] This application provides a vehicle cargo overload detection method based on semantic lines, referring to Figure 1 , Figure 1 This is a flow chart of a semantic line-based vehicle cargo overweight detection process provided by an embodiment of the present application. The method is applied to a server, which is a server that executes a semantic line-based vehicle cargo overweight detection program. The method includes steps S101 to S108, which are as follows: Step S101: Acquire an initial image of a transport vehicle.

[0030] In step S101, the server establishes a stable data communication connection with the camera deployed on-site. After receiving the real-time video stream data from the camera, the server decodes it. The video stream transmitted by the camera is usually encoded in a format such as H.264 or H.265. The server uses a corresponding decoder (such as FFmpeg) to decode it into a raw RGB image sequence. Each decoded frame can be used as an independent initial image to trigger the subsequent vehicle cargo overload detection process.

[0031] Step S102: Recognize the initial image based on a preset vehicle positioning model to obtain the initial coordinates of the transport vehicle.

[0032] In step S102, the server presets the image input into the preset vehicle positioning model, and the preset vehicle positioning model is the preset YOLOv8-Detect vehicle positioning model. This model is an end-to-end convolutional neural network that can simultaneously complete vehicle positioning and classification in a single forward reasoning. The main structure of the model includes: the backbone network is used to extract multi-scale features of the image; the neck network is used to aggregate the multi-scale features extracted by the backbone network, and enhance the semantic information and position information of the features through structures such as feature pyramids; the probe is used to predict the position coordinates and category of the vehicle based on the aggregated feature map. After the model inference is completed, a set of detection results will be output, each detection result includes the position coordinates of the vehicle bounding box (x min ,y min , x max ,y max ), representing the vehicle's position in the image; the vehicle category, indicating the specific type of vehicle; and the confidence score, indicating the reliability of the detection result. After post-processing, the server ultimately obtains the coordinates of the transport vehicle's position in the initial image, typically represented as a rectangular area. To facilitate subsequent processing, the server converts the coordinates from the pixel coordinate system to the image coordinate system, with the upper left corner of the image as the origin (0, 0) and the lower right corner as (width, height).

[0033] For example, suppose the server receives an initial image of 1920*1080 pixels. After processing the vehicle localization model, a truck detection result with a confidence level of 0.95 is obtained, and its bounding box coordinates are (450, 200, 1050, 720). The server first normalizes these coordinates, converting them to a range of 0-1, to obtain the normalized coordinates (0.234, 0.185, 0.547, 0.667). The normalized coordinates are then multiplied by the image width and height to obtain the vehicle's true position in the image coordinate system: (450, 200, 1050, 720). The rectangular area represented by these coordinates is the initial position of the transport vehicle in the initial image.

[0034] In one possible embodiment, the process of constructing a preset vehicle positioning model includes: before obtaining an initial image of a transport vehicle, the method further includes: obtaining a picture dataset of multiple vehicles; using annotation software to annotate the picture dataset with target boxes to obtain a labeled training dataset; and using the training dataset to train a YOLOv8-Detect target detection model to obtain the preset vehicle positioning model.

[0035] Specifically, the service obtains a large number of vehicle image datasets as the raw material for training the preset vehicle positioning model. The dataset can come from various channels, such as traffic monitoring camera collection, vehicle driving recorder upload, network image crawling, etc. In order to ensure the diversity and coverage of the data, the image dataset should contain vehicle images of various types, angles, and lighting conditions to improve the generalization ability of the model. Next, the server annotates the obtained vehicle image dataset. The annotation process is usually completed with the help of specialized annotation software. The annotator manually draws the vehicle's bounding box on each vehicle image and marks the vehicle's category. After the annotation is completed, each image will be associated with a corresponding annotation file that records the location coordinates and category information of the vehicle's bounding box.

[0036] After the labeling is completed, the server randomly divides the labeled vehicle image dataset into a training set and a validation set according to a certain ratio. The training set is used for model training optimization, and the validation set is used for model performance evaluation and hyperparameter selection. The division ratio can usually be 8:2 or 9:1. Next, the server converts the images and annotation files of the training set and validation set into a specific format to adapt to the input requirements of the YOLOv8-Detect model. Then, the server uses the converted training set data to train the YOLOv8-Detect model. In the embodiment of the present application, YOLOv8-Detect can be understood as a single-stage target detection algorithm that simultaneously completes vehicle positioning and classification through feature extraction, multi-scale feature fusion and bounding box regression operations. During the training process, the server inputs the vehicle images and annotation information into the YOLOv8-Detect model, and through optimization algorithms such as backpropagation and gradient descent, the model parameters are gradually adjusted to fit the training data. After sufficient training, the server uses the validation set to evaluate the performance indicators of the model. By comparing the performance under different hyperparameters and network structures, the optimal model configuration is selected. Finally, the YOLOv8-Detect model with the best performance is saved as the preset vehicle positioning model for subsequent vehicle overload detection tasks.

[0037] Step S103: expanding and cropping the initial image according to the initial coordinates to obtain a vehicle sub-image of the transport vehicle, wherein the vehicle sub-image includes a truck bed area of the transport vehicle.

[0038] In step S103, the server expands and crops the original image based on the initial coordinates output by the vehicle positioning model, thereby obtaining a vehicle sub-image that includes the entire truck bed area. The purpose of this step is to provide a relatively independent image area for subsequent cargo segmentation and compartment edge detection, thereby reducing the interference of irrelevant background. Specifically, the server first extracts the initial coordinates of the transport vehicle from the detection results obtained in step S102. The coordinates are represented as a rectangular area (xmin ,y min , x max ,y max ), where (x min ,y min ) is the coordinate of the upper left corner of the rectangle, (x max ,y max ) is the coordinate of the lower right corner of the rectangle. The server then appropriately expands the initial coordinates to ensure that the cropped vehicle sub-image fully covers the entire truck bed area. Typically, the initial coordinates output by the preset vehicle positioning model only include the front of the vehicle and most of the compartment, and may miss the top and rear of the truck bed. To avoid this, the server performs pixel expansion in four directions: upward, downward, left, and right, based on the initial coordinates. The selection of the expansion size takes into account factors such as vehicle type, image resolution, and truck bed position. Generally speaking, for large trucks, due to their larger truck bed size, relatively more expansion pixels are required; for small trucks, relatively fewer expansion pixels can be used. The server can set different expansion parameters based on vehicle type. For example, for large trucks, the expansion is 100 pixels upward, 50 pixels downward, and 30 pixels to the left and right; for small trucks, the expansion is 60 pixels upward, 30 pixels downward, and 20 pixels to the left and right.

[0039] After the expansion is completed, the server obtains the coordinates of the expanded rectangular area (x min1 ,y min1 , x max1 ,y max1 ). It should be noted that the expanded coordinates may exceed the boundaries of the original image. To avoid this, the server crops the expanded coordinates to ensure that they do not exceed the image boundaries. Specifically, you can min1 Compare the value of x with 0 and take the larger value; max1 Compare the value of y with the image width and take the smaller value; min1 Compare the value of y with 0 and take the larger value; max1 Compare the value of with the image height and take the smaller value.

[0040] Finally, the server uses the cropped, expanded coordinates to crop the original image, creating a vehicle subimage encompassing the entire truck bed area. Cropping can be performed using the crop function in image processing libraries such as OpenCV. The cropped vehicle subimage only contains the entire transport vehicle, with the majority of the area consisting of the truck bed and bed, and relatively little background information.

[0041] Step S104: performing instance segmentation on the vehicle subgraph, identifying the cargo area within the vehicle box area, and outputting the cargo segmentation area.

[0042] In step S104, instance segmentation is performed on the vehicle sub-image, the cargo area within the truck bed area is identified, and the cargo segmentation area is output, specifically including: inputting the vehicle sub-image into a preset instance segmentation model for feature extraction to obtain a feature map of the vehicle sub-image; decoding the feature map to obtain segmentation results of each instance in the vehicle sub-image; based on the segmentation results, identifying the instance located in the truck bed area, and outputting the instance located in the truck bed area as the cargo segmentation area.

[0043] Specifically, the server performs instance segmentation on the vehicle sub-image obtained in step S103, identifies the cargo area within the truck bed area, and outputs it as the cargo segmentation area. First, the server inputs the vehicle sub-image into a preset instance segmentation model to extract features. The preset instance segmentation model is an end-to-end model based on a convolutional neural network, which adopts an encoder-decoder architecture. In this embodiment, the model adopts the YOLOv8-Segment algorithm, which is an instance segmentation method based on the YOLOv8 target detection model. When the vehicle sub-image is input into the YOLOv8-Segment model, it first passes through the encoder network for feature extraction. The encoder is usually composed of a series of convolutional layers and downsampling layers, and its purpose is to convert the input image into a set of feature maps with high-level semantic information. After being processed by the encoder, the vehicle sub-image is converted into a set of feature maps with rich semantic information, denoted as F. The feature map F usually has multiple scales (such as 1 / 8, 1 / 16, 1 / 32), and each scale corresponds to a different receptive field and semantic granularity. Next, the server decodes the feature map F to obtain segmentation results for each instance in the vehicle subimage. The decoder consists of a series of upsampling and convolutional layers, whose purpose is to gradually restore the high-level feature map extracted by the encoder to the original image resolution and generate per-pixel segmentation predictions. In YOLOv8-Segment, the decoder fuses the encoder's feature maps at different scales through lateral connections to produce a set of decoded feature maps with rich spatial detail. The decoder then applies a 1x1 convolutional layer to the decoded feature map at the last scale to generate a per-pixel class probability map and instance mask map. The class probability map is a tensor of size H*W*C, where H and W are the height and width of the input image, respectively, and C is the number of instance categories (e.g., 2 for foreground and background). Each element of the class probability map represents the probability that the corresponding pixel belongs to each category. The instance mask map is a tensor of size H*W*M, where M is the maximum number of instances supported by the model (e.g., 100). Each channel of the instance mask map corresponds to an instance, where the pixel value 1 represents the mask area of the instance, and the pixel value 0 represents the background area.

[0044] The server combines the class probability map and the instance mask map to obtain pixel-level segmentation results for each instance in the vehicle sub-image. Specifically, for each channel in the instance mask map, the server first determines the class corresponding to the instance based on the class probability map. It then uses the mask region of that channel as the segmentation result for that instance. Finally, the server identifies the cargo instances within the cargo area based on the segmentation results and outputs them as cargo segmentation regions. This step is achieved by analyzing the location information of each instance. Specifically, the server calculates the bounding rectangle of each instance's mask region and determines whether the bounding rectangle is completely within the cargo area of the vehicle sub-image. The cargo area can be obtained based on prior knowledge (such as the approximate location of the cargo area in the vehicle sub-image) or another cargo detection model. For instances within the cargo area, the server retains their mask regions as cargo segmentation regions, while filtering out instances outside the cargo area. Ultimately, the server outputs a set of cargo segmentation regions, each corresponding to a cargo instance within the cargo area.

[0045] For example, suppose the server performs instance segmentation on an 800x600 vehicle subimage. After processing it with the YOLOv8-Segment model, it obtains segmentation results for four cargo instances. Three of these instances are located within the vehicle bed, and one is outside. The server outputs the masked regions of these three instances as the cargo segmentation regions, ignoring the instance outside the vehicle bed. Finally, the server outputs three cargo segmentation regions, each consisting of three binary mask images, where pixel values of 1 represent the corresponding cargo area.

[0046] In one possible embodiment, the process of constructing a preset instance segmentation model specifically includes: using annotation software to perform target box annotation on the image dataset to obtain a labeled training dataset, and the method further includes: cropping the corresponding vehicle area according to the target box coordinates of the training dataset to obtain a first vehicle sub-image dataset; using the annotation software to perform instance segmentation annotation on the cargo area of the first vehicle sub-image dataset to obtain a cargo segmentation training dataset; using the cargo segmentation training dataset to train a YOLOv8-Segment instance segmentation model to obtain a preset instance segmentation model.

[0047] Specifically, based on the construction of the preset vehicle positioning model, the server further constructs a preset instance segmentation model for fine segmentation of the cargo area in the vehicle sub-image. Specifically, the server first uses the annotation dataset used to construct the preset vehicle positioning model in step S102 to crop the vehicle area therein to obtain a new vehicle sub-image dataset. This step can be regarded as a secondary processing of the dataset, the purpose of which is to extract the vehicle area of interest from the original image and provide more targeted data for subsequent cargo segmentation and annotation. The server traverses each annotated image and reads the vehicle bounding box coordinates (x min ,y min , x max ,y max ), and based on the coordinates, the corresponding rectangular area is cropped out in the original image to obtain a vehicle sub-image. Cropping can be implemented using the crop function of an image processing library such as OpenCV. The cropped vehicle sub-image usually contains only one vehicle, with most of the area being the carriage and the cargo box, similar to the vehicle sub-image obtained by expanding the frame and cropping in step S103. After cropping, the server obtains a new vehicle sub-image dataset, which is recorded as the first vehicle sub-image dataset. Compared with the original image dataset, the images in this dataset are smaller in size and more focused in content, making it more suitable for cargo segmentation and annotation.

[0048] Next, the server performs instance segmentation annotation of the cargo area in the first vehicle sub-image dataset. Unlike bounding box annotation in the vehicle localization task, instance segmentation annotation requires outlining the outline of each individual cargo item in the image at the pixel level, which is a more detailed and time-consuming task. The server can use the same annotation software used for the vehicle localization task, such as LabelMe, and use the Polygon tool to perform pixel-by-pixel outline annotation of each cargo item. During the annotation process, the server identifies each vehicle sub-image, identifies each cargo item in the image, and accurately outlines its outline using the Polygon tool. For occluded or overlapping cargo items, the outline of the visible portion is also annotated as much as possible. The annotation information for each cargo item is typically stored as a sequence of polygon vertex coordinates and the corresponding cargo category. After annotating the first vehicle sub-image dataset, the server obtains a cargo segmentation training dataset. Each vehicle sub-image corresponds to a corresponding annotation file, which records the pixel-level outline coordinates and category information of each cargo item in the image.

[0049] Finally, the server trains the YOLOv8-Segment instance segmentation model using the cargo segmentation training dataset, resulting in the pre-set instance segmentation model. YOLOv8-Segment is an instance segmentation algorithm based on the YOLOv8 object detection model. By introducing an additional segmentation branch in the detection head, it can simultaneously output object bounding box coordinates and pixel-level segmentation masks. Similar to the training process for the pre-set vehicle localization model, the server first divides the cargo segmentation training dataset into training and validation sets according to a certain ratio and converts them into the data format required by YOLOv8-Segment. The server then defines the network structure and loss function of the YOLOv8-Segment model and trains the model using the training data. During training, the model continuously improves its performance on the cargo segmentation task by learning pixel-level segmentation masks for each individual item. After multiple iterations of optimization, the model's performance on the validation set continues to improve until reaching the pre-set convergence criteria. Finally, the server obtains a trained YOLOv8-Segment instance segmentation model, which is used as the pre-set instance segmentation model for the cargo segmentation task.

[0050] Step S105: performing semantic line detection on the vehicle sub-graph to obtain the upper edge position of the vehicle bucket area.

[0051] In step S105, semantic line detection is performed on the vehicle sub-image to obtain the upper edge position of the vehicle bucket area, specifically including: inputting the vehicle sub-image into a preset semantic line detection model for feature extraction to obtain an edge feature map of the vehicle sub-image; decoding the edge feature map to obtain semantic lines in the vehicle sub-image; and determining a target semantic line located above and parallel to the vehicle bucket area based on the position of the semantic line in the vehicle sub-image, and determining the target semantic line as the upper edge position of the vehicle bucket area.

[0052] Specifically, the server performs semantic line detection on the vehicle sub-image obtained in step S103 to identify the upper edge of the truck bed area. This step is key to determining whether the cargo is overloaded. By accurately locating the upper edge of the truck bed, it provides an important reference for subsequent cargo height calculations. First, the server inputs the vehicle sub-image into a preset semantic line detection model to perform feature extraction. The preset semantic line detection model is an end-to-end model based on deep learning. In this embodiment, the model uses the DeepHough algorithm, a semantic line detection method based on the Hough transform.

[0053] When a vehicle subimage is input into the DeepHough model, it first passes through the encoder network for feature extraction. The encoder typically consists of a series of convolutional and downsampling layers, and its purpose is to convert the input image into a set of feature maps with high-level semantic information. In DeepHough, the encoder uses a classic convolutional neural network architecture such as ResNet, followed by an FPN (Feature Pyramid Network) module to achieve multi-scale feature fusion and enhancement.

[0054] After processing by the encoder, the vehicle sub-image is converted into a set of feature maps rich in edge information, denoted as E. The edge feature maps E typically have multiple scales (e.g., 1 / 8, 1 / 16, and 1 / 32), each corresponding to a different receptive field and semantic granularity. Unlike ordinary feature maps, edge feature maps focus more on structural information within the image, such as lines and contours. The server then decodes the edge feature maps E to obtain semantic line detection results in the vehicle sub-image. The decoder typically consists of a series of upsampling and convolutional layers. Its purpose is to gradually restore the high-level feature maps extracted by the encoder to the resolution of the original image and generate semantic line predictions for each pixel. In DeepHough, the decoder uses skip connections to concatenate the encoder feature maps at different scales with the corresponding decoder layers to fuse low-level spatial information with high-level semantic information. In the final decoder layer, a 1x1 convolutional layer generates a set of Hough transform parameters, including the distance ρ and angle θ of the line segments. By decoding and post-processing the Hough transform parameters, the server obtains a set of semantic line segments in the vehicle sub-image. Specifically, the server first filters the Hough transform parameters based on a preset threshold to remove line segments with low confidence. The server then performs non-maximum suppression on the remaining line segments and merges line segments with high overlap. Finally, the server converts the endpoint coordinates of the line segments back to the original image coordinate system to obtain the semantic line detection results in the vehicle sub-image. After obtaining the semantic line detection results, the server further determines the upper edge of the vehicle bed area. The server can achieve this by analyzing the position of the semantic lines in the vehicle sub-image. Specifically, the server first determines the approximate extent of the vehicle bed area in the vehicle sub-image based on prior knowledge (such as the approximate position of the vehicle bed in the vehicle sub-image) or an additional vehicle bed detection model. The server then iterates over each line segment in the semantic line detection results and selects target semantic lines that lie above and parallel to the vehicle bed area based on the following conditions: the midpoint coordinates of the line segment should be in the upper half of the vehicle bed area; the angle θ of the line segment should be close to the horizontal direction, with an error not exceeding a certain threshold (e.g., 10°); and the length of the line segment should exceed a certain percentage of the vehicle bed width (e.g., 50%) to ensure that it covers the majority of the vehicle bed area.

[0055] After selecting the target semantic line, the server outputs it as the upper edge of the truck bed area. If there are multiple semantic lines that meet the requirements, the server can select the one with the highest confidence or average multiple semantic lines to obtain a more robust upper edge position estimate.

[0056] In one possible embodiment, the construction process of the preset semantic line detection model is: using annotation software to perform target box annotation on the image dataset to obtain a labeled training dataset, the method also includes: cropping the corresponding vehicle area according to the target box coordinates of the training dataset to obtain a second vehicle sub-image dataset; using the annotation software to perform semantic line segment annotation on the vehicle bed edge line of the second vehicle sub-image dataset to obtain a vehicle bed edge training dataset; using the vehicle bed edge training dataset to train the DeepHough semantic line detection model to obtain a preset semantic line detection model.

[0057] Specifically, based on the pre-set vehicle positioning model and the pre-set instance segmentation model, the server further constructs a pre-set semantic line detection model to detect semantic line segments along the edge of the cargo box in the vehicle sub-image. Unlike the pre-set instance segmentation model, which focuses on pixel-level segmentation of the cargo area, the semantic line detection model needs to identify straight or curved lines along the cargo box edge within the image, thus requiring more targeted annotation and training data. First, the server uses the same annotated dataset used to construct the vehicle positioning model in step S102 to crop the vehicle region within it, generating a new vehicle sub-image dataset, denoted as the second vehicle sub-image dataset. This step is similar to the operation used to construct the instance segmentation model; its purpose is to extract the vehicle region of interest from the original image, providing more targeted data for subsequent cargo box edge annotation. The server iterates through each annotated image, reads the vehicle bounding box coordinates, and based on these coordinates, crops the corresponding rectangular region from the original image to generate a vehicle sub-image. The cropped vehicle sub-image typically contains only one vehicle, with relatively clear outlines of the car body and cargo box edges, making it more suitable for semantic line segment annotation.

[0058] Next, the server performs semantic line segment annotation on the second vehicle sub-image dataset, capturing the edges of the cargo bed. Unlike instance segmentation, which requires outlining the pixel-level outlines of the cargo, semantic line segment annotation simply marks the edges of the cargo bed in the image, typically using simple geometric elements like line segments or Bezier curves.

[0059] The server can use the same annotation software as the previous task, using line or curve tools to annotate the top edge of the vehicle bed in each vehicle sub-image. During the annotation process, the server identifies the top edge of the vehicle bed in the image and fits it to the outline using the fewest possible line or curve elements. For cases where the bed edge is obscured or the shape is irregular, the annotator must use empirical inference and approximation.

[0060] The edge annotation information for the truck bed is typically stored as a set of line segment or curve endpoint coordinates and their corresponding semantic categories. For line segments, each segment is represented by its start and end coordinates (x1, y1, x2, y2); for Bezier curves, each curve is represented by its start, control point, and end coordinates (x1, y1, x2, y2, x3, y3).

[0061] After fully annotating the second vehicle sub-image dataset, the server generated a training dataset for the vehicle bed edge. Each vehicle sub-image corresponds to a corresponding annotation file, which records the coordinates of the line segments or curves on the upper edge of the vehicle bed in the image, as well as the corresponding semantic category. Finally, the server uses the vehicle bed edge training dataset to train the DeepHough semantic line detection model, resulting in a pre-defined semantic line detection model. DeepHough is a semantic line detection algorithm based on deep learning. It accurately detects significant line segments or curves in an image by regressing a set of Hough transform parameters.

[0062] Similar to the previous model training process, the server first divides the truck bed edge training dataset into a training set and a validation set according to a certain ratio, and converts it into the data format required by the DeepHough model. Then, the server defines the network structure and loss function of the DeepHough model, and uses the training set data to train the model. During the training process, the model continuously improves its performance in the semantic line detection task by learning the Hough transform parameters of the truck bed edge segments or curves in the image. After multiple rounds of iterative optimization, the model's performance indicators on the validation set continue to improve until the preset convergence conditions are reached. Finally, the server obtains the trained DeepHough semantic line detection model, which is used as the preset semantic line detection model for subsequent truck bed edge detection tasks. The model can accept vehicle sub-images of any size as input, and output the line segment or curve parameters of the upper edge of the truck bed in the image, as well as the corresponding confidence score.

[0063] For example, suppose a server trains a DeepHough model for line segment detection on a training dataset of vehicle bed edges. Each vehicle subimage in the training dataset is annotated with a set of line segments along the top edge of the vehicle bed, with each line segment represented by its start and end coordinates (x1, y1, x2, y2). The DeepHough model takes the vehicle subimage as input and outputs a set of Hough transform parameters (ρ, θ), where ρ represents the distance of the line segment from the origin and θ represents the angle of the line segment. By decoding and post-processing the output parameters, the line segment detection results for the top edge of the vehicle bed in the image can be obtained.

[0064] Step S106: Compare the cargo segmentation area with the upper edge position to determine whether the cargo exceeds the upper edge position.

[0065] In step S106, the cargo segmentation area is compared with the upper edge position to determine whether the cargo exceeds the upper edge position, specifically including: determining whether the contour point set of the cargo segmentation area is contained in the vehicle bucket area; if it is determined that a target contour point exists outside the vehicle bucket area, determining that the cargo exceeds the upper edge position, the target contour point being any one of the multiple contour points included in the contour point set; if it is determined that the target contour point does not exist outside the vehicle bucket area, determining that the cargo does not exceed the upper edge position.

[0066] Specifically, the server compares the cargo segmentation area obtained in step S104 with the position of the truck bed's upper edge obtained in step S105 to determine whether the cargo exceeds the upper edge of the truck bed, i.e., whether the cargo is over-gauge. This step is the final step in the entire over-gauge detection process. The final over-gauge determination result is obtained through geometric analysis of the cargo position and the truck bed's upper edge.

[0067] First, the server converts the representation of the cargo segmentation area into a set of contour points. In step S104, the cargo segmentation area is typically represented as a binary mask image, where pixel values of 1 represent the cargo area and pixel values of 0 represent the background area. To facilitate comparison with the top edge of the truck bed, the server converts the mask image into a set of contour point coordinates.

[0068] Specifically, the server uses a contour extraction algorithm to extract contours from the cargo segmentation mask. The contour extraction algorithm returns one or more contours of the cargo area, each consisting of a set of pixel coordinates. The server then collects the coordinates of all contour points to obtain a complete cargo contour point set, denoted as P.

[0069] After obtaining the cargo contour point set P, the server determines whether all points in P are located within the truck bed area. The server can achieve this by comparing the vertical coordinates of the contour points with the location of the truck bed's upper edge. Specifically, based on the location of the truck bed's upper edge obtained in step S105, the server first determines a horizontal line representing the truck bed's upper edge, denoted as L. The server then iterates over each point in the cargo contour point set P and determines whether its vertical coordinate is lower than the vertical coordinate of line L. If the vertical coordinates of all points in P are lower than the vertical coordinate of L, the cargo contour is completely located within the truck bed area and no overrun has occurred. Conversely, if the vertical coordinate of at least one point in P is higher than or equal to the vertical coordinate of L, the cargo contour extends beyond the truck bed area and an overrun has occurred.

[0070] In the latter case, the server marks points in P whose ordinates are greater than or equal to those in L as target contour points, indicating that the cargo exceeds the upper edge of the truck bed at these locations. Theoretically, the presence of any target contour point indicates that the cargo is over-limit. However, to improve the robustness of this determination, in one possible implementation, the server can set a threshold T. Only when the number of target contour points exceeds T is the cargo considered to be over-limit. This threshold T can be dynamically adjusted based on factors such as cargo size and image resolution, and is not limited in this application.

[0071] Finally, the server analyzes the target contour points and makes a final determination of whether the cargo is over-limit. If no target contour points exist, or the number of target contour points is less than or equal to a threshold value T, the server determines that the cargo has not exceeded the upper edge of the truck bed, meaning that the cargo is not over-limited. Conversely, if any target contour point exists, or the number of target contour points is greater than T, the server determines that the cargo has exceeded the upper edge of the truck bed, meaning that the cargo is over-limited.

[0072] For example, suppose the server performs cargo over-limit detection on an 800*600 vehicle sub-image. After steps S104 and S105, the server obtains the cargo segmentation mask and the location of the upper edge of the vehicle bed. The cargo segmentation mask contains a rectangular area of 200*150, representing the location of the cargo; the upper edge of the vehicle bed is at the vertical coordinate y=250. The server performs contour extraction on the cargo segmentation mask, obtaining a cargo contour point set P containing 800 points. The server then iterates over each point in P and finds that 30 points have vertical coordinates greater than 250, indicating that there are 30 target contour points. Assuming the threshold T is set to 10, the number of target contour points is greater than T, and the server determines that the cargo has exceeded the upper edge of the vehicle bed, indicating an over-limit situation.

[0073] Step S107: If it is determined that the cargo exceeds the upper edge position, the transport vehicle is determined to be an oversized vehicle.

[0074] Step S108: If it is determined that the cargo does not exceed the upper edge position, the transport vehicle is determined to be a normal vehicle.

[0075] In steps S107 and S108, the server retrieves the result from step S106 regarding whether the cargo exceeds the upper edge of the vehicle's bucket. If the result is "cargo does not exceed the upper edge of the vehicle's bucket," the vehicle's loading height complies with regulations and does not exceed the limit. The server marks the vehicle as normal and sets its over-limit status to "not exceeded." Conversely, if the result is "cargo exceeds the upper edge of the vehicle's bucket," the vehicle's loading height exceeds the bucket's height limit, indicating an over-limit situation. The server marks the vehicle as over-limit and sets its over-limit status to "over-limited."

[0076] Reference Figure 2 The present application also provides a vehicle cargo overload detection device based on semantic lines, which is a server. The server includes an image acquisition module 201, a vehicle positioning module 202, a sub-image cropping module 203, a cargo segmentation module 204, a bucket edge detection module 205 and an overload judgment module 206, wherein: the image acquisition module 201 is used to acquire an initial image of the transport vehicle; the vehicle positioning module 202 is used to identify the initial image based on a preset vehicle positioning model to obtain the initial coordinates of the transport vehicle; the sub-image cropping module 203 is used to expand and crop the initial image according to the initial coordinates to obtain a vehicle sub-image of the transport vehicle, and the vehicle sub-image includes the bucket area of the transport vehicle. ; The cargo segmentation module 204 is used to perform instance segmentation on the vehicle sub-graph, identify the cargo area within the vehicle bucket area, and output the cargo segmentation area; the vehicle bucket edge detection module 205 is used to perform semantic line detection on the vehicle sub-graph to obtain the upper edge position of the vehicle bucket area; the over-limit judgment module 206 is used to compare the cargo segmentation area with the upper edge position to determine whether the cargo exceeds the upper edge position; the over-limit judgment module 206 is also used to determine that the transport vehicle is an over-limit vehicle if it is determined that the cargo exceeds the upper edge position; the over-limit judgment module 206 is also used to determine that the transport vehicle is a normal vehicle if it is determined that the cargo does not exceed the upper edge position.

[0077] In one possible implementation, the cargo segmentation module 204 performs instance segmentation on the vehicle sub-image, identifies the cargo area within the truck bed area, and outputs the cargo segmentation area, specifically including: the cargo segmentation module 204 inputs the vehicle sub-image into a preset instance segmentation model for feature extraction to obtain a feature map of the vehicle sub-image; the cargo segmentation module 204 decodes the feature map to obtain segmentation results of each instance in the vehicle sub-image; the cargo segmentation module 204 identifies the instance located within the truck bed area based on the segmentation results, and outputs the instance located within the truck bed area as the cargo segmentation area.

[0078] In a possible implementation, the bucket edge detection module 205 performs semantic line detection on the vehicle sub-image to obtain the upper edge position of the bucket area, specifically including: the bucket edge detection module 205 inputs the vehicle sub-image into a preset semantic line detection model for feature extraction to obtain an edge feature map of the vehicle sub-image; the bucket edge detection module 205 decodes the edge feature map to obtain the semantic line in the vehicle sub-image; the bucket edge detection module 205 determines the target semantic line located above and parallel to the bucket area based on the position of the semantic line in the vehicle sub-image, and determines the target semantic line as the upper edge position of the bucket area.

[0079] In a possible implementation, the over-limit judgment module 206 compares the cargo segmentation area with the upper edge position to determine whether the cargo exceeds the upper edge position, specifically including: the over-limit judgment module 206 determines whether the contour point set of the cargo segmentation area is contained in the vehicle bucket area; if the over-limit judgment module 206 determines that there is a target contour point outside the vehicle bucket area, it determines that the cargo exceeds the upper edge position, and the target contour point is any one of the multiple contour points included in the contour point set; if the over-limit judgment module 206 determines that there is no target contour point outside the vehicle bucket area, it determines that the cargo does not exceed the upper edge position.

[0080] In one possible embodiment, before the image acquisition module 201 acquires the initial image of the transport vehicle, the method further includes: the image acquisition module 201 acquires a picture dataset of multiple vehicles; the image acquisition module 201 uses annotation software to annotate the picture dataset with target frames to obtain a labeled training dataset; the image acquisition module 201 uses the training dataset to train a YOLOv8 target detection model to obtain the preset vehicle positioning model.

[0081] In one possible implementation, after the image acquisition module 201 uses annotation software to perform target frame annotation on the image dataset to obtain a labeled training dataset, the method further includes: the image acquisition module 201 crops the corresponding vehicle area according to the target frame coordinates of the training dataset to obtain a first vehicle sub-image dataset; the image acquisition module 201 uses the annotation software to perform instance segmentation annotation on the cargo area of the first vehicle sub-image dataset to obtain a cargo segmentation training dataset; and the image acquisition module 201 uses the cargo segmentation training dataset to train a YOLOv8 instance segmentation model to obtain a preset instance segmentation model.

[0082] In a possible embodiment, the image acquisition module 201 uses annotation software to perform target frame annotation on the image dataset. After obtaining the annotated training dataset, the method further includes: the image acquisition module 201 crops the corresponding vehicle area according to the target frame coordinates of the training dataset to obtain a second vehicle sub-image dataset; the image acquisition module 201 uses the annotation software to perform semantic line segment annotation on the vehicle bed edge line of the second vehicle sub-image dataset to obtain a vehicle bed edge training dataset; the image acquisition module 201 uses the vehicle bed edge training dataset to train a DeepHough semantic line detection model to obtain a preset semantic line detection model.

[0083] It should be noted that the above embodiments provide devices that implement their functions using only the division of the above functional modules as examples. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above. In addition, the device and method embodiments provided in the above embodiments are based on the same concept. The specific implementation process is detailed in the method embodiment and will not be repeated here.

[0084] This application also provides an electronic device. Figure 3 , Figure 3 3. This is a schematic diagram of the structure of an electronic device provided by an embodiment of the present application. The electronic device 300 may include: at least one processor 301, at least one network interface 304, a user interface 303, a memory 305, and at least one communication bus 302.

[0085] The communication bus 302 is used to implement the connection and communication between these components.

[0086] The user interface 303 may include a display screen (Display) and a camera (Camera). Optionally, the user interface 303 may also include a standard wired interface and a wireless interface.

[0087] The network interface 304 may optionally include a standard wired interface or a wireless interface (such as a Wi-Fi interface).

[0088] The processor 301 may include one or more processing cores. Using various interfaces and circuits, the processor 301 connects to various components within the server. It executes instructions, programs, code sets, or instruction sets stored in the memory 305, as well as accesses data stored in the memory 305, to perform various server functions and process data. Optionally, the processor 301 may be implemented using at least one of the following hardware forms: a digital signal processing (DSP), a field-programmable gate array (FPGA), or a programmable logic array (PLA). The processor 301 may integrate one or a combination of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. The CPU primarily processes the operating system, user interface, and application programs; the GPU is responsible for rendering and drawing content displayed on the display screen; and the modem handles wireless communications. It is understood that the modem may not be integrated into the processor 301 but implemented as a separate chip.

[0089] Among them, the memory 305 may include a random access memory (RAM) or a read-only memory (Read-Only Memory). Optionally, the memory 305 includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 may include a program storage area and a data storage area, wherein the program storage area may store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playback function, an image playback function, etc.), instructions for implementing the above-mentioned various method embodiments, etc.; the data storage area may store data involved in the above-mentioned various method embodiments, etc. The memory 305 may also optionally be at least one storage device located away from the aforementioned processor 301. Refer to Figure 3 The memory 305 as a computer storage medium may include an operating system, a network communication module, a user interface module, and an application program of a vehicle cargo overload detection method based on semantic lines.

[0090] exist Figure 3 In the electronic device 300 shown, the user interface 303 is mainly used to provide an input interface for the user and obtain the data input by the user; and the processor 301 can be used to call the application program of the vehicle cargo overload detection method based on semantic lines stored in the memory 305. When executed by one or more processors 301, the electronic device 300 executes one or more of the methods described in the above embodiments. It should be noted that for the aforementioned method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should know that this application is not limited by the order of the actions described, because according to this application, certain steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also know that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required for this application.

[0091] The present application further provides a computer-readable storage medium storing instructions, which, when executed by one or more processors 301 , enable the electronic device 300 to perform one or more of the methods described in the above embodiments.

[0092] In the above embodiments, the description of each embodiment has its own focus. For parts that are not described in detail in a certain embodiment, reference can be made to the relevant descriptions of other embodiments.

[0093] In the several embodiments provided in this application, it should be understood that the disclosed devices can be implemented in other ways. For example, the device embodiments described above are merely schematic, such as the division of units, which is only a logical function division. In actual implementation, there may be other division methods, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some service interface, and the indirect coupling or communication connection of devices or units can be electrical or other forms.

[0094] Units described as separate components may or may not be physically separate, and components shown as units may or may not be physical units, that is, they may be located in one place or distributed across multiple network units. Some or all of these units may be selected to achieve the purpose of this embodiment according to actual needs.

[0095] In addition, the functional units in the various embodiments of the present application may be integrated into a single processing unit, or each unit may exist physically separately, or two or more units may be integrated into a single unit. The aforementioned integrated units may be implemented in the form of hardware or software functional units.

[0096] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable memory. Based on this understanding, the technical solution of this application, or the portion that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a memory and includes several instructions for causing a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the various embodiments of the method of this application. The aforementioned memory includes various media that can store program code, such as USB flash drives, mobile hard drives, magnetic disks, or optical disks.

[0097] The foregoing is merely an exemplary embodiment of the present disclosure and is not intended to limit the scope of the present disclosure. In other words, any equivalent variations and modifications made in accordance with the teachings of the present disclosure are still within the scope of the present disclosure. Those skilled in the art will readily conceive of other embodiments of the present disclosure after considering the disclosure and the practical implications thereof.

[0098] This application is intended to cover any variations, uses, or adaptations of the present disclosure that follow the general principles of the present disclosure and include common knowledge or customary techniques in the art not described herein. The description and examples are to be considered as exemplary only, and the scope and spirit of the present disclosure are to be defined by the claims.

Claims

1. A vehicle cargo overload detection method based on semantic lines, characterized in that: The method comprises: Acquire an initial image of the transport vehicle; Recognizing the initial image based on a preset vehicle positioning model to obtain initial coordinates of the transport vehicle; Expanding and cropping the initial image according to the initial coordinates to obtain a vehicle sub-image of the transport vehicle, wherein the vehicle sub-image includes a truck bed area of the transport vehicle; Performing instance segmentation on the vehicle subgraph, identifying the cargo area within the vehicle box area, and outputting the cargo segmentation area; Performing semantic line detection on the vehicle sub-graph to obtain the upper edge position of the vehicle bucket area; Comparing the cargo segmentation area with the upper edge position to determine whether the cargo exceeds the upper edge position; If it is determined that the cargo exceeds the upper edge position, the transport vehicle is determined to be an oversized vehicle; If it is determined that the cargo does not exceed the upper edge position, the transport vehicle is determined to be a normal vehicle.

2. The method according to claim 1, characterized in that The performing instance segmentation on the vehicle subgraph, identifying the cargo area within the vehicle compartment area, and outputting the cargo segmentation area specifically includes: Inputting the vehicle sub-image into a preset instance segmentation model to perform feature extraction, thereby obtaining a feature map of the vehicle sub-image; Decoding the feature map to obtain segmentation results for each instance in the vehicle sub-map; According to the segmentation result, the instance located in the vehicle container area is identified, and the instance located in the vehicle container area is output as the cargo segmentation area.

3. The method according to claim 1, characterized in that The performing semantic line detection on the vehicle sub-graph to obtain the upper edge position of the vehicle bucket area specifically includes: Inputting the vehicle sub-image into a preset semantic line detection model to perform feature extraction to obtain an edge feature map of the vehicle sub-image; Decoding the edge feature map to obtain semantic lines in the vehicle sub-map; According to the position of the semantic line in the vehicle sub-image, a target semantic line located above and parallel to the vehicle bucket area is determined, and the target semantic line is determined as the upper edge position of the vehicle bucket area.

4. The method according to claim 1, wherein The step of comparing the cargo segmentation area with the upper edge position to determine whether the cargo exceeds the upper edge position specifically includes: Determine whether the contour point set of the cargo segmentation area is contained in the vehicle container area; If it is determined that there is a target contour point outside the container area, then determining that the cargo exceeds the upper edge position, the target contour point is any one of the multiple contour points included in the contour point set; If it is determined that the target contour point does not exist outside the container area, it is determined that the cargo does not exceed the upper edge position.

5. The method according to claim 1, wherein Before acquiring the initial image of the transport vehicle, the method further includes: Get a dataset of multiple vehicle images; Using annotation software to annotate the image dataset with target frames to obtain an annotated training dataset; The training data set is used to train the YOLOv8 target detection model to obtain the preset vehicle positioning model.

6. The method according to claim 5, characterized in that After the image dataset is annotated with target frames using annotation software to obtain an annotated training dataset, the method further includes: According to the target frame coordinates of the training data set, the corresponding vehicle area is cropped to obtain a first vehicle sub-image data set; Using the annotation software to perform instance segmentation and annotation of the cargo area on the first vehicle sub-image dataset to obtain a cargo segmentation training dataset; The cargo segmentation training dataset is used to train the YOLOv8 instance segmentation model to obtain a preset instance segmentation model.

7. The method according to claim 5, characterized in that After the image dataset is annotated with target frames using annotation software to obtain an annotated training dataset, the method further includes: According to the target frame coordinates of the training data set, the corresponding vehicle area is cropped to obtain a second vehicle sub-image data set; Using the annotation software to perform semantic line segment annotation of the vehicle bucket edge line on the second vehicle sub-image dataset to obtain a vehicle bucket edge training dataset; The truck bed edge training data set is used to train the DeepHough semantic line detection model to obtain a preset semantic line detection model.

8. The vehicle cargo overload detection device based on semantic lines is characterized by: The device comprises an image acquisition module (201), a vehicle positioning module (202), a sub-image clipping module (203), a cargo segmentation module (204), a vehicle bucket edge detection module (205) and an over-limit judgment module (206), wherein: The image acquisition module (201) is used to acquire an initial image of the transport vehicle; The vehicle positioning module (202) is used to identify the initial image based on a preset vehicle positioning model to obtain the initial coordinates of the transport vehicle; The sub-image cropping module (203) is used to expand and crop the initial image according to the initial coordinates to obtain a vehicle sub-image of the transport vehicle, wherein the vehicle sub-image includes a truck bed area of the transport vehicle; The cargo segmentation module (204) is used to perform instance segmentation on the vehicle subgraph, identify the cargo area within the vehicle container area, and output the cargo segmentation area; The vehicle bucket edge detection module (205) is used to perform semantic line detection on the vehicle sub-graph to obtain the upper edge position of the vehicle bucket area; The overrun judgment module (206) is used to compare the cargo segmentation area with the upper edge position to judge whether the cargo exceeds the upper edge position; The over-limit judgment module (206) is further configured to determine that the transport vehicle is an over-limit vehicle if it is determined that the cargo exceeds the upper edge position; The over-limit judgment module (206) is further configured to determine that the transport vehicle is a normal vehicle if it is determined that the cargo does not exceed the upper edge position.

9. An electronic device, characterized in that: The electronic device (300) comprises a processor (301), a memory (305), a user interface (303) and a network interface (304), wherein the memory (305) is used to store instructions, the user interface (303) and the network interface (304) are used to communicate with other devices, and the processor (301) is used to execute the instructions stored in the memory (305) so that the electronic device (300) executes the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that The computer-readable storage medium stores instructions, and when the instructions are executed, the method according to any one of claims 1 to 7 is executed.

Images