Loader blind area monitoring method and device applied to engineering construction process

Abstract

The embodiment of the invention discloses a loader blind area monitoring method and device applied to the engineering construction process. A specific embodiment of the method comprises the following steps: in response to a situation that the loader is in a starting state, controlling a blind area camera mounted on the loader to collect a real-time image according to a raising angle of a loading device corresponding to the loader; the visibility grade is determined through a scattering type visibility sensor arranged in a corresponding working area of the loading machine; quickly defogging the real-time images in the real-time image group sequence according to the visibility grade; determining living obstacle information according to the defogged image group sequence; and in response to the situation that the obstacle position is located in an early warning area corresponding to the loader or the obstacle movement trend represents that a living obstacle corresponding to the living obstacle information is close to the loader, initiating blind area early warning to a driver corresponding to the loader. According to the implementation mode, the living obstacles in the blind area of the loader are effectively monitored, and the construction safety is guaranteed.

Description

Technical Field

[0001] Embodiments of the present disclosure relate to the fields of engineering and computer technology, and in particular to a method and device for monitoring blind spots of a loader used in an engineering construction process. Background Art

[0002] Loaders are a type of construction machinery widely used in engineering construction. They are mainly used for shoveling bulk materials such as sand and gravel, light excavation of hard soil, backfilling of earth, etc. Due to its high efficiency and good maneuverability, it has become one of the main equipment in the field of engineering construction. However, due to the high height of the loader itself and the obstruction of the driver's vision when the loading device is raised, the loader has a large blind spot, which affects construction safety.

[0003] The above information disclosed in this Background section is only for enhancement of understanding of the background of the inventive concept and therefore it may contain information that does not form the prior art that is already known to a person of ordinary skill in the art. Summary of the invention

[0004] The content of this disclosure is used to introduce concepts in a brief form, which will be described in detail in the detailed implementation section below. The content of this disclosure is not intended to identify the key features or essential features of the technical solution claimed for protection, nor is it intended to limit the scope of the technical solution claimed for protection.

[0005] Some embodiments of the present disclosure propose a loader blind spot monitoring method and device applied in an engineering construction process to solve the technical problems mentioned in the above background technology section.

[0006] In a first aspect, some embodiments of the present disclosure provide a method for monitoring blind spots of a loader used in an engineering construction process, the method comprising: in response to the loader being in a startup state, according to the lifting angle of the loading device corresponding to the loader, controlling the blind spot camera installed on the loader to collect real-time images to obtain a real-time image group sequence; determining the visibility level by means of a scattering visibility sensor arranged in a working area corresponding to the loader, wherein the visibility level represents the dust state of the working area; performing rapid image defogging on the real-time images in the real-time image group sequence according to the visibility level to obtain a defogged image group sequence; determining living obstacle information according to the defogged image group sequence, wherein the living obstacle information comprises: obstacle position and obstacle movement trend; in response to the obstacle position being located in the warning area corresponding to the loader, or the obstacle movement trend representing that the living obstacle corresponding to the living obstacle information is close to the loader, initiating a blind spot warning to the driver corresponding to the loader.

[0007] Second aspect, some embodiments of the present disclosure provide a blind area monitoring device applied to the engineering construction process. The device includes: a control unit configured to, in response to the loader being in a startup state, control a blind area camera installed on the loader to collect real-time images according to the lifting angle of the loading device corresponding to the loader, so as to obtain a sequence of real-time image groups; a first determination unit configured to determine a visibility level through a scattered visibility sensor arranged in the corresponding working area of the loader, where the visibility level characterizes the dust-raising state of the working area; an image fast dehazing unit configured to perform image fast dehazing on the real-time images in the sequence of real-time image groups according to the visibility level, so as to obtain a sequence of dehazed images; a second determination unit configured to determine living obstacle information according to the sequence of dehazed images, where the living obstacle information includes: obstacle position and obstacle movement trend; a blind area warning unit configured to, in response to the obstacle position being within the warning area corresponding to the loader, or the obstacle movement trend indicating that the living obstacle corresponding to the living obstacle information approaches the loader, initiate a blind area warning to the driver corresponding to the loader.

[0008] Third aspect, some embodiments of the present disclosure provide a blind area monitoring system applied to the method described in any implementation manner of the first aspect. The system includes: an image acquisition module, where the image acquisition module includes: a first blind area camera, a second blind area camera, a third blind area camera, a fourth blind area camera, a fifth blind area camera, and a sixth blind area camera, and the blind area cameras included in the image acquisition module are used to collect real-time images around the loader body; an ultra-wideband signal transmitter, where the ultra-wideband signal transmitter is used for signal interaction with an ultra-wideband positioning base station; an ultra-wideband positioning base station, where the ultra-wideband positioning base station is used to locate the signal source carrying the ultra-wideband signal transmitter; a wearable warning device, where the wearable warning device is used to initiate a position risk warning when the wearer is within the warning area or close to the loader; a visual blind area warning module, where the visual blind area warning module is arranged inside the loader cockpit to visually display the position of the living obstacle when the living obstacle is within the warning area or close to the loader.

[0009] Fourth aspect, some embodiments of the present disclosure provide an electronic device, including: one or more processors; a storage device storing one or more programs thereon, and when the one or more programs are executed by the one or more processors, the one or more processors implement the method described in any implementation manner of the first aspect.

[0010] Fifth aspect, some embodiments of the present disclosure provide a computer-readable medium with a computer program stored thereon. When the program is executed by a processor, the method described in any implementation manner of the above first aspect is implemented.

[0011] The above various embodiments of the present disclosure have the following beneficial effects: Through the loader blind area monitoring method applied to the engineering construction process in some embodiments of the present disclosure, effective monitoring of living obstacles in the blind area of the loader is achieved, ensuring construction safety. In practice, although loaders are often equipped with relatively large visible windows, due to the high body height of the loader and the line of sight obstruction caused by the lifting of the loading device for the driver, there are still quite large blind areas. When living obstacles are in the blind area, safety accidents are extremely likely to occur, thus affecting construction safety. Based on this, for the loader blind area monitoring method applied to the engineering construction process in some embodiments of the present disclosure, first, in response to the loader being in a startup state, according to the lifting angle of the loading device corresponding to the loader, control the blind area camera installed on the loader to collect real-time images, obtaining a real-time image group sequence, thereby obtaining real-time images in the blind area. Secondly, determine the visibility level through a scattering visibility sensor set in the corresponding working area of the loader, where the visibility level characterizes the dust-raising state of the working area. In practice, the working environment of the loader is complex and is often accompanied by environmental factors such as dust. Dust will affect the image quality and further affect the accurate recognition of living obstacles. Therefore, it is necessary to determine the corresponding visibility level. Then, according to the visibility level, perform fast image defogging on the real-time images in the real-time image group sequence to obtain a defogged image group sequence. By combining the visibility level, batch fast image defogging is realized, thereby improving the speed of data image defogging on the premise of ensuring the defogging effect. Further, according to the defogged image group sequence, determine the information of living obstacles, where the information of living obstacles includes: obstacle position and obstacle movement trend. Thus, living obstacle recognition is carried out. Finally, in response to the obstacle position being within the warning area corresponding to the loader, or the obstacle movement trend indicating that the living obstacle corresponding to the information of the living obstacle is approaching the loader, initiate a blind area warning to the driver corresponding to the loader. Thus, a warning is timely issued when the living obstacle approaches. Through this method, effective monitoring of living obstacles in the blind area of the loader is achieved, ensuring construction safety. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In combination with the accompanying drawings and with reference to the following specific embodiments, the above and other features, advantages and aspects of the various embodiments of the present disclosure will become more obvious. Throughout the drawings, the same or similar reference numerals represent the same or similar elements. It should be understood that the drawings are schematic and the elements and elements are not necessarily drawn to scale.

[0013] Figure 1 is a flowchart of some embodiments of a loader blind area monitoring method applied to the engineering construction process according to the present disclosure;

[0014] Figure 2 is a schematic diagram of the position corresponding to the blind area of vision and the blind area camera provided on the loader;

[0015] Figure 3 is another schematic diagram of the position of the blind area camera provided on the loader;

[0016] Figure 4 is a schematic diagram of the model structure of the defogging parameter mapping model;

[0017] Figure 5 is a schematic diagram of the generation process of defogging parameters;

[0018] Figure 6 is a schematic diagram of the model structure of the image feature extraction module included in the obstacle recognition model;

[0019] Figure 7 is a schematic diagram of the position of the warning area;

[0020] Figure 8 is a schematic diagram of the structure of some embodiments of a loader blind area monitoring device applied to the engineering construction process according to the present disclosure;

[0021] Figure 9 is a schematic diagram of the structure of an electronic device suitable for implementing some embodiments of the present disclosure. Detailed implementation manners

[0022] Embodiments of the present disclosure will be described in more detail below with reference to the accompanying drawings. Although some embodiments of the present disclosure are shown in the drawings, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments set forth herein. On the contrary, these embodiments are provided to more thoroughly and completely understand the present disclosure. It should be understood that the accompanying drawings and embodiments of the present disclosure are only for exemplary purposes and are not used to limit the protection scope of the present disclosure.

[0023] In addition, it should be noted that for the sake of convenience of description, only parts related to the relevant invention are shown in the accompanying drawings. Without conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other.

[0024] It should be noted that the concepts such as "first" and "second" mentioned in the present disclosure are only used to distinguish different devices, modules or units, and are not used to limit the order of the functions executed by these devices, modules or units or the interdependent relationship.

[0025] It should be noted that the modifications of "one" and "multiple" mentioned in this disclosure are illustrative rather than restrictive. Those skilled in the art should understand that, unless otherwise clearly specified in the context, it should be understood as "one or more".

[0026] The names of the messages or information exchanged between multiple devices in the embodiments of this disclosure are only for illustrative purposes and are not used to limit the scope of these messages or information.

[0027] The following will detail this disclosure with reference to the accompanying drawings and in conjunction with embodiments.

[0028] Reference Figure 1 shows a flow 100 of some embodiments of a loader blind spot monitoring method applied to the engineering construction process according to this disclosure. The loader blind spot monitoring method applied to the engineering construction process includes the following steps:

[0029] Step 101, in response to the loader being in a startup state, according to the lifting angle of the loading device corresponding to the loader, control the blind spot camera installed on the loader to collect real-time images, obtaining a real-time image group sequence.

[0030] In some embodiments, the execution subject (for example, a computing device) of the loader blind spot monitoring method applied to the engineering construction process can, in response to the loader being in a startup state, according to the lifting angle of the loading device corresponding to the loader, control the blind spot camera installed on the loader to collect real-time images, obtaining a real-time image group sequence. Among them, the loader is an engineering instrument used for shoveling bulk materials such as sand and gravel, slightly excavating hard soil, and backfilling earthwork, etc. The loading device is a device included in the loader for excavation and loading. In practice, the loading device is composed of components such as a bucket, a boom, a swing arm, and a tie rod. The blind spot camera is a camera used to collect images at the blind spot position of the loader's field of vision. In practice, multiple blind spot cameras can be set on the loader to collect images around the vehicle body of the loader in a moving state. Among them, each blind spot camera corresponds to a real-time image group, and thus a real-time image group sequence is collected.

[0031] As an example, refer to Figure 2 the schematic diagram of the blind spot of the field of vision and the corresponding positions of the blind spot cameras set on the loader shown, where Figure 2Four main blind spots of vision are shown, namely blind spot of vision A, blind spot of vision B, blind spot of vision C, and blind spot of vision D. Among them, first, due to the occlusion of the rear wheels and the rear of the loader, blind spot of vision A is easily formed. Secondly, due to the occlusion of the front wheels of the loader, symmetrical blind spots of vision B and D are easily formed in front of the front wheels of the loader. Then, since the bucket included in the loading device has a certain height, and as the loading operation is carried out, the loading device will frequently lift or lower, so a blind spot of vision C with a changing area will be formed in front of the loading device.

[0032] It should be noted that the above computing device can be hardware or software. When the computing device is hardware, it can be implemented as a distributed cluster composed of multiple servers or terminal devices, or as a single server or a single terminal device. When the computing device is embodied as software, it can be installed in the above-listed hardware devices. It can be implemented as, for example, multiple software or software modules for providing distributed services, or as a single software or software module. No specific limitation is made here.

[0033] Optionally, a first blind spot camera and a second blind spot camera are provided on the loading device, a third blind spot camera is provided above the cab included in the loader, and a fourth blind spot camera, a fifth blind spot camera, and a sixth blind spot camera are provided around the body of the loader.

[0034] In some optional implementation manners of some embodiments, the above execution subject controls the blind spot cameras installed on the loader to collect real-time images according to the lifting angle of the loading device corresponding to the loader, and obtains a sequence of real-time image groups, including:

[0035] In the first step, in response to the lifting angle of the loading device being less than the preset lifting angle, control the first blind spot camera, the third blind spot camera, the fourth blind spot camera, the fifth blind spot camera, and the fourth blind spot camera to synchronously collect real-time images around the loader, and obtain the sequence of real-time image groups.

[0036] In practice, in order to ensure the clock synchronization of multiple blind spot cameras during the real-time image acquisition process, therefore, by means of CAN (Controller Area Network) bus control, control the synchronous acquisition of images of the first blind spot camera, the second blind spot camera, the third blind spot camera, the fourth blind spot camera, the fifth blind spot camera, and the sixth blind spot camera.

[0037] In the second step, in response to the lifting angle of the loading device being greater than or equal to the preset lifting angle, control the second blind area camera, the third blind area camera, the fourth blind area camera, the fifth blind area camera, and the fourth blind area camera to synchronously collect real-time images around the loader to obtain the real-time image group sequence.

[0038] As an example, further refer to Figure 2 the schematic diagram of the corresponding positions of the blind areas of vision and the blind area cameras provided on the loader shown in Figure 3 and another schematic diagram of the positions of the blind area cameras provided on the loader shown in

[0039] Step 102: Determine the visibility level through a scattered visibility sensor provided in the corresponding working area of the loader.

[0040] In some embodiments, the above-mentioned execution entity can determine the visibility level through a scattering-type visibility sensor arranged in the corresponding working area of the loader. Among them, the scattering-type visibility sensor is an instrument that estimates the meteorological optical range by measuring the scattering coefficient. Specifically, the attenuation of light in the atmosphere is mainly caused by scattering and absorption. In most scenarios, the absorption situation can be ignored, and the scattering situation caused by reflection, refraction, or diffraction is the main factor affecting the attenuation of light in the atmosphere. Therefore, the scattering-type visibility sensor can estimate the meteorological optical range by measuring the scattering coefficient. Further, by mapping the estimated meteorological optical range, the corresponding visibility level is obtained. Specifically, the lower the meteorological optical range, the lower the corresponding visibility level. The higher the meteorological optical range, the higher the corresponding visibility level.

[0041] In particular, since the working area of the loader is often accompanied by dust generation, vibration, etc., the sensor for determining the visibility level needs to meet the requirements of strong environmental adaptability, low maintenance cost, and certain accuracy. Therefore, the present disclosure selects a scattering-type visibility sensor, which has relatively low implementation cost and maintenance cost compared with the transmission-type visibility sensor or the multi-sensor fusion method.

[0042] Step 103, according to the visibility level, perform fast image defogging on the real-time images in the real-time image group sequence to obtain a defogged image group sequence.

[0043] In some embodiments, the above-mentioned execution entity can perform fast image defogging on the real-time images in the real-time image group sequence according to the visibility level to obtain a defogged image group sequence. Among them, the defogging parameters include: histogram parameters. In practice, corresponding defogging parameters can be preset for different visibility levels. The above-mentioned execution entity can combine the defogging parameters corresponding to the visibility level and perform fast batch image defogging on the real-time images in the real-time image group sequence through histogram adjustment to obtain a defogged image group sequence.

[0044] In particular, during the loading and unloading process of the loader, there are often situations such as dust that affect visibility. The lower visibility will affect the image clarity of the real-time images obtained, thereby affecting the subsequent recognition of living obstacles. Using conventional real-time image noise reduction and defogging each real-time image one by one according to the image characteristics of the real-time image has low defogging efficiency and is difficult to meet the requirements of timely recognition of living obstacles in the blind area during the construction process of the loader. Combining the construction scenario of the loader, the working area of the loader can be abstracted as a homogeneous area with changing visibility (decreasing visibility) caused by dust. Therefore, the present disclosure quickly estimates the visibility level corresponding to the working area through a scattering-type visibility sensor and performs fast batch image defogging in combination with the visibility level, which can greatly improve the defogging speed on the premise of meeting the defogging effect, so as to meet the requirements of timeliness for blind area monitoring.

[0045] In some optional implementation manners of some embodiments, the above-mentioned execution subject performs fast image defogging on the real-time images in the above-mentioned real-time image group sequence according to the above-mentioned visibility level, and obtains a defogged image group sequence, including:

[0046] First step, determine the defogging parameters corresponding to the above-mentioned visibility level, where the defogging parameters corresponding to the visibility level are periodically updated through a pre-trained defogging parameter mapping model.

[0047] In practice, the defogging parameter mapping model can be trained by a supervised training method. Specifically, in the model training stage, the input of the defogging parameter mapping model is the training images labeled with the corresponding visibility levels, and the output is the defogged images corresponding to the training images. Since the working area of the loader can be abstracted as a homogeneous area with visibility changes (decrease in visibility) caused by dust, directly using the defogging parameter mapping model to perform real-time image defogging requires a large amount of computing resources and it is difficult to ensure real-time performance. Therefore, in the present disclosure, the defogging parameters corresponding to different visibility levels are obtained through a pre-trained defogging parameter mapping model, and the defogging parameters are directly applied during subsequent image defogging, so as to achieve the purpose of fast image defogging.

[0048] As an example, refer to Figure 4Schematic diagram of the model structure of the dehazing parameter mapping model shown, where the dehazing parameter mapping model adopts a symmetric structure similar to U-Net. Specifically, the downsampling side part of the dehazing parameter mapping model includes: 3 image processing blocks, where 1 downsampling layer is set between every 2 image processing blocks. The upsampling side part includes: 3 image processing blocks, where 1 upsampling layer is set between every 2 image processing blocks. 1 image processing block is set in the middle of the downsampling side and the upsampling side. A convolutional layer is set at the input end of the dehazing parameter mapping model for shallow image feature extraction of the real-time image. A convolutional layer is set at the output end of the dehazing parameter mapping model for image detail restoration. Between every two downsampling layers, between every two upsampling layers, and between the last downsampling layer on the downsampling side and the first upsampling layer on the upsampling side, a skip connection method is adopted to superimpose the output of the corresponding previous multi-branch encoding layer and the output of the previous downsampling layer (upsampling layer) as the input of the current downsampling side (upsampling layer). The image processing block includes: a multi-scale patch embedding layer and a multi-branch transformer block. Among them, the multi-scale patch embedding layer is used to convert an image or a feature map into multi-scale token tags. Specifically, the multi-scale patch embedding layer uses deformable convolution to obtain different visual features. In particular, during the deformable convolution process, multiple small convolutional kernels are used, which can reduce the number of parameters and the computational burden compared with the method of using multiple large convolutional kernels. The multi-branch transformer block includes multiple Transformer encoders for parallel encoding of the multi-scale token tags output by the multi-scale patch embedding layer. After the multi-branch transformer block encodes the multi-scale token tags, the output features of the Transformer encoders corresponding to different branches are feature fused as the input of the downsampling layer (or upsampling layer). Through the multi-scale patch embedding layer, multi-scale image patch embedding can be realized, which can extract more hierarchical visual features compared with the single-scale method. At the same time, parallel encoding is realized through the multi-branch transformer block to improve the encoding efficiency. Considering that the model structure of the dehazing parameter mapping model is relatively shallow, and in order to reduce the amount of data processing to improve the data processing speed, the 2 image processing blocks at the symmetric positions on the upsampling side part and the downsampling side part abandon the skip connection method. In particular, through the above dehazing parameter mapping model, the dehazing parameters corresponding to different visibility levels can be updated during the non-working period of the loader.

[0049] In the second step, according to the above dehazing parameters, batch image dehazing is performed on the real-time images in the above real-time image group sequence to obtain the above dehazed image group sequence.

[0050] In practice, combined with the construction scenario of a loader, the working area of the loader can be abstracted as a homogeneous area where the visibility changes (decreases) due to dust. Therefore, a batch image dehazing method can be adopted to quickly dehaze the real-time images in the real-time image group sequence, and obtain the dehazed image group sequence.

[0051] As an example, refer to Figure 5 the schematic diagram of the generation process of the dehazing parameters shown in the figure. Among them, during the non-working period of the loader, the dehazing parameter mapping model takes the (real-time) image 501 with a known corresponding visibility level as the input of the dehazing parameter mapping model 502, obtains the dehazed image 503, and takes the histogram parameter 504 corresponding to the dehazed image 503 as the dehazing parameter corresponding to the visibility level corresponding to the (real-time) image 501.

[0052] The content in the above "in some optional implementation manners of some embodiments" is a core inventive point of the present disclosure. By setting the dehazing parameter mapping model, the extraction of dehazing parameters for different visibility levels can be quickly realized, and combined with the dehazing parameters, the rapid batch dehazing of real-time images can be achieved. Usually, this method can obtain accurate dehazing parameters, and at the same time, compared with the method of directly dehazing images by combining models, the image dehazing speed is greatly improved.

[0053] Step 104, determine the information of living obstacles according to the dehazed image group sequence.

[0054] In some embodiments, the above execution subject can determine the information of living obstacles according to the dehazed image group sequence. Among them, the information of living obstacles includes: the position of the obstacle and the movement trend of the obstacle. The position of the obstacle represents the position of the obstacle in the dehazed image. The movement trend of the obstacle represents the distance change trend of the living obstacle from the loader. The movement trend of the obstacle can include: a far-away trend, a close-up trend, and a relatively static trend. The far-away trend represents that the living obstacle is far away from the loader. The close-up trend represents that the living obstacle is close to the loader. The relatively static trend represents that the distance between the living obstacle and the loader remains relatively unchanged. In practice, the information of living obstacles can be determined according to the dehazed image group sequence by using a model such as Tiny-YOLO.

[0055] In some optional implementation manners of some embodiments, the above execution subject determines the information of living obstacles according to the above dehazed image group sequence, including:

[0056] The first step, for each dehazed image group in the above dehazed image group sequence, perform batch image feature extraction through the image feature extraction module included in the pre-trained obstacle recognition model to obtain an image feature group.

[0057] Among them, the above image feature extraction module includes: a backbone module and K skip extraction modules. The backbone module is used to extract complete and different-scale image features from the first dehazed image in the dehazed image group, and the K skip extraction modules are used to perform skip-type image feature extraction on the dehazed images in the dehazed image group except the first dehazed image.

[0058] As an example, refer to Figure 6 The schematic diagram of the model structure of the image feature extraction module included in the obstacle recognition model shown in. Among them, the backbone module included in the image feature extraction module is composed of 5 serially connected convolutional layer blocks (convolutional layer block B1, convolutional layer block B2, convolutional layer block B3, convolutional layer block B4, and convolutional layer block B5). Each convolutional layer block contains 3 convolutional layers. Among them, convolutional layer block B1 includes: convolutional layer B11, convolutional layer B12, and convolutional layer B13. Convolutional layer block B2 includes: convolutional layer B21, convolutional layer B22, and convolutional layer B23. Convolutional layer block B3 includes: convolutional layer B31, convolutional layer B32, and convolutional layer B33. Convolutional layer block B4 includes: convolutional layer B41, convolutional layer B42, and convolutional layer B43. Convolutional layer block B5 includes: convolutional layer B51, convolutional layer B52, and convolutional layer B53. Each of the skip extraction module S1, skip extraction module S2, skip extraction module S3, skip extraction module S4, and skip extraction module S5 includes 1 convolutional layer. Specifically, skip extraction module S1 includes: convolutional layer B11. Skip extraction module S2 includes: convolutional layer B21. Skip extraction module S3 includes: convolutional layer B31. Skip extraction module S4 includes: convolutional layer B41. Skip extraction module S5 includes convolutional layer B51.

[0059] As another example, taking the dehazed image group including the dehazed image I1 and the dehazed image I2 as an example. First, the backbone module will perform image feature extraction on the dehazed image I1 through the included convolutional layer blocks B1, B2, B3, B4, and B5, and obtain 5 feature maps, namely: feature map BI11, feature map BI21, feature map BI31, feature map BI41, and feature map BI51. Among them, the feature map BI11 is the feature map output by the convolutional layer B11 included in the convolutional layer block B1. The feature map BI21 is the feature map output by the convolutional layer B21 included in the convolutional layer block B2. The feature map BI31 is the feature map output by the convolutional layer B31 included in the convolutional layer block B3. The feature map BI41 is the feature map output by the convolutional layer B41 included in the convolutional layer block B4. The feature map BI51 is the feature map output by the convolutional layer B51 included in the convolutional layer block B5. For the dehazed image I2, first, the convolutional layer B11 included in the skip extraction module S1 will perform image feature extraction on the dehazed image I2 to obtain the feature map S1. Then, the execution body will calculate the graph similarity between the feature map S1 and the feature map BI11. If the graph similarity is less than the preset threshold, the image feature extraction of the dehazed image will start from the convolutional layer block B1 included in the backbone module. Secondly, if the graph similarity is greater than the preset threshold, the convolutional layer B21 included in the skip extraction module S2 will perform image feature extraction on the dehazed image I2 to obtain the feature map S2. Then, the execution body will calculate the graph similarity between the feature map S2 and the feature map BI21. If the graph similarity is less than the preset threshold, the dehazed image I2 will be reduced to the input size of the convolutional block B2, and then the image feature extraction will start from the convolutional block B2. Next, if the graph similarity is greater than the preset threshold, the convolutional layer B31 included in the skip extraction module S3 will perform image feature extraction on the dehazed image I2 to obtain the feature map S3. Then, the execution body will calculate the graph similarity between the feature map S3 and the feature map BI31. If the graph similarity is less than the preset threshold, the dehazed image I2 will be reduced to the input size of the convolutional block B3, and then the image feature extraction will start from the convolutional block B3. Further, if the graph similarity is greater than the preset threshold, the convolutional layer B41 included in the skip extraction module S4 will perform image feature extraction on the dehazed image I2 to obtain the feature map S4. Then, the execution body will calculate the graph similarity between the feature map S4 and the feature map BI41. If the graph similarity is less than the preset threshold, the dehazed image I2 will be reduced to the input size of the convolutional block B4, and then the image feature extraction will start from the convolutional block B4. Finally, if the graph similarity is greater than the preset threshold, the convolutional layer B51 included in the skip extraction module S5 will perform image feature extraction on the dehazed image I2 to obtain the feature map S5. Then, the execution body will calculate the graph similarity between the feature map S5 and the feature map BI51. If the graph similarity is less than the preset threshold, the dehazed image I2 will be reduced to the input size of the convolutional block B5, and then the image feature extraction will start from the convolutional block B5.

[0060] In practice, due to the inter-frame similarity between the real-time images collected by the (blind area) camera, if image feature extraction is performed on all real-time images in their entirety (all through the complete backbone module), it will result in a huge amount of data processing, affecting the recognition efficiency. Therefore, by setting up a skipping module, for each group of dehazed image groups, taking the first dehazed image as a reference, only the differential features of the remaining dehazed images are concerned, and image feature extraction is performed in a skipping manner to reduce the amount of data processing and thereby improve the recognition speed.

[0061] In the second step, according to the obtained sequence of image feature groups, the above-mentioned living obstacle information is determined through the positioning model included in the above-mentioned obstacle recognition model.

[0062] In practice, the positioning model may include: an RPN (Region Proposal Network) detection head. For each group of image feature maps, through the RPN detection head, it can be determined whether each image feature contains a living obstacle, and thus the position of the obstacle can be recognized. Then, according to the characteristic of frame-by-frame continuity between images, the movement trend of the obstacle is determined by means of the optical flow method.

[0063] Step 105, in response to the position of the obstacle being within the warning area corresponding to the loader, or the movement trend of the obstacle indicating that the living obstacle corresponding to the living obstacle information is approaching the loader, a blind area warning is issued to the driver corresponding to the loader.

[0064] In some embodiments, in response to the position of the obstacle being within the warning area corresponding to the loader, or the movement trend of the obstacle indicating that the living obstacle corresponding to the living obstacle information is approaching the loader, the above-mentioned execution entity may issue a blind area warning to the driver corresponding to the loader.

[0065] Optionally, the above-mentioned warning area includes: a first-level warning area and a second-level warning area, where the above-mentioned first-level warning area represents a preset area corresponding to the loader along the traveling direction, and the above-mentioned second-level warning area represents a preset area around the body of the loader that is not the first-level warning area.

[0066] As an example, refer to Figure 7 the schematic diagram of the position of the warning area shown, where, along the traveling direction, the first-level warning area is located in front of the above-mentioned loader. The second-level warning area is the area in the circular area centered on the loader excluding the first-level warning area. Specifically, when the traveling direction of the loader changes, for example, when the loader is reversing, the first-level warning area is located at the rear part of the loader.

[0067] In some optional implementation manners of some embodiments, when the above-mentioned execution entity responds that the position of the above-mentioned obstacle is within the early warning area corresponding to the above-mentioned loader, or the movement trend of the above-mentioned obstacle indicates that the living obstacle corresponding to the above-mentioned living obstacle information approaches the above-mentioned loader, a blind area early warning is initiated to the driver corresponding to the above-mentioned loader, including:

[0068] In the first step, in response to the position of the above-mentioned obstacle indicating that the living obstacle is within the first-level early warning area, a vehicle braking prompt is initiated to the driver corresponding to the above-mentioned loader.

[0069] In practice, the vehicle braking prompt can be in the form of a sound prompt, such as a voice prompt.

[0070] In the second step, in response to the driver corresponding to the above-mentioned loader not initiating a vehicle braking action within a preset time period, an emergency braking for the above-mentioned loader is automatically initiated.

[0071] Among them, the above-mentioned preset time period is dynamically determined according to the distance of the obstacle and the vehicle speed of the loader. In practice, the above-mentioned execution entity can link the emergency braking system of the loader to automatically control the loader to perform emergency braking.

[0072] In the third step, in response to the position of the above-mentioned obstacle indicating that the living obstacle is within the second-level early warning area, a blind area obstacle early warning is initiated to the driver corresponding to the above-mentioned loader.

[0073] In practice, the blind area obstacle early warning can be in the form of a sound prompt, such as a beep prompt.

[0074] In the fourth step, in response to the position of the above-mentioned obstacle indicating that the living obstacle is not within the above-mentioned early warning area, and the movement trend of the above-mentioned obstacle indicates that the living obstacle corresponding to the above-mentioned living obstacle information approaches the above-mentioned loader, a blind area obstacle approaching early warning is initiated to the driver corresponding to the above-mentioned loader.

[0075] In practice, the blind area obstacle approaching early warning can also be in the form of a visual prompt, such as through a visual blind area early warning module set inside the cab of the loader for visual early warning. The visual blind area early warning module can include: a visual screen.

[0076] Optionally, the above method further includes:

[0077] In the first step, the position of the loader corresponding to the above-mentioned loader is determined through an ultra-wideband positioning base station.

[0078] In practice, the loader can be equipped with an ultra-wideband signal transmitter. After the ultra-wideband positioning base station receives the ultra-wideband signal transmitted by the ultra-wideband signal transmitter, the position of the loader is determined through parameters such as the time of flight of the signal and the incident angle.

[0079] Step 2: Determine the region of interest based on the above loader position and the above obstacle position.

[0080] In practice, since the loader position is known and the obstacle position of the living obstacle relative to the loader is known, the region of interest where the living obstacle is located can be determined by coordinate transformation. In particular, the obstacle coordinates are image coordinates and the loader position is three-dimensional coordinates. Therefore, the blind area camera needs to be calibrated in advance, and then the obstacle coordinates are transformed into the same coordinate system as the loader position through coordinate transformation to obtain the region of interest.

[0081] Step 3: Determine whether the above ultra-wideband positioning base station receives an ultra-wideband signal emitted from within the above region of interest.

[0082] In practice, the ultra-wideband signal in any region can estimate the location of the region through parameters such as signal flight time and incident angle. Therefore, it can be queried whether the ultra-wideband positioning base station receives an ultra-wideband signal emitted from within the region with coordinates consistent with those of the region of interest, so as to determine whether an ultra-wideband signal emitted from within the above region of interest is received.

[0083] Step 4: In response to receiving the above ultra-wideband signal and the device type of the signal source corresponding to the above ultra-wideband signal being a wearable warning device, send a warning instruction to the above wearable warning device through the above ultra-wideband positioning base station, so that a location risk warning is initiated to the living obstacle corresponding to the above living obstacle information through the above wearable warning device.

[0084] The above-mentioned various embodiments of the present disclosure have the following beneficial effects: Through the blind area monitoring method of a loader applied in the engineering construction process according to some embodiments of the present disclosure, effective monitoring of living obstacles in the blind area of the loader is achieved, ensuring construction safety. In practice, although loaders are often equipped with relatively large visible windows, due to the high body height of the loader and the fact that the loading device blocks the driver's line of sight when it is lifted, there are still quite large blind areas. When living obstacles are located in the blind area, safety accidents are extremely likely to occur, thus affecting construction safety. Based on this, for the blind area monitoring method of a loader applied in the engineering construction process according to some embodiments of the present disclosure, first, in response to the loader being in a startup state, according to the lifting angle of the loading device corresponding to the above-mentioned loader, control the blind area camera installed on the above-mentioned loader to collect real-time images, obtaining a real-time image group sequence, thereby obtaining real-time images in the blind area. Secondly, determine the visibility level through a scattering-type visibility sensor set in the corresponding working area of the above-mentioned loader, where the above-mentioned visibility level represents the dust-raising state of the above-mentioned working area. In practice, the working environment of the loader is complex, often accompanied by environmental conditions such as dust, and the dust will affect the image quality, and further affect the accurate identification of living obstacles. Therefore, it is necessary to determine the corresponding visibility level. Then, according to the above-mentioned visibility level, perform fast image defogging on the real-time images in the above-mentioned real-time image group sequence, obtaining a defogged image group sequence. By combining the visibility level, batch fast image defogging is achieved, thereby improving the data image defogging speed while ensuring the defogging effect. Further, according to the above-mentioned defogged image group sequence, determine the information of living obstacles, where the above-mentioned information of living obstacles includes: obstacle position and obstacle movement trend. Thereby, the identification of living obstacles is carried out. Finally, in response to the above-mentioned obstacle position being within the warning area corresponding to the above-mentioned loader, or the above-mentioned obstacle movement trend indicating that the living obstacle corresponding to the above-mentioned information of living obstacles is approaching the above-mentioned loader, initiate a blind area warning to the driver corresponding to the above-mentioned loader. Thereby, timely warning is given when the living obstacle approaches. Through this method, effective monitoring of living obstacles in the blind area of the loader is achieved, ensuring construction safety.

[0085] Secondly, the present disclosure discloses a blind spot monitoring system. The blind spot monitoring system includes: an image acquisition module, an ultra-wideband signal transmitter, an ultra-wideband positioning base station, a wearable warning device, and a visual warning module. Among them: The image acquisition module, where the image acquisition module includes: a first blind spot camera, a second blind spot camera, a third blind spot camera, a fourth blind spot camera, a fifth blind spot camera, and a sixth blind spot camera. The blind spot cameras included in the image acquisition module are used to collect real-time images around the loader body. The ultra-wideband signal transmitter, where the ultra-wideband signal transmitter is used for signal interaction with the ultra-wideband positioning base station. Specifically, the ultra-wideband signal transmitter can be set on the loader and the wearable warning device. The ultra-wideband positioning base station, where the ultra-wideband positioning base station is used to locate the signal source carrying the ultra-wideband signal transmitter. The wearable warning device, where the wearable warning device is used to initiate a position risk warning when the wearer is within the warning area or close to the loader. In practice, the wearable warning device can include, but is not limited to: an ultra-wideband signal transmitter, a sound reminder, and a light reminder. The visual blind spot warning module, where the visual blind spot warning module is set inside the loader cockpit to visually display the position of the living obstacle when the living obstacle is within the warning area or close to the loader. In practice, the visual blind spot warning module can include, but is not limited to: a visual screen, a sound reminder, and a light reminder.

[0086] Further referring to Figure 8 , as an implementation of the methods shown in the above figures, the present disclosure provides some embodiments of a loader blind spot monitoring device applied to the engineering construction process. These device embodiments correspond to Figure 1 the method embodiments shown, and the loader blind spot monitoring device applied to the engineering construction process can be specifically applied to various electronic devices.

[0087] As Figure 8As shown in the figure, the blind area monitoring device 800 applied to the engineering construction process in some embodiments includes: a control unit 801, a first determination unit 802, an image fast dehazing unit 803, a second determination unit 804, and a blind area warning unit 805. Among them, the control unit 801 is configured to, in response to the loader being in a startup state, control the blind area camera installed on the loader to collect real-time images according to the lifting angle of the loading device corresponding to the loader, so as to obtain a real-time image group sequence; the first determination unit 802 is configured to determine the visibility level through a scattering type visibility sensor arranged in the corresponding working area of the loader, where the visibility level represents the dust emission state of the working area; the image fast dehazing unit 803 is configured to perform fast image dehazing on the real-time images in the real-time image group sequence according to the visibility level, so as to obtain a dehazed image group sequence; the second determination unit 804 is configured to determine the living obstacle information according to the dehazed image group sequence, where the living obstacle information includes: the obstacle position and the obstacle movement trend; the blind area warning unit 805 is configured to, in response to the obstacle position being within the warning area corresponding to the loader, or the obstacle movement trend indicating that the living obstacle corresponding to the living obstacle information approaches the loader, initiate a blind area warning to the driver corresponding to the loader.

[0088] It can be understood that the units described in the blind area monitoring device 800 applied to the engineering construction process correspond to the respective steps in the method described in the reference Figure 1 description. Therefore, the operations, features, and beneficial effects described above for the method are equally applicable to the blind area monitoring device 800 applied to the engineering construction process and the units included therein, and will not be elaborated herein.

[0089] Next, refer to Figure 9 , which shows a schematic structural diagram of an electronic device (for example, a computing device) suitable for use in implementing some embodiments of the present disclosure. Figure 9 The electronic device shown is only an example and should not impose any limitations on the functions and usage scopes of the embodiments of the present disclosure. As Figure 9As shown, the computer device includes a processor, a memory, and a network interface connected via a system bus. Among them, the memory may include a non-volatile storage medium and an internal memory. The non-volatile storage medium can store an operating system and computer programs. The computer programs include program instructions, which, when executed, can cause the processor to execute any of the above methods. The processor is used to provide computing and control capabilities to support the operation of the entire computer device. The internal memory provides an environment for the operation of the computer programs in the non-volatile storage medium, and when the computer programs are executed by the processor, the processor can be caused to execute any of the above methods. The network interface is used for network communication, such as sending the assigned tasks, etc. Those skilled in the art can understand that Figure 9 the structure shown in is only a block diagram of some structures related to the solution of the present disclosure, and does not constitute a limitation on the computer device to which the solution of the present disclosure is applied. The specific computer device may include more or fewer components than those shown in the figure, or combine some components, or have different component arrangements.

[0090] It should be understood that the processor may be a central processing unit (CPU), and the processor may also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. Among them, the general-purpose processor may be a microprocessor or the processor may also be any conventional processor, etc.

[0091] Among them, in one embodiment, the above-mentioned processor is used to run a computer program stored in the memory to implement the following steps: in response to the loader being in a startup state, according to the lifting angle of the loading device corresponding to the loader, control the blind area camera installed on the loader to collect real-time images, and obtain a real-time image group sequence; determine the visibility level through a scattering visibility sensor set in the corresponding working area of the loader, where the visibility level characterizes the dust-raising state of the working area; according to the visibility level, perform fast image defogging on the real-time images in the real-time image group sequence to obtain a defogged image group sequence; according to the defogged image group sequence, determine the information of living obstacles, where the information of living obstacles includes: obstacle position and obstacle movement trend; in response to the obstacle position being within the warning area corresponding to the loader, or the obstacle movement trend indicating that the living obstacle corresponding to the information of living obstacles approaches the loader, initiate a blind area warning to the driver corresponding to the loader.

[0092] The embodiments of the present disclosure also provide a computer-readable storage medium, on which a computer program is stored, and the computer program includes program instructions. The method implemented when the program instructions are executed can refer to the various embodiments of the above-mentioned method of the present disclosure.

[0093] Among them, the above-mentioned computer-readable storage medium may be an internal storage unit of the computer device in the foregoing embodiment, such as the hard disk or memory of the computer device. The above-mentioned computer-readable storage medium may also be an external storage device of the computer device, such as a plug-in hard disk equipped on the computer device, a Smart Media Card (SMC), a Secure Digital (SD) card, a Flash Card, etc.

[0094] It should be noted that in this article, the term "including", "comprising" or any other variant thereof is intended to cover non-exclusive inclusion, so that a process, method, article or system including a series of elements not only includes those elements, but also includes other elements not explicitly listed, or further includes elements inherent to such process, method, article or system. Without further limitations, the element defined by the statement "including a..." does not exclude the existence of another identical element in the process, method, article or system including the element.

[0095] The above description is only some preferred embodiments of the present disclosure and an explanation of the technical principles applied. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by the specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the technical solutions formed by mutually replacing the above features with the technical features (but not limited to) disclosed in the embodiments of the present disclosure that have similar functions.

Claims

1. A method for monitoring the blind area of a loader during the engineering construction process, characterized in that, Including: In response to the loader being in a startup state, according to the lifting angle of the loading device corresponding to the loader, control the blind area camera installed on the loader to collect real-time images, and obtain a sequence of real-time image groups; Determine the visibility level through a scattering type visibility sensor arranged in the corresponding working area of the loader, where the visibility level characterizes the dust raising state of the working area; According to the visibility level, perform fast image defogging on the real-time images in the sequence of real-time image groups to obtain a sequence of defogged images; According to the sequence of defogged images, determine the information of living obstacles, where the information of living obstacles includes: obstacle position and obstacle movement trend; In response to the obstacle position being within the warning area corresponding to the loader, or the obstacle movement trend indicating that the living obstacle corresponding to the information of the living obstacle approaches the loader, initiate a blind area warning to the driver corresponding to the loader.

2. The method according to claim 1, characterized in that, The warning area includes: a primary warning area and a secondary warning area, where the primary warning area represents a preset area corresponding to the loader along the traveling direction, and the secondary warning area represents a preset area around the loader body that is not the primary warning area; and The step of, in response to the obstacle position being within the warning area corresponding to the loader, or the obstacle movement trend indicating that the living obstacle corresponding to the information of the living obstacle approaches the loader, initiating a blind area warning to the driver corresponding to the loader, includes: In response to the obstacle position indicating that the living obstacle is within the primary warning area, initiate a vehicle braking prompt to the driver corresponding to the loader; In response to the driver corresponding to the loader not initiating a vehicle braking action within a preset duration, automatically initiate an emergency braking for the loader, where the preset duration is dynamically determined according to the obstacle distance and the vehicle speed of the loader; In response to the obstacle position indicating that the living obstacle is within the secondary warning area, initiate a blind area obstacle warning to the driver corresponding to the loader; In response to the obstacle position indicating that the living obstacle is not within the warning area, and the obstacle movement trend indicating that the living obstacle corresponding to the information of the living obstacle approaches the loader, initiate a blind area obstacle approaching warning to the driver corresponding to the loader.

3. The method according to claim 2, wherein The method further includes: Determine the position of the loader corresponding to the loader through an ultra-wideband positioning base station; Determine the region of interest according to the loader position and the obstacle position; Determine whether the ultra-wideband positioning base station receives an ultra-wideband signal emitted from within the region of interest; In response to receiving the ultra-wideband signal and the device type of the signal source corresponding to the ultra-wideband signal being a wearable warning device, send a warning instruction to the wearable warning device through the ultra-wideband positioning base station, so as to initiate a position risk warning to the living obstacle corresponding to the information of the living obstacle through the wearable warning device.

4. The method according to claim 3, wherein A first blind - area camera and a second blind - area camera are provided on the loading device. A third blind - area camera is provided on the upper side of the cab included in the loader, and fourth, fifth, and sixth blind - area cameras are provided around the body of the loader. And Controlling the blind - area cameras installed on the loader to collect real - time images according to the lifting angle of the loading device corresponding to the loader, and obtaining a real - time image group sequence, including: In response to the lifting angle of the loading device being less than the preset lifting angle, controlling the first blind - area camera, the third blind - area camera, the fourth blind - area camera, the fifth blind - area camera, and the fourth blind - area camera to synchronously collect real - time images around the loader, and obtaining the real - time image group sequence; In response to the lifting angle of the loading device being greater than or equal to the preset lifting angle, controlling the second blind - area camera, the third blind - area camera, the fourth blind - area camera, the fifth blind - area camera, and the fourth blind - area camera to synchronously collect real - time images around the loader, and obtaining the real - time image group sequence.

5. The method according to claim 4, wherein Performing fast image defogging on the real - time images in the real - time image group sequence according to the visibility level, and obtaining a defogged image group sequence, including: Determining the defogging parameters corresponding to the visibility level, where the defogging parameters corresponding to the visibility level are regularly updated through a pre - trained defogging parameter mapping model; Performing batch image defogging on the real - time images in the real - time image group sequence according to the defogging parameters, and obtaining the defogged image group sequence.

6. The method according to claim 5, wherein Determining living obstacle information according to the defogged image group sequence, including: For each defogged image group in the defogged image group sequence, performing batch image feature extraction through an image feature extraction module included in a pre - trained obstacle recognition model to obtain an image feature group, where the image feature extraction module includes: a backbone module and K skip - extraction modules. The backbone module is used to extract complete and different - scale image features from the first defogged image in the defogged image group, and the K skip - extraction modules are used to perform skip - type image feature extraction on the defogged images other than the first defogged image in the defogged image group; According to the obtained image feature group sequence, determining the living obstacle information through a positioning model included in the obstacle recognition model.

7. A blind spot monitoring system, characterized in that, Applied to the method according to any one of claims 1 to 6, including: An image acquisition module, where the image acquisition module includes: a first blind - area camera, a second blind - area camera, a third blind - area camera, a fourth blind - area camera, a fifth blind - area camera, and a sixth blind - area camera. The blind - area cameras included in the image acquisition module are used to collect real - time images around the body of the loader; An ultra - wideband signal transmitter, where the ultra - wideband signal transmitter is used for signal interaction with an ultra - wideband positioning base station; An ultra - wideband positioning base station, where the ultra - wideband positioning base station is used to locate the signal source carrying the ultra - wideband signal transmitter; A wearable warning device, wherein the wearable warning device is used to initiate a location risk warning when the wearer is within the warning area or near the loader. A visual blind area warning module, wherein the visual blind area warning module is arranged inside the loader cockpit to visually display the position of the living obstacle when the living obstacle is within the warning area or near the loader.

8. A blind area monitoring device for loaders applied during the engineering construction process, characterized in that, Comprising: A control unit configured to, in response to the loader being in a startup state, control a blind area camera installed on the loader to collect real-time images according to the raising angle of the loading device corresponding to the loader, and obtain a real-time image group sequence. A first determination unit configured to determine the visibility level through a scattering type visibility sensor arranged in the corresponding working area of the loader, wherein the visibility level characterizes the dust emission state of the working area. An image fast dehazing unit configured to perform fast image dehazing on the real-time images in the real-time image group sequence according to the visibility level, and obtain a dehazed image group sequence. A second determination unit configured to determine living obstacle information according to the dehazed image group sequence, wherein the living obstacle information includes: the position of the obstacle and the movement trend of the obstacle. A blind area warning unit configured to, in response to the position of the obstacle being within the warning area corresponding to the loader, or the movement trend of the obstacle indicating that the living obstacle corresponding to the living obstacle information is approaching the loader, initiate a blind area warning to the driver corresponding to the loader.

9. An electronic device, characterized in that, Comprising: One or more processors; A storage device having one or more programs stored thereon; When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 6.

10. A computer-readable medium, characterized in that, Having a computer program stored thereon, wherein when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images