An engineering vehicle remote control method and device, a remote control device and a medium

Abstract

The application relates to the field of remote sensing, in particular to an engineering vehicle remote control method and device, a remote control equipment and a medium. The method comprises the following steps: acquiring a monitoring video corresponding to a target engineering vehicle in real time, and performing detailed analysis on the monitoring video, so as to obtain a vehicle form of the target engineering vehicle and a current environment feature, acquiring a BIM (building information model) model corresponding to the target engineering vehicle, and updating the BIM model based on the real-time vehicle form and the environment feature, so that the BIM model is no longer a static and fixed model, but a dynamic model capable of reflecting changes of the engineering vehicle and the environment in real time, then displaying the updated BIM model on a display module, and enhancing the accuracy and intuitiveness of remote control, so that an operator can perform accurate operation and control as if he is on the scene even if he is far away from the engineering vehicle, the number of on-site operators is reduced, and then the labor cost is reduced.

Description

Technical Field

[0001] The present application relates to the field of remote control sensing, and in particular to a method, apparatus, remote control device and medium for remotely controlling an engineering vehicle. Background Art

[0002] Construction vehicles are the backbone of a construction project, primarily used for transportation, excavation, and repairs, making them essential tools on construction sites. Common types of construction vehicles include heavy-duty transport vehicles, large cranes, excavators, bulldozers, rollers, loaders, power repair vehicles, and emergency rescue vehicles. Currently, most construction vehicles are controlled by an operator inside the vehicle's cab. However, driver control of the vehicle requires not only driving skills but also the operator's presence inside the cab, incurring significant labor costs.

[0003] With technological advancements, the concept of remote intelligent control of construction vehicles has emerged, allowing operators to control them using remote devices such as joysticks or controllers. However, to more effectively implement control and ensure operational safety, operators often need to operate the vehicles directly from relatively close distances using joysticks and controllers. This approach does not reduce the number of on-site operators and, therefore, does not reduce labor costs. Summary of the Invention

[0004] In order to reduce the manpower cost in operating an engineering vehicle, the present application provides an engineering vehicle remote control method, apparatus, remote control equipment and medium.

[0005] In a first aspect, the present application provides a method for remotely controlling an engineering vehicle, which adopts the following technical solutions:

[0006] A remote control method for an engineering vehicle, comprising:

[0007] Acquire a surveillance video corresponding to the target engineering vehicle in real time, wherein the surveillance video includes the complete target engineering vehicle and the environment;

[0008] Analyzing the surveillance video to obtain the vehicle shape and current environment characteristics corresponding to the target engineering vehicle;

[0009] Obtaining the BIM model corresponding to the target engineering vehicle;

[0010] Based on the vehicle form and the current environmental characteristics, the BIM model is updated, and the updated BIM model is displayed on a display module.

[0011] By employing the above technical solution, real-time surveillance video of the target construction vehicle is acquired and analyzed in detail to determine the target vehicle's vehicle shape and current environmental characteristics. A corresponding BIM (Building Information Model) model is then generated and updated based on the real-time vehicle shape and environmental characteristics. This model is no longer static and fixed, but rather a dynamic model that reflects real-time changes in the construction vehicle and its environment. The updated BIM model is then displayed on a display module, allowing operators to understand the current state of the construction vehicle (e.g., whether it is in normal working order, whether there is any damage or abnormality), as well as the characteristics of the surrounding environment (e.g., terrain, obstacles, other vehicles, or pedestrians). This intuitive visual display significantly enhances the accuracy and intuitiveness of remote control, allowing operators to perform precise operations and control as if they were on-site, even from a distance from the construction vehicle. This reduces the number of on-site operators and thus reduces labor costs. This also improves operational efficiency and significantly enhances safety, mitigating potential risks caused by information lags or misjudgments.

[0012] In one possible implementation, analyzing the surveillance video to obtain the vehicle shape and current environment characteristics corresponding to the target engineering vehicle includes:

[0013] Segmenting the surveillance video into a plurality of image frames, and detecting the target engineering vehicle in each image frame to obtain vehicle features in each image frame;

[0014] Based on the vehicle features in each of the image frames, separating the vehicle features from the corresponding image frame to obtain a vehicle foreground and an environmental background;

[0015] Associating the vehicle foreground corresponding to each of the image frames to obtain the vehicle shape corresponding to the surveillance video;

[0016] Based on the environmental background corresponding to each of the image frames, current environmental features corresponding to the surveillance video are determined.

[0017] By employing this technical solution, the surveillance video is segmented into multiple image frames and the target construction vehicle is detected in each frame, accurately capturing the vehicle's detailed features, such as size, shape, and color. Furthermore, by correlating the vehicle foreground within each frame, the target construction vehicle's continuous form within the surveillance video can be tracked. This helps understand the vehicle's motion trajectory, speed changes, and potential behavior patterns, providing a reliable data foundation for subsequent vehicle behavior analysis or anomaly detection. Because each image frame is individually processed and correlated, recognition errors caused by poor video quality (such as blur and jitter) are reduced, thereby improving the accuracy of vehicle form recognition.

[0018] In one possible implementation, the vehicle features include shape sub-features, color sub-features, and size sub-features. Detecting the target engineering vehicle in each image frame to obtain the vehicle features in each image frame includes:

[0019] Extracting the target engineering vehicle in each image frame based on a contour extraction function to obtain a vehicle contour corresponding to each image frame, and determining a shape sub-feature corresponding to each image frame based on the vehicle contour corresponding to each image frame;

[0020] Converting the pixels of each sub-image corresponding to the vehicle outline into an HSV color space, and calculating the HSV histogram of the pixels in each sub-image to obtain a color sub-feature corresponding to each image frame;

[0021] Obtaining a size ratio corresponding to the image frame, where the size ratio is used to represent a ratio between an actual size and an image size;

[0022] Identifying a first size of each of the vehicle outlines, and calculating a second size corresponding to each of the vehicle outlines based on the size ratio to obtain a size sub-feature corresponding to each of the image frames;

[0023] The vehicle features in each of the image frames are determined based on the shape sub-features, color sub-features, and size sub-features corresponding to each of the image frames.

[0024] By employing the above technical solution, the contour of the target construction vehicle in each image frame is extracted based on a contour extraction function, accurately capturing the vehicle's shape characteristics. This helps distinguish different types of construction vehicles, such as excavators, loaders, and road rollers. By converting the pixels of the sub-image corresponding to the vehicle contour into the HSV color space and calculating the HSV histogram, the vehicle's color characteristics can be stably obtained. Compared to the RGB color space, the HSV color space has better color stability under varying lighting conditions and can therefore more accurately reflect the vehicle's true color. Furthermore, by obtaining the size ratio of the image frame and calculating the actual size (second size) of the vehicle contour based on this ratio, the vehicle's dimensional characteristics can be accurately obtained. Combining the three sub-features of shape, color, and size can comprehensively and accurately describe the characteristics of the target construction vehicle, providing reliable data support for subsequent vehicle identification, tracking, and behavior analysis.

[0025] In one possible implementation, determining the current environmental features corresponding to the surveillance video based on the environmental background corresponding to each image frame includes:

[0026] Use feature point detection algorithm to identify the salient elements in each environmental background and calculate the descriptor corresponding to each salient element;

[0027] Extract texture features and color features from each environmental background;

[0028] Based on the texture features, color features, and descriptors corresponding to salient elements in each of the environmental backgrounds, current environmental features corresponding to the surveillance video are determined.

[0029] By adopting the above technical solution and a feature point detection algorithm, salient elements in the environmental background, such as buildings, trees, and road signs, can be accurately identified, facilitating subsequent analysis and understanding of environmental features. For each identified salient element, a corresponding descriptor is calculated. These descriptors contain key information such as the element's shape, size, and position. The precise calculation of these descriptors makes subsequent environmental features more accurate and reliable in subsequent analysis and comparison. Texture and color features are further extracted from the environmental background. Texture features reflect the surface structure and details of the environment, while color features provide information on the overall hue and color distribution of the environment. Because this solution combines multiple features, including salient elements, texture, and color, it maintains high recognition accuracy and robustness even under certain environmental conditions, such as lighting changes and occlusion.

[0030] In a possible implementation, associating the vehicle foreground corresponding to each image frame to obtain the vehicle shape corresponding to the surveillance video includes:

[0031] Based on the surveillance video, a plurality of the image frames are sequenced to obtain a target sequence;

[0032] According to the target sequence and the vehicle foreground corresponding to each image frame, a tracking algorithm is used to associate the vehicle foregrounds to obtain the vehicle shape corresponding to the monitoring video.

[0033] By employing this technical solution, multiple image frames are arranged in chronological order to form a target sequence, ensuring the temporal continuity of the vehicle's morphology. This facilitates the observation and analysis of key information such as the vehicle's position, speed, and direction at different points in time, providing reliable data support for subsequent vehicle behavior analysis or anomaly detection. A tracking algorithm is used to correlate the vehicle's foreground in each image frame, accurately reconstructing the vehicle's complete morphology within the surveillance video. This morphological reconstruction encompasses not only the vehicle's appearance but also its motion trajectory and dynamic changes within the video. This is crucial for understanding vehicle behavior patterns, predicting its future location, and determining whether it exhibits abnormal behavior.

[0034] In a possible implementation, before updating the BIM model, the method further includes:

[0035] receiving a connection signal with an engineering vehicle, and determining the engineering vehicle corresponding to the connection signal as a target engineering vehicle;

[0036] Acquire a current image corresponding to the target engineering vehicle;

[0037] Determine the vehicle coating and the environmental coating based on the current image, and establish a three-dimensional coordinate system;

[0038] A BIM model corresponding to the target engineering vehicle is established based on the vehicle coating, the environmental coating and the three-dimensional coordinate system.

[0039] By adopting the above technical solution, the actual status of the vehicle can be captured in real time by receiving connection signals from the construction vehicle and obtaining the current image corresponding to the target construction vehicle. Based on the vehicle coating and environmental coating determined by the current image and the established three-dimensional coordinate system, a precise three-dimensional model of the target vehicle can be constructed. This precise construction method ensures that the BIM model is highly consistent with the actual situation, providing a reliable foundation for subsequent construction management and maintenance. As construction vehicles move and the construction environment changes, the BIM model needs to be constantly updated to reflect the latest conditions. By continuously receiving connection signals and obtaining the current image, key information such as the vehicle coating, environmental coating, and three-dimensional coordinate system in the BIM model can be updated in real time. This real-time update capability ensures that the BIM model always remains synchronized with the actual construction situation, improving the efficiency and accuracy of construction management.

[0040] In one possible implementation, the method further includes:

[0041] Receive the corresponding operation instructions of the operator;

[0042] Determining a standard vehicle action corresponding to the operation instruction, and determining an actual vehicle action corresponding to the operation instruction;

[0043] Calculating the similarity between the standard vehicle action and the actual vehicle action;

[0044] And based on the similarity, it is determined whether to generate an operation abnormality signal.

[0045] By employing this technical solution, the standard vehicle motion corresponding to an operation instruction and the actual vehicle motion can be determined, and the accuracy of the operation can be assessed by calculating the similarity between them. This comparison method can promptly detect deviations or errors in the operation process, thereby avoiding potential safety hazards. When the similarity between the actual vehicle motion and the standard vehicle motion falls below a preset threshold, the system generates an operation abnormality signal. This early warning mechanism can prompt the operator to adjust the operation in a timely manner to ensure that the vehicle operates as expected, thereby greatly improving the safety and accuracy of the operation.

[0046] In a second aspect, the present application provides a remote control device for an engineering vehicle, which adopts the following technical solution:

[0047] A remote control device for an engineering vehicle, comprising:

[0048] A video acquisition module is used to acquire a monitoring video corresponding to a target engineering vehicle in real time, wherein the monitoring video includes the complete target engineering vehicle and the environment;

[0049] An analysis module is used to analyze the monitoring video to obtain the vehicle shape corresponding to the target engineering vehicle and the current environment characteristics;

[0050] A model acquisition module is used to obtain the BIM model corresponding to the target engineering vehicle;

[0051] The display module is used to update the BIM model based on the vehicle form and the current environmental characteristics, and display the updated BIM model on the display module.

[0052] In a third aspect, the present application provides a remote control device, which adopts the following technical solution:

[0053] A remote control device, comprising:

[0054] Display module, used to display the BIM model;

[0055] at least one processor;

[0056] Memory;

[0057] At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute the engineering vehicle remote control method described in the first aspect above.

[0058] In a fourth aspect, the present application provides a computer-readable storage medium, which adopts the following technical solution:

[0059] A computer-readable storage medium includes: storing a computer program that can be loaded by a processor and execute the engineering vehicle remote control method described in the first aspect.

[0060] In summary, this application has the following beneficial technical effects:

[0061] By acquiring real-time surveillance video of the target construction vehicle and conducting detailed analysis, the system determines the vehicle's current state and environmental characteristics. This system then updates the corresponding BIM (Building Information Model) model based on these real-time changes, transforming the BIM model from a static, fixed one into a dynamic model that reflects real-time changes in the vehicle and its surroundings. The updated BIM model is then displayed on a display module, allowing operators to understand the vehicle's current state (e.g., whether it is operating normally, whether it has any damage or anomalies), as well as surrounding characteristics (e.g., terrain, obstacles, other vehicles, or pedestrians). This intuitive visualization significantly enhances the accuracy and intuitiveness of remote control, allowing operators to operate and control the system with the same precision as if they were on-site, even from a distance. This reduces the number of on-site operators and, consequently, reduces labor costs. This improves operational efficiency and significantly enhances safety, mitigating potential risks caused by information lags or misjudgments. BRIEF DESCRIPTION OF THE DRAWINGS

[0062] Figure 1 This is a schematic diagram of the connection and interaction between a remote control device and a terminal on an engineering vehicle provided in an embodiment of the present application;

[0063] Figure 2 This is a flow chart of a method for remotely controlling an engineering vehicle provided in an embodiment of the present application;

[0064] Figure 3 1 is a block diagram of a remote control device for an engineering vehicle provided in an embodiment of the present application;

[0065] Figure 4 This is a schematic diagram of a remote control device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0066] The following is combined with Figure 1-4 This application is described in further detail.

[0067] To make the purpose, technical solutions, and advantages of the embodiments of this application more clear, the technical solutions in the embodiments of this application will be clearly and completely described below in conjunction with the drawings in the embodiments of this application. Obviously, the described embodiments are part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative efforts are within the scope of protection of this application.

[0068] In order to facilitate understanding of the technical solutions proposed in this application, several elements that will be introduced in the description of this application are first introduced here. It should be understood that the following introduction is only for the convenience of understanding these elements, so as to understand the content of the embodiments of this application, and does not necessarily cover all possible situations.

[0069] Remote control cars rely primarily on radio technology and electromagnetic wave transmission to enable remote control of the vehicle. Specifically, a remote control system primarily consists of a remote control and a vehicle control unit. The remote control is a handheld device consisting of a transmitter and a receiver (in this case, the part of the handheld device that transmits signals, not the device that receives feedback from the vehicle). The transmitter transmits radio signals via buttons or joystick controls, which are then transmitted via an antenna. The vehicle control unit primarily includes a receiver, decoder, control module, and actuators (such as motors and servos). The receiver, built into the vehicle, receives signals from the transmitter via the antenna. The decoder decodes the received signals and converts them into electrical signals. The control module controls various vehicle functions, such as forward, reverse, left, and right turns, based on the received signals. The actuators, in turn, drive the vehicle to perform corresponding actions based on the control module's instructions.

[0070] Construction vehicles are the backbone of a construction project, primarily used for transportation, excavation, and repairs, making them essential tools on construction sites. Common types of construction vehicles include heavy-duty transport vehicles, large cranes, excavators, bulldozers, rollers, loaders, power repair vehicles, and emergency rescue vehicles. Currently, most construction vehicles are controlled by an operator inside the vehicle's cab. However, driver control of the vehicle requires not only driving skills but also the operator's presence inside the cab, incurring significant labor costs.

[0071] With technological advancements, the concept of remote intelligent control of construction vehicles has emerged, allowing operators to control them using remote devices such as joysticks or controllers. However, to more effectively implement control and ensure operational safety, operators often need to operate the vehicles directly from relatively close distances using joysticks and controllers. This approach does not reduce the number of on-site operators and, therefore, does not reduce labor costs.

[0072] In view of this, an embodiment of the present application provides a method for remotely controlling an engineering vehicle. The method is performed by a remote control device, which is equipped with a transmitter, a receiver, and a display screen. Each engineering vehicle is equipped with a receiver. Figure 1 A remote control device can be connected to at least one engineering vehicle 10 via wireless communication technology, such as Bluetooth, Wi-Fi, ZigBee, or more advanced communication technology, to enable remote control of the engineering vehicle via the remote control device. The remote control device can be an electronic device, such as a mobile phone, tablet, or computer, and is not limited in this embodiment of the present application.

[0073] The present application embodiment provides a remote control method for an engineering vehicle, such as Figure 2 As shown, the method provided in the embodiment of the present application is executed by a remote control device, which can be a server or a terminal device, wherein the server can be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services. The terminal device can be a smartphone, tablet computer, laptop computer, desktop computer, etc., but is not limited to these. The terminal device and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiment of the present application. The method includes steps S201 to S204, wherein:

[0074] Step S201: Acquire the surveillance video corresponding to the target engineering vehicle in real time.

[0075] The surveillance video includes the entire target construction vehicle and its surroundings. Specifically, each construction vehicle is equipped with multiple image acquisition devices, each capable of video surveillance. The image acquisition devices on the target construction vehicle can capture surveillance video that includes the entire construction vehicle and its surroundings.

[0076] Before the remote control device controls a construction vehicle, it obtains information about each construction vehicle connected to the remote control device and receives a selection signal to determine a construction vehicle to be controlled from among the multiple construction vehicles connected to the remote control device. This construction vehicle to be controlled is the target construction vehicle. Furthermore, after determining the target construction vehicle, the remote control device receives, in real time, surveillance video captured by each image acquisition device on the target construction vehicle to obtain surveillance video corresponding to the target construction vehicle.

[0077] Step S202: Analyze the monitoring video to obtain the vehicle shape corresponding to the target engineering vehicle and the current environment characteristics.

[0078] The vehicle form includes the target vehicle's shape and operating state. The operating state includes a driving sub-state and an operating sub-state. The driving sub-state can be driving or stopped. The operating sub-state is related to the type of construction vehicle, and different construction vehicles have different corresponding operating states. For example, if the target construction vehicle is an excavator, the target construction vehicle's operating sub-states may include bucket digging, bucket stationary, and bucket other operations. Therefore, each construction vehicle type corresponds to a set of operating sub-states, and the operating sub-state corresponding to the target construction vehicle belongs to the corresponding set of operating sub-states.

[0079] The current environment features include road surface sub-features and obstacle sub-features.

[0080] Specifically, after receiving surveillance video of a target construction vehicle, a deep learning algorithm can be used to perform real-time analysis of the surveillance video to obtain the corresponding vehicle morphology and current environmental characteristics of the target construction vehicle. More specifically, the received surveillance video of the target construction vehicle is frame-processed to obtain multiple frames of surveillance images. Each two adjacent frames of surveillance images are input into a trained deep network model in chronological order, and the vehicle morphology and current environmental characteristics output by the deep network model are obtained. The deep network model is trained using a large number of adjacent frame image samples, vehicle morphology samples, and current environmental characteristic samples.

[0081] Step S203: Obtain the BIM model corresponding to the target engineering vehicle.

[0082] After determining the target construction vehicle corresponding to the remote control device, the surveillance video corresponding to the target construction vehicle is received in real time, and a BIM model corresponding to the target construction vehicle is established based on the surveillance video. That is, the target construction vehicle corresponds to a BIM model, and the BIM model corresponding to the target construction vehicle can be directly obtained.

[0083] Step S204: updating the BIM model based on the vehicle form and current environmental characteristics, and displaying the updated BIM model on the display module.

[0084] After obtaining the vehicle shape and current environmental characteristics of the target engineering vehicle, the BIM model can be updated based on the vehicle shape and current environmental characteristics so that the BIM model displays the current state of the target engineering vehicle, making it easier for the operator to operate. Specifically, the vehicle shape data obtained by video analysis is compared with the vehicle shape in the BIM model to identify the parts of the vehicle shape in the BIM model that need to be updated, such as whether the height and width of the vehicle have changed, whether the color of the vehicle is consistent with that in the video, etc. At the same time, the environmental feature data obtained by video analysis is compared with the environmental information in the BIM model to identify changes in environmental features that need to be reflected in the BIM model, such as whether the road surface is smooth and whether there are new obstacles.

[0085] Furthermore, based on the results of video analysis, the vehicle's shape in the BIM model is dynamically adjusted. For example, if the video shows an increase in vehicle height, the vehicle height in the BIM model is adjusted accordingly; if the vehicle's color changes, the material properties in the BIM model are updated. Similarly, tags or attributes related to environmental characteristics are added or updated in the BIM model. For example, if the video shows a slippery road surface, a slippery road tag is added to the BIM model; if a new obstacle appears in the video, the obstacle's location and shape information is added to the BIM model accordingly.

[0086] Furthermore, after the BIM model of the target engineering vehicle is updated, the updated BIM model is displayed on a display module of the remote control device.

[0087] The present invention provides a method for remotely controlling a construction vehicle. This method acquires real-time surveillance video of a target construction vehicle and analyzes it in detail to determine the target vehicle's vehicle configuration and current environmental characteristics. A BIM (Building Information Model) model corresponding to the target construction vehicle is then acquired and updated based on the real-time vehicle configuration and environmental characteristics. This model is no longer static and fixed, but rather a dynamic model that reflects real-time changes in the construction vehicle and its environment. The updated BIM model is then displayed on a display module, allowing operators to understand the current configuration of the construction vehicle (e.g., whether it is in normal working order, whether it has any damage or abnormalities, etc.) and the characteristics of the surrounding environment (e.g., terrain, obstacles, other vehicles, or pedestrians). This intuitive visual display significantly enhances the accuracy and intuitiveness of remote control, allowing operators, even remotely located from the construction vehicle, to perform precise operations and control as if they were on-site, thereby reducing the number of on-site operators and, consequently, lowering labor costs. This method also improves operational efficiency and significantly enhances operational safety, mitigating potential risks caused by information lags or misjudgments.

[0088] In a possible implementation of the embodiment of the present application, in the step S202, the monitoring video is analyzed to obtain the vehicle shape corresponding to the target engineering vehicle and the current environment feature, including:

[0089] The monitoring video is segmented into multiple image frames, and the target engineering vehicle in each image frame is detected to obtain the vehicle feature in each image frame;

[0090] Based on the vehicle feature in each image frame, the vehicle feature is separated from the corresponding image frame to obtain the vehicle foreground and the environment background;

[0091] The vehicle foreground corresponding to each image frame is associated to obtain the vehicle shape corresponding to the monitoring video;

[0092] Based on the environment background corresponding to each image frame, the current environment feature corresponding to the monitoring video is determined.

[0093] After obtaining the monitoring video of the target engineering vehicle, the preprocessed monitoring video can be segmented into multiple continuous image frames to facilitate subsequent extraction of the vehicle shape and the current environment feature, wherein one image frame corresponds to one time point in the monitoring video.

[0094] Further, after obtaining the segmented image frames, the target engineering vehicle and the environment in which the target engineering vehicle is located in each image frame can be detected by using a target detection algorithm. Specifically, the processing process for each image frame can be: obtaining the color and material corresponding to the target engineering vehicle, and identifying the target engineering vehicle in the image frame based on the color and material of the target engineering vehicle by using the target detection algorithm to obtain the overall vehicle feature of the target engineering vehicle in the image frame. After obtaining the vehicle feature in the image frame, the recognized target engineering vehicle can be segmented along the edge to obtain the vehicle foreground and the environment background in the image frame. Wherein, one engineering vehicle corresponds to one color and material.

[0095] Further, after obtaining the vehicle foreground corresponding to each image frame, the vehicle foreground corresponding to each image frame can be associated, the vehicle foreground in the image frames between adjacent frames is analyzed based on the time continuity between the image frames, whether the vehicle foreground between adjacent frames is completely consistent is determined, if completely consistent, the vehicle shape of the target engineering vehicle is obtained based on the edge of the vehicle foreground, and the driving sub-state and the working sub-state of the target engineering vehicle are determined as not driving and not working; when the vehicle foreground between adjacent frames is inconsistent, the inconsistent vehicle components are determined, and the driving sub-state and the working sub-state of the target engineering vehicle are determined based on the inconsistent vehicle components, so as to obtain the complete vehicle shape of the target engineering vehicle in the monitoring video.

[0096] More specifically, based on the inconsistent vehicle component, the process of determining the target construction vehicle's driving sub-state and working sub-state can be as follows: if the inconsistent vehicle component includes a wheel, the target construction vehicle's driving sub-state is determined to be driving; if the inconsistent vehicle component does not include a wheel, the target construction vehicle's driving sub-state is determined to be non-driving; if the inconsistent vehicle component includes a working component, the target construction vehicle's working sub-state is determined to be working; if the inconsistent vehicle component does not include a working component, the target construction vehicle's working sub-state is determined to be non-working. Different types of construction vehicles correspond to different working components.

[0097] Furthermore, based on the environmental background corresponding to each image frame, road features and obstacle features in the environmental background are identified to obtain current environmental features in the surveillance video.

[0098] In one possible implementation of the embodiment of the present application, in the above embodiment, when the vehicle features include shape sub-features, color sub-features, and size sub-features, detecting the target engineering vehicle in each image frame to obtain the vehicle features in each image frame includes:

[0099] Extract the target engineering vehicle in each image frame based on the contour extraction function to obtain the vehicle contour corresponding to each image frame, and determine the shape sub-feature corresponding to each image frame based on the vehicle contour corresponding to each image frame;

[0100] Convert the pixels of the sub-image corresponding to each vehicle outline into the HSV color space and calculate the HSV histogram of the pixels in each sub-image to obtain the color sub-features corresponding to each image frame;

[0101] Obtain the size ratio corresponding to the image frame, which is used to represent the ratio between the actual size and the image size;

[0102] Identifying a first size of each vehicle outline and calculating a second size corresponding to each vehicle outline based on the size ratio to obtain a size sub-feature corresponding to each image frame;

[0103] The vehicle features in each image frame are determined based on the shape sub-features, color sub-features, and size sub-features corresponding to each image frame.

[0104] Specifically, contour extraction function in image processing library (such as findContours function in OpenCV) can be used to perform contour extraction on the target engineering vehicle in each image frame, so as to identify and extract the contour of all objects in the image, and then filter out the contour matched with the target engineering vehicle. Then, shape features are calculated based on the extracted vehicle contour. For example, the perimeter, area, aspect ratio of the circumscribed rectangle and other parameters of the contour can be calculated as shape sub-features.

[0105] The pixels of the sub-image corresponding to each vehicle contour are converted from RGB color space to HSV color space. HSV color space is more suitable for color feature extraction because it decomposes color information into hue (Hue), saturation (Saturation) and value (Value) three independent components. By counting the number of pixels with different values in each HSV channel, the HSV histogram of the pixels in each sub-image is calculated to obtain the statistical information of the color distribution, and the statistical information of the color distribution is taken as the color sub-feature of the sub-image corresponding to the vehicle contour.

[0106] The size ratio of the image frame, i.e. the ratio between the actual size and the image size, is obtained, and the first size of each vehicle contour, i.e. the size in the image (such as the width and height of the circumscribed rectangle of the contour), is identified. Then, based on the size ratio, the first size is converted into the actual size, i.e. the second size, which is the size sub-feature.

[0107] Further, by splicing these feature vectors together, the shape sub-features, color sub-features and size sub-features corresponding to each image frame are fused to form a complete vehicle feature vector. The extracted vehicle features are stored in a feature database for subsequent vehicle recognition or classification. In the recognition stage, the similarity between the vehicle features of the image frame to be identified and the features in the feature database can be calculated to determine the type or identity of the vehicle in the image frame to be identified.

[0108] One possible implementation of an embodiment of the present application is that, in the above embodiment, the current environment feature corresponding to the monitoring video is determined based on the environment background corresponding to each image frame, including:

[0109] A feature point detection algorithm is used to identify the salient elements in each environment background, and a descriptor corresponding to each salient element is calculated;

[0110] Texture features and color features in each environment background are extracted;

[0111] The current environment feature corresponding to the monitoring video is determined based on the texture features, color features and descriptors corresponding to the salient elements in each environment background.

[0112] Among them, salient elements are elements with prominent colors, shapes or other features in the environment. Salient elements are more likely to be obstacles in the environment. Therefore, salient elements in the environmental background can be identified to obtain the current environmental features in the surveillance video.

[0113] Specifically, a feature point detection algorithm (such as SIFT, SURF, or ORB) is applied to the background portion of each image frame to automatically detect the feature points of salient elements in the image. For each detected feature point, a corresponding descriptor is calculated. A descriptor is a vector containing information about the image surrounding the feature point, describing its uniqueness and positional relationship.

[0114] Furthermore, texture analysis techniques (such as gray-level co-occurrence matrix and local binary pattern) are used to extract texture features from each environmental background. Color space conversion and color histogram methods are also used to extract color features from each environmental background. Texture features describe the arrangement and distribution of pixels in an image, reflecting the image's visual texture. Color features describe the distribution and proportion of colors in an image, providing important clues for distinguishing different environments.

[0115] Furthermore, by concatenating feature vectors, the descriptors corresponding to salient elements, texture features, and color features are fused to form a comprehensive feature vector that contains multiple environmental information. This fused feature vector is then analyzed to identify the current environmental characteristics in the surveillance video.

[0116] A possible implementation of the embodiment of the present application, in the above embodiment, is to associate the vehicle foreground corresponding to each image frame to obtain the vehicle shape corresponding to the surveillance video, including:

[0117] Based on the surveillance video, multiple image frames are sequenced to obtain the target sequence;

[0118] According to the target sequence and the vehicle foreground corresponding to each image frame, the tracking algorithm is used to associate the various vehicle foregrounds to obtain the vehicle shape corresponding to the surveillance video.

[0119] After obtaining the surveillance video of the target engineering vehicle, the surveillance video can be parsed and split into a series of continuous image frames. Then, from the parsed image frames, image frames containing the foreground of the target engineering vehicle are selected and sorted in the chronological order of their appearance in the video to form a target sequence.

[0120] Furthermore, based on the characteristics of the surveillance video and the motion characteristics of the target engineering vehicle, a tracking algorithm (feature matching-based tracking, template matching-based tracking, detection-based tracking, etc.) is employed to correlate the vehicle foreground within each frame, leveraging the temporal continuity between image frames and the similarity of the vehicle foreground. Based on the selected tracking algorithm, the vehicle foreground is tracked for each image frame in the target sequence, resulting in a continuous sequence containing the vehicle's position and motion trajectory at different time points. Furthermore, based on this correlated vehicle foreground sequence, the vehicle shape corresponding to the surveillance video is constructed by superimposing, fusing, or interpolating the vehicle foreground within each image frame to form a complete, continuous three-dimensional vehicle shape or motion trajectory, thereby obtaining the vehicle shape corresponding to the surveillance video.

[0121] In a possible implementation of the embodiment of the present application, before the above step S204 of updating the BIM model, the method may further include:

[0122] receiving a connection signal with an engineering vehicle, and determining the engineering vehicle corresponding to the connection signal as a target engineering vehicle;

[0123] Get the current image corresponding to the target engineering vehicle;

[0124] Based on the current image, determine the vehicle coating and the environmental coating and establish a three-dimensional coordinate system;

[0125] Based on vehicle coating, environmental coating and three-dimensional coordinate system, a BIM model corresponding to the target engineering vehicle is established.

[0126] Once the remote control device establishes a current control connection with a construction vehicle, it identifies the construction vehicle as the target vehicle and uses a camera mounted on the target vehicle or a remote monitoring camera to capture a current image of the target vehicle. The current image should clearly show the exterior of the target vehicle, including details such as coating color and texture.

[0127] Furthermore, after obtaining a current image of the target construction vehicle, the image is processed, using image processing techniques (such as color recognition and texture analysis) to determine the color, texture, and other characteristics of the vehicle coating and environmental coating. A three-dimensional coordinate system is then constructed based on the acquired image information. This three-dimensional coordinate system accurately reflects the real-world position, orientation, and size of the target construction vehicle. Calibration can be performed using information such as feature points, edges, or the dimensions of known objects in the image. For example, the three-dimensional coordinate system can be based on the target construction vehicle's left front tire as its origin, with the positive x-axis pointing east to west, the positive y-axis pointing east to north, and the positive z-axis pointing from the ground to the sky.

[0128] Furthermore, a basic BIM model framework is initialized based on basic information such as the target vehicle's type and size. By adjusting material properties, color, texture, and other parameters in the BIM model, the identified vehicle coating and environmental coating characteristics are mapped to the corresponding components in the BIM model. The established 3D coordinate system is then matched to the BIM model to ensure that the positions of the components in the BIM model in 3D space are consistent with those of the target vehicle in the real world, thus obtaining the corresponding BIM model of the target vehicle.

[0129] In a possible implementation of the embodiment of the present application, based on the above embodiment, the method may further include: receiving an operation instruction corresponding to an operator;

[0130] Determine the standard vehicle action corresponding to the operating instruction, and determine the actual vehicle action corresponding to the operating instruction;

[0131] Calculating the similarity between the standard vehicle action and the actual vehicle action;

[0132] And based on the similarity, it is determined whether to generate an operation abnormality signal.

[0133] Each operation instruction corresponds to a standard vehicle maneuver. Standard maneuvers are a set of predefined standard actions based on factors such as vehicle design, operating specifications, and safety requirements. Each operation step performed by the operator on the remote control device is considered an operation instruction. The remote control device transmits the operation instruction to the target engineering vehicle, controlling it to perform specific maneuvers such as forward, reverse, steering, acceleration, and deceleration.

[0134] When an operation instruction sent by an operator on a remote control device is received, the standard vehicle action corresponding to the operation instruction is obtained according to the received operation instruction, and the actual action state of the target engineering vehicle is obtained.

[0135] Furthermore, key features are extracted from standard actions and actual actions. These features can be time series data of physical quantities such as position, velocity, acceleration, etc., or processed feature vectors.

[0136] Specific algorithms (such as dynamic time warping, cosine similarity, and Euclidean distance) are used to calculate the similarity between the standard motion features and the actual motion features. A higher similarity value indicates a closer match between the actual motion and the standard motion; a lower similarity value indicates a greater difference between the actual motion and the standard motion.

[0137] If the calculated similarity falls below a set threshold, the operator's operation is deemed significantly different from the standard action, potentially constituting an operational anomaly. In this case, an operational anomaly signal is generated and notified to the operator through audio and visual alarms, display screen prompts, and other means. The similarity threshold is set based on actual needs and safety requirements to determine whether the difference between the actual action and the standard action reaches an abnormal level.

[0138] The above embodiment introduces a remote control method for an engineering vehicle from the perspective of a method flow, and the following embodiment introduces a remote control device for an engineering vehicle from the perspective of a virtual module or a virtual unit. Please refer to the following embodiment for details.

[0139] See also Figure 3 The engineering vehicle remote control device 30 may specifically include: a video acquisition module 301, an analysis module 302, a model acquisition module 303 and a display module 304. Specifically:

[0140] A remote control device 30 for an engineering vehicle, comprising:

[0141] The video acquisition module 301 is used to acquire the monitoring video corresponding to the target engineering vehicle in real time, and the monitoring video includes the complete target engineering vehicle and the environment;

[0142] An analysis module 302 is used to analyze the surveillance video to obtain the vehicle shape corresponding to the target engineering vehicle and the current environment characteristics;

[0143] The model acquisition module 303 is used to obtain the BIM model corresponding to the target engineering vehicle;

[0144] The display module 304 is used to update the BIM model based on the vehicle shape and current environmental characteristics, and display the updated BIM model on the display module.

[0145] In one possible implementation of the embodiment of the present application, the analysis module 302 analyzes the surveillance video to obtain the vehicle shape and current environmental characteristics corresponding to the target engineering vehicle, including:

[0146] The surveillance video is divided into multiple image frames, and the target engineering vehicle in each image frame is detected to obtain the vehicle features in each image frame;

[0147] Based on the vehicle features in each image frame, the vehicle features are separated from the corresponding image frame to obtain the vehicle foreground and the environmental background;

[0148] Correlate the vehicle foreground corresponding to each image frame to obtain the vehicle shape corresponding to the surveillance video;

[0149] Based on the environmental background corresponding to each image frame, the analysis module 302 determines the current environmental features corresponding to the surveillance video.

[0150] In one possible implementation of the embodiment of the present application, the vehicle features include shape sub-features, color sub-features, and size sub-features. The analysis module 302 detects the target engineering vehicle in each image frame to obtain the vehicle features in each image frame, including:

[0151] Extract the target engineering vehicle in each image frame based on the contour extraction function to obtain the vehicle contour corresponding to each image frame, and determine the shape sub-feature corresponding to each image frame based on the vehicle contour corresponding to each image frame;

[0152] Convert the pixels of the sub-image corresponding to each vehicle outline into the HSV color space and calculate the HSV histogram of the pixels in each sub-image to obtain the color sub-features corresponding to each image frame;

[0153] Obtain the size ratio corresponding to the image frame, which is used to represent the ratio between the actual size and the image size;

[0154] Identifying a first size of each vehicle outline and calculating a second size corresponding to each vehicle outline based on the size ratio to obtain a size sub-feature corresponding to each image frame;

[0155] The vehicle features in each image frame are determined based on the shape sub-features, color sub-features, and size sub-features corresponding to each image frame.

[0156] In one possible implementation of the embodiment of the present application, the analysis module 302 determines the current environmental features corresponding to the surveillance video based on the environmental background corresponding to each image frame, including:

[0157] Use feature point detection algorithm to identify the salient elements in each environmental background and calculate the descriptor corresponding to each salient element;

[0158] Extract texture features and color features from each environmental background;

[0159] Based on the texture features, color features and descriptors corresponding to salient elements in each environmental background, the current environmental features corresponding to the surveillance video are determined.

[0160] In one possible implementation of the embodiment of the present application, the analysis module 302 associates the vehicle foreground corresponding to each image frame to obtain the vehicle shape corresponding to the surveillance video, including:

[0161] Based on the surveillance video, multiple image frames are sequenced to obtain the target sequence;

[0162] According to the target sequence and the vehicle foreground corresponding to each image frame, the tracking algorithm is used to associate the various vehicle foregrounds to obtain the vehicle shape corresponding to the surveillance video.

[0163] In a possible implementation of the embodiment of the present application, the engineering vehicle remote control device 30 may further include:

[0164] The first receiving module is configured to receive a connection signal with an engineering vehicle and determine the engineering vehicle corresponding to the connection signal as a target engineering vehicle;

[0165] An image acquisition module is used to acquire the current image corresponding to the target engineering vehicle;

[0166] A first determination module is used to determine the vehicle coating and the environmental coating based on the current image and establish a three-dimensional coordinate system;

[0167] A module is established to build a BIM model corresponding to the target engineering vehicle based on the vehicle coating, environmental coating and three-dimensional coordinate system.

[0168] In a possible implementation of the embodiment of the present application, the engineering vehicle remote control device 30 may further include:

[0169] The second receiving module is used to receive the corresponding operation instructions of the operator;

[0170] A second determination module is used to determine a standard vehicle action corresponding to the operation instruction and to determine an actual vehicle action corresponding to the operation instruction;

[0171] A calculation module, used for calculating the similarity between the standard vehicle action and the actual vehicle action;

[0172] The third determination module is configured to determine whether to generate an operation abnormality signal based on the similarity.

[0173] Those skilled in the art will clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0174] See also Figure 4 , the embodiment of the present application also introduces a remote control device from the perspective of a physical device, such as Figure 4 As shown, Figure 4 The remote control device 40 shown includes a processor 401 and a memory 403. The processor 401 and the memory 403 are connected, for example, via a bus 402. Optionally, the remote control device 40 may also include a transceiver 404 and a display module 405. It should be noted that in actual applications, the number of transceivers 404 is not limited to one, and the structure of the remote control device 40 does not constitute a limitation on the embodiments of the present application.

[0175] Processor 401 may be a CPU (Central Processing Unit), a general-purpose processor, a DSP (Digital Signal Processor), an ASIC (Application Specific Integrated Circuit), an FPGA (Field Programmable Gate Array), or other programmable logic device, transistor logic device, hardware component, or any combination thereof. It may implement or execute the various exemplary logic blocks, modules, and circuits described in conjunction with the disclosure of this application. Processor 401 may also be a combination that implements computing functions, such as a combination of one or more microprocessors, or a combination of a DSP and a microprocessor.

[0176] Bus 402 may include a path for transmitting information between the above components. Bus 402 may be a PCI (Peripheral Component Interconnect) bus or an EISA (Extended Industry Standard Architecture) bus. Bus 402 may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Figure 4 Only one thick line is used in the diagram, but this does not mean that there is only one bus or one type of bus.

[0177] The memory 403 may be a ROM (Read Only Memory) or other type of static storage device that can store static information and instructions, a RAM (Random Access Memory) or other type of dynamic storage device that can store information and instructions, or an EEPROM (Electrically Erasable Programmable Read Only Memory), a CD-ROM (Compact Disc Read Only Memory) or other optical disk storage, optical disk storage (including compact discs, laser discs, optical discs, digital versatile discs, Blu-ray discs, etc.), a magnetic disk storage medium or other magnetic storage device, or any other medium that can be used to carry or store desired program code in the form of instructions or data structures and can be accessed by a computer, but is not limited thereto.

[0178] The memory 403 is used to store application code for executing the solution of the present application, and the execution is controlled by the processor 401. The processor 401 is used to execute the application code stored in the memory 403 to implement the content shown in the above method embodiment.

[0179] Display module 405 is used to display the BIM model. This display module 405 can be implemented using various technologies and components, responsible for converting digital signals into visual images or text. Specifically, it can be an LED (Light Emitting Diode) display module, an LCD (Liquid Crystal Display) display module, an OLED (Organic Light Emitting Diode) display module, or an E Ink electronic paper display module, though this embodiment of the present application is not limited thereto.

[0180] Among them, the remote control devices include but are not limited to: mobile terminals such as mobile phones, laptops, digital broadcast receivers, PDAs (personal digital assistants), PADs (tablet computers), PMPs (portable multimedia players), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc., and can also be servers, etc. Figure 4 The remote control device shown is only an example and should not limit the functions and scope of use of the embodiments of the present application.

[0181] An embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon. When the computer-readable storage medium is run on a computer, the computer can execute the corresponding contents of the aforementioned method embodiment.

[0182] It should be understood that although the steps in the flowcharts of the accompanying drawings are shown in sequence as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some of the steps in the flowcharts of the accompanying drawings may include multiple sub-steps or multiple stages, and these sub-steps or stages are not necessarily executed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be executed in turn or alternately with other steps or at least a portion of the sub-steps or stages of other steps.

[0183] The above are only some of the implementation methods of the present application. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the principles of the present application. These improvements and modifications should also be regarded as the scope of protection of the present application.

Claims

1. A remote control method for an engineering vehicle, characterized in that: The remote control method of an automated engineering vehicle is performed by a remote control device, and the method includes: Acquire a surveillance video corresponding to the target engineering vehicle in real time, wherein the surveillance video includes the complete target engineering vehicle and the environment; Analyzing the surveillance video to obtain the vehicle shape and current environment characteristics corresponding to the target engineering vehicle; Obtaining the BIM model corresponding to the target engineering vehicle; updating the BIM model based on the vehicle form and the current environmental characteristics, and displaying the updated BIM model on a display module; The analyzing the surveillance video to obtain the vehicle shape and current environment characteristics corresponding to the target engineering vehicle includes: Segmenting the surveillance video into a plurality of image frames, and detecting the target engineering vehicle in each image frame to obtain vehicle features in each image frame; Based on the vehicle features in each of the image frames, separating the vehicle features from the corresponding image frame to obtain a vehicle foreground and an environmental background; Associating the vehicle foreground corresponding to each of the image frames to obtain the vehicle shape corresponding to the surveillance video; Determining the current environmental features corresponding to the surveillance video based on the environmental background corresponding to each of the image frames; The vehicle features include shape sub-features, color sub-features, and size sub-features. Detecting the target engineering vehicle in each image frame to obtain the vehicle features in each image frame includes: Extracting the target engineering vehicle in each of the image frames based on a contour extraction function to obtain a vehicle contour corresponding to each image frame, and determining a shape sub-feature corresponding to each of the image frames based on the vehicle contour corresponding to each image frame; Converting the pixels of each sub-image corresponding to the vehicle outline into an HSV color space, and calculating the HSV histogram of the pixels in each sub-image to obtain a color sub-feature corresponding to each image frame; Obtaining a size ratio corresponding to the image frame, where the size ratio is used to represent a ratio between an actual size and an image size; Identifying a first size of each of the vehicle outlines, and calculating a second size corresponding to each of the vehicle outlines based on the size ratio to obtain a size sub-feature corresponding to each of the image frames; Determining a vehicle feature in each of the image frames based on the shape sub-features, color sub-features, and size sub-features corresponding to each of the image frames; The determining, based on the environmental background corresponding to each of the image frames, the current environmental features corresponding to the surveillance video includes: Use feature point detection algorithm to identify the salient elements in each environmental background and calculate the descriptor corresponding to each salient element; Extract texture features and color features from each environmental background; Determining current environmental features corresponding to the surveillance video based on texture features, color features, and descriptors corresponding to salient elements in each of the environmental backgrounds; The step of associating the vehicle foreground corresponding to each of the image frames to obtain the vehicle form corresponding to the surveillance video includes: Based on the surveillance video, a plurality of the image frames are sequenced to obtain a target sequence; According to the target sequence and the vehicle foreground corresponding to each image frame, a tracking algorithm is used to associate each vehicle foreground to obtain the vehicle shape corresponding to the monitoring video; Before updating the BIM model, the method further includes: receiving a connection signal with an engineering vehicle, and determining the engineering vehicle corresponding to the connection signal as a target engineering vehicle; Acquire a current image corresponding to the target engineering vehicle; Determine the vehicle coating and the environmental coating based on the current image, and establish a three-dimensional coordinate system; A BIM model corresponding to the target engineering vehicle is established based on the vehicle coating, the environmental coating and the three-dimensional coordinate system.

2. The remote control method for an engineering vehicle according to claim 1, characterized in that: The method further comprises: Receive the corresponding operation instructions from the operator; Determining a standard vehicle action corresponding to the operation instruction, and determining an actual vehicle action corresponding to the operation instruction; Calculating the similarity between the standard vehicle action and the actual vehicle action; And based on the similarity, it is determined whether to generate an operation abnormality signal.

3. A remote control device for an engineering vehicle, characterized in that: Executing the remote control method for an engineering vehicle according to any one of claims 1 to 2 comprises: A video acquisition module is used to acquire a monitoring video corresponding to a target engineering vehicle in real time, wherein the monitoring video includes the complete target engineering vehicle and the environment; An analysis module is used to analyze the monitoring video to obtain the vehicle shape corresponding to the target engineering vehicle and the current environment characteristics; A model acquisition module is used to obtain the BIM model corresponding to the target engineering vehicle; The display module is used to update the BIM model based on the vehicle form and the current environmental characteristics, and display the updated BIM model on the display module.

4. A remote control device, characterized in that: The remote control device includes: Display module, used to display the BIM model; at least one processor; Memory; At least one application, wherein the at least one application is stored in a memory and configured to be executed by at least one processor, and the at least one application is configured to: execute the engineering vehicle remote control method according to any one of claims 1 to 2.

5. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed in a computer, the computer is caused to execute the remote control method for an engineering vehicle according to any one of claims 1 to 2.

Images