Laser point cloud and video fusion method and device based on 3D deploy and control ball

Abstract

The invention provides a laser point cloud and video fusion method and device based on a 3D deploy and control ball, and relates to the technical field of data processing, and the method comprises the steps: carrying out the encryption processing of laser point cloud data and video data at a data transmission stage, and completing the decryption, timestamp alignment and data format standardization at a receiving end. After standardization, the data is distributed to a plurality of computing nodes for parallel processing, image texture, edge and motion vector features are extracted on the video side, space coordinates, surface normal vectors and point density features are extracted on the point cloud side, and a fusion feature set is generated through cross-modal correlation calculation and matching. And constructing a multi-modal three-dimensional scene representation based on the set, performing automatic exploration and analysis on a working site, outputting analysis results including terrains, obstacles and the like, and further generating a working scheme with spatial geometric constraints and working safety boundaries. According to the invention, end-to-end security and time-space consistency processing of the video monitoring data and the laser point cloud data can be realized.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of data processing, and in particular to a method and device for fusing laser point cloud and video based on a 3D control ball. BACKGROUND

[0002] The monitoring and analysis of a work site usually rely on video monitoring equipment or laser point cloud surveying equipment operating independently. The video monitoring equipment can provide two-dimensional image information for target recognition and scene perception, while the laser point cloud surveying equipment can provide spatial coordinates and geometric structure information for constructing a three-dimensional model. The two types of data are usually processed independently in actual applications, and it is difficult to form a complement in a unified space-time framework, so that the texture details, geometric structures and dynamic behavior characteristics cannot be considered in complex work environments, resulting in incomplete scene representation and lack of reliability in work scheme generation.

[0003] A prominent defect of the prior art is that video monitoring data and laser point cloud data lack end-to-end security and space-time consistency in data transmission, feature extraction and scene modeling. The features on the video side and the point cloud side often have time misalignment and spatial mismatch, which can easily cause misjudgment of obstacle position or error in work path planning, thereby affecting work safety.

[0004] Therefore, a method is needed to realize the end-to-end security and space-time consistency of video monitoring data and laser point cloud data. SUMMARY

[0005] The present application provides a method and device for fusing laser point cloud and video based on a 3D control ball, which can realize the end-to-end security and space-time consistency of video monitoring data and laser point cloud data.

[0006] In a first aspect of the present application, a method for fusing laser point cloud and video based on a 3D control ball is provided, the method comprising: After the 3D control ball collects laser point cloud data and video data, encryption processing is performed during data transmission; The laser point cloud data and the video data are decrypted at the receiving end, and the laser point cloud data and the video data are respectively subjected to timestamp alignment and data format standardization processing; The standardized laser point cloud data and video data are distributed to a plurality of computing nodes, and each computing node performs feature extraction operations in parallel, wherein multi-dimensional feature vectors based on image texture, edge and motion vector are extracted for the video data, and three-dimensional geometric feature vectors based on spatial coordinates, surface normal vector and point density are extracted for the laser point cloud data; cross-modal correlation calculation is performed on the multi-dimensional feature vectors output by each of the computing nodes and the three-dimensional geometric feature vectors, and the multi-dimensional feature vectors and the three-dimensional geometric feature vectors are matched by using the timestamp alignment result, so as to generate a fusion feature set; Based on the fusion feature set, a multi-modal three-dimensional scene representation is constructed, and an intelligent and automated survey and analysis is performed on the work site, and a work site analysis result is output. Based on the multi-modal three-dimensional scene representation and the work site analysis result, a work scheme with spatial geometric constraints and work safety boundaries is automatically generated.

[0007] In a second aspect of the present application, a device for fusing laser point cloud and video based on 3D control ball is provided, which is used to perform any one of the methods described above, and the device comprises an acquisition module, a processing module and an output module, wherein: The acquisition module is used to encrypt the laser point cloud data and video data collected by the 3D control ball during data transmission. The processing module is used to decrypt the laser point cloud data and video data at the receiving end, and perform timestamp alignment and data format standardization processing on the laser point cloud data and video data, respectively. The processing module is used to distribute the standardized laser point cloud data and video data to a plurality of computing nodes, and each computing node performs feature extraction operation in parallel, wherein a multi-dimensional feature vector based on image texture, edge and motion vector is extracted for the video data, and a three-dimensional geometric feature vector based on spatial coordinates, surface normal vector and point density is extracted for the laser point cloud data. The processing module is used to perform cross-modal correlation calculation on the multi-dimensional feature vectors output by each of the computing nodes and the three-dimensional geometric feature vectors, and match the multi-dimensional feature vectors and the three-dimensional geometric feature vectors by using the timestamp alignment result, so as to generate a fusion feature set. The processing module is used to construct a multi-modal three-dimensional scene representation based on the fusion feature set, and perform intelligent and automated survey and analysis on the work site, and output a work site analysis result. The output module is used to automatically generate a work scheme with spatial geometric constraints and work safety boundaries based on the multi-modal three-dimensional scene representation and the work site analysis result.

[0008] In a third aspect of the present application, an electronic device is provided, comprising a processor, a memory, a user interface and a network interface, the memory being configured to store instructions, the user interface and the network interface each being configured to communicate with other devices, and the processor being configured to execute the instructions stored in the memory to cause the electronic device to perform the method according to any one of the preceding aspects.

[0009] In a fourth aspect of the present application, a computer-readable storage medium is provided, which stores instructions that, when executed, perform the method according to any one of the preceding aspects.

[0010] In summary, the one or more technical solutions provided in the embodiments of the present application have at least the following technical effects or advantages: 1. By introducing an encryption mechanism based on device identity credentials and session key sets in the data transmission link, the confidentiality and integrity of the laser point cloud data and video data in the link are ensured, and decryption, replay detection and integrity verification are completed in combination with the access token at the receiving end, thereby realizing end-to-end security; at the same time, the unified timing label is used for timestamp alignment and frame loss compensation, the extrinsic parameter matrix and the intrinsic parameter matrix are used to establish cross-modal space mapping, and after cross-node parallel feature extraction, the fusion feature set is generated through timing alignment and cross-modal matching, ensuring that the video data and point cloud data maintain consistent time base and spatial reference in the whole process, and realizing end-to-end spatiotemporal consistency processing.

[0011] 2. By generating time window batches and distributing them to multiple computing nodes, the video data and laser point cloud data are processed in parallel within a unified timing range, which not only ensures the consistency of cross-modal data, but also improves the processing efficiency by using parallel computing, thereby realizing efficient and synchronous feature extraction in large-scale scenarios.

[0012] 3. By establishing an encryption mechanism based on device identity credentials and hardware root keys at the 3D control ball end, the point cloud session key and the video session key are used to protect and authenticate the data at the frame level, and replay protection and encrypted channels are added during transmission, thereby ensuring the confidentiality, integrity and anti-replay ability of the laser point cloud data and video data during transmission.

[0013] 4. By constructing a multi-modal three-dimensional scene representation based on the fusion feature set, terrain and topography segmentation, obstacle detection and crossing object analysis are performed in the triangular mesh combined with geometric and visual attributes, and a risk index is used to generate a job safety boundary, and finally the analysis results with passable corridors, device placement sites and job path suggestions are output, thereby realizing intelligent and automated survey and risk assessment of the job site.

[0014] 5. By establishing a path and process joint optimization model on the task constraint graph, path planning, equipment layout and time schedule scheduling are uniformly solved, and executable instruction set with dynamic monitoring and emergency rollback mechanism is generated by combining with operation safety boundary, so that the automatic operation scheme generation meeting the space geometric constraint and safety constraint is realized. BRIEF DESCRIPTION OF DRAWINGS

[0015] Figure 1 is a flowchart of a method for laser point cloud and video fusion based on a 3D control ball disclosed by an embodiment of the present application; Figure 2 is a module schematic diagram of a device for laser point cloud and video fusion based on a 3D control ball disclosed by an embodiment of the present application; Figure 3 is a structural schematic diagram of an electronic device disclosed by an embodiment of the present application.

[0016] Explanation of reference signs: 201, acquisition module; 202, processing module; 203, output module; 301, processor; 302, communication bus; 303, user interface; 304, network interface; 305, memory. DETAILED DESCRIPTION

[0017] In order for those skilled in the art to better understand the technical solutions in the specification, the technical solutions in the specification will be clearly and completely described below in combination with the drawings in the embodiments of the specification. Obviously, the described embodiments are only some of the embodiments of the present application, not all the embodiments.

[0018] In the description of the embodiments of the present application, the words such as "for example" or "for instance" are used to represent an example, illustration or description. Any embodiment or design scheme described as "for example" or "for instance" in the embodiments of the present application should not be interpreted as more preferred or more advantageous than other embodiments or design schemes. Rather, the words such as "for example" or "for instance" are intended to present the relevant concept in a specific manner.

[0019] In the description of the embodiments of the present application, the term "a plurality of" means two or more. For example, a plurality of systems means two or more systems, and a plurality of screen terminals means two or more screen terminals. In addition, the terms "first" and "second" are used for description purposes only, and should not be interpreted as indicating or implying relative importance or implicitly indicating the indicated technical features. Therefore, the features limited by "first" and "second" can explicitly or implicitly include one or more features. The terms "include", "contain", "have" and their variants mean "include but are not limited to", unless otherwise specifically emphasized.

[0020] In the prior art, the video monitoring device and the laser point cloud surveying device usually operate independently, the former focuses on two-dimensional image recognition, and the latter focuses on three-dimensional geometric modeling, both lack unified space-time fusion, leading to incomplete scene representation, insufficient reliability of operation scheme generation, and lack of end-to-end security and space-time consistency guarantee in the data transmission and processing process, which is easy to cause feature misplacement, obstacle misjudgment and path planning error, and therefore an end-to-end secure and space-time consistent processing method for video monitoring data and laser point cloud data is urgently needed.

[0021] The embodiment discloses a method for fusing laser point cloud and video based on a 3D control ball, referring to Figure 1 , comprising the following steps S110-S160: S110, after collecting laser point cloud data and video data by the 3D control ball, performing encryption processing in the data transmission process.

[0022] The method for fusing laser point cloud and video based on the 3D control ball disclosed by the embodiment of the application is applied to a server, the server includes but is not limited to electronic devices such as mobile phones, tablet computers, wearable devices, PC (Personal Computer, personal computer) and the like, and can also be a background server running a method for fusing laser point cloud and video based on the 3D control ball. The server can be realized by an independent server or a server cluster composed of multiple servers.

[0023] In a possible implementation, after collecting laser point cloud data and video data by the 3D control ball, encryption processing is performed in the data transmission process, specifically including: after generating a device identity certificate by the 3D control ball, binding the device identity certificate and a hardware root key in a security chip, outputting a device identity public key and a device identity certificate; verifying the device identity certificate and issuing an access token and a negotiation parameter set, wherein the 3D control ball generates a session key set based on the negotiation parameter set and the device identity public key, the session key set including a point cloud session key for laser point cloud data and a video session key for video data; establishing a frame-level protection structure for the laser point cloud data by using the point cloud session key, and establishing a frame-level protection structure for the video data by using the video session key, injecting a time sequence label, a frame sequence number and a source identification in the frame header, and performing encryption processing with authentication on the frame payload, to generate a point cloud ciphertext frame sequence and a video ciphertext frame sequence; in the transmission process, attaching a replay bitmap of a rolling window and an accumulated counter to the point cloud ciphertext frame sequence and the video ciphertext frame sequence, and establishing a connection-oriented encrypted channel based on the access token, to perform real-time transmission by taking the point cloud ciphertext frame sequence and the video ciphertext frame sequence as the load.

[0024] Specifically, after generating the device identity credential of the 3D surveillance ball, in the process of binding the device identity credential with the hardware root key in the security chip, first, a unique device identity credential is generated in the trusted execution environment of the 3D surveillance ball by calling the security module, which usually contains a device identifier, a timestamp and a random challenge code, for identifying the uniqueness of the 3D surveillance ball in the network. Then, the device identity credential is bound by encryption using the hardware root key inside the security chip, ensuring that the credential cannot be forged or copied. The result of the hardware root key binding can generate a device identity public key and a device identity certificate, where the device identity public key is used to interact with the external authentication service, and the device identity certificate is used to prove the authenticity and integrity of the device identity. The hardware root key here refers to the key material written in the security chip manufacturing stage and cannot be exported, which is the root of trust for all security operations.

[0025] In the process of checking the device identity certificate and issuing an access token and a set of negotiation parameters, the authentication service of the receiving end first checks the validity of the received device identity certificate, including signature verification, timestamp validity check and certificate chain integrity verification. When the verification is passed, the authentication service will issue an access token and a set of negotiation parameters, where the access token is used as the only credential for the device to establish an encrypted channel later, and the set of negotiation parameters includes key negotiation algorithm, encryption suite selection and random seed, etc., which are used to guide the device to generate a new session key set. On this basis, the 3D surveillance ball uses the set of negotiation parameters and the device identity public key to execute a key exchange protocol to generate a session key set, which contains a point cloud session key and a video session key, respectively used for the encryption protection of laser point cloud data and video data. The session key here refers to a temporary key generated dynamically during a session, which is used to ensure the confidentiality and forward security of communication.

[0026] In the process of establishing frame-level protection structure for laser point cloud data using point cloud session key and establishing frame-level protection structure for video data using video session key, first, each frame of laser point cloud data and each frame of video data are organized according to frame sequence, and a timing label, a frame number and a source identifier are inserted in the frame header to ensure that data frames of different sources can be correctly reorganized and aligned at the decryption end. Then, authenticated encryption processing is performed on the frame payload, i.e. confidentiality encryption and integrity check are completed at the same time, where confidentiality encryption ensures that frame data cannot be stolen, and integrity check ensures that frame data cannot be tampered with during transmission. The point cloud ciphertext frame sequence and the video ciphertext frame sequence are the encryption results obtained through the above process, which can protect three-dimensional space coordinates and image texture information respectively. The frame-level protection structure here refers to the security encapsulation mechanism performed at the level of each frame of data, ensuring that each independent frame can be independently verified and decrypted.

[0027] In the process of attaching the rolling window replay bitmap and the cumulative counter to the point cloud ciphertext frame sequence and the video ciphertext frame sequence during transmission, the system generates a dynamically updated bitmap window for each batch of transmitted data frames, which is used to record the frame sequence numbers that have been received by the current receiving end, thereby achieving detection of replay attacks. The cumulative counter is used to ensure the monotonic increasing property of the sequence, thereby avoiding the disturbance of the timing by the attacker by inserting old frames. Based on the access token, a connection-oriented encrypted channel is established between the 3D surveillance sphere and the receiving end. In this encrypted channel, all transmitted data frames are protected within the context of a session and are subject to reliable handshake and key confirmation. Finally, the point cloud ciphertext frame sequence and the video ciphertext frame sequence are transmitted in real time as the payload in the encrypted channel, ensuring that the data can be delivered to the receiving end in a low-latency, secure and tamper-proof manner in a complex operation environment. The encrypted channel here refers to a secure communication link established through key negotiation and access token authentication, which has the ability to prevent eavesdropping, tampering and replay.

[0028] S120, decrypting the laser point cloud data and the video data at the receiving end, and performing timestamp alignment and data format standardization processing on the laser point cloud data and the video data.

[0029] In one possible implementation, decrypting the laser point cloud data and the video data at the receiving end specifically includes: after completing access token verification and locating the corresponding session key set, importing the point cloud session key and the video session key, initializing the decryption context and resetting the replay window, the cumulative counter and the timing label buffer; inputting the point cloud ciphertext frame sequence and the video ciphertext frame sequence into the corresponding decryption context, performing replay detection and sequence rearrangement based on the timing label, the frame sequence number and the source identifier, and removing illegal ciphertext frames using the rolling window replay bitmap and the cumulative counter; performing authenticated integrity verification on the ciphertext frames that pass the replay detection, decrypting the ciphertext frames that pass the verification using the point cloud session key and the video session key respectively, outputting the point cloud decrypted frame object and the video decrypted frame object, and retaining the timing label, the frame sequence number and the source identifier in the decrypted frame object; performing forward error correction recovery and frame loss compensation on the point cloud decrypted frame object and the video decrypted frame object, and completing timing label correction and cross-modal time delay compensation based on the clock mapping relationship constructed based on the negotiation parameter set, outputting the timing-aligned point cloud decrypted frame object sequence and the timing-aligned video decrypted frame object sequence; parsing the timing-aligned point cloud decrypted frame object sequence into standardized laser point cloud data, and parsing the timing-aligned video decrypted frame object sequence into standardized video data.

[0030] Specifically, after the access token verification is completed at the receiving end and the corresponding session key set is located, the point cloud session key and the video session key are imported, and decryption contexts are constructed for the two types of data in a secure process, respectively. The symmetric key, random number generator, and authentication integrity check parameter set are initialized, and the replay window, cumulative counter, and time tag buffer are reset. The replay window records the acceptable frame number sliding range, the cumulative counter provides a monotonically increasing arrival reference, and the time tag buffer maintains the cross-modal arrival time reference. The point cloud decryption context and the video decryption context in a usable state are output as inputs for ciphertext frame reception and preliminary screening.

[0031] The point cloud ciphertext frame sequence and the video ciphertext frame sequence are input into the corresponding decryption contexts, respectively. The time tag, frame number, and source identifier in the header of each ciphertext frame are parsed, and replay detection and sequential rearrangement are performed using the replay bitmap of the replay window and the cumulative counter. The replay bitmap is used to mark the processed or seen frame numbers, and the cumulative counter is used to identify abnormal backward or skipped frame sequences. Entries determined to be illegal ciphertext frames are discarded and events are recorded. Ciphertext frame candidates determined to be legal are bucketed by source identifier and stably rearranged by frame number in the bucket. The candidate ciphertext frame set is output as input for authentication integrity check and decryption calculation.

[0032] Authentication integrity check is performed on the candidate ciphertext frame set frame by frame. Ciphertext frames that pass the check call the corresponding point cloud session key or video session key to perform payload decryption. The decryption output is packaged with the authentication mark as a decrypted frame object, and the time tag, frame number, and source identifier are retained in the decrypted frame object to maintain the traceability link. Ciphertext frames that fail authentication or have mismatched keys are rejected and an alarm count is triggered, resulting in point cloud decrypted frame objects and video decrypted frame objects. The two sets of decrypted frame objects are output as inputs for error recovery and time correction.

[0033] Forward error correction recovery and missing frame compensation are performed on the point cloud decrypted frame objects and the video decrypted frame objects. In the presence of forward error correction redundancy, encoding blocks are reorganized by source identifier and frame number, and error correction code is solved. Equivalent decrypted frame objects are generated for recoverable positions, and missing frame placeholders are registered for non-recoverable positions. Subsequently, according to the clock mapping relationship constructed based on the negotiated parameter set, the receiving end local clock is aligned to the 3D deployment sphere collection clock. The time tags of the two types of decrypted frame objects are corrected and jitter smoothed. Cross-modal time delay compensation is completed using a bounded lag buffer, and a sequence of time-synchronized frame objects is produced. The time-synchronized point cloud decrypted frame object sequence and the time-synchronized video decrypted frame object sequence are output as inputs for payload analysis and data standardization.

[0034] The time-aligned point cloud decryption frame object sequence is parsed into fields such as spatial coordinates, surface normal vectors and point density according to a predefined load structure, and the field order and numerical coding are unified to form standardized laser point cloud data; the time-aligned video decryption frame object sequence is parsed into fields such as texture matrix, edge response and motion vector, and the color space, resolution and time baseline are unified to form standardized video data; the two types of standardized results simultaneously inherit the source identifier and the corrected time label, which are used as the unified input for subsequent distributed parallel feature extraction, so that the feature calculation process consistent across nodes can be directly entered after decryption is completed.

[0035] Further, after obtaining the point cloud decryption frame object sequence and the video decryption frame object sequence at the receiving end, first, the time labels and the arrival times of the two types of decryption frame objects are used to construct a clock mapping sample set, stable sample pairs are obtained by eliminating abnormal jump points, and the 3D surveillance ball collection clock is mapped to the unified time axis of the receiving end through affine mapping, and the affine mapping is:

[0036] Among them, represents the time label carried by the decryption frame object; represents the corrected time label after mapping to the unified time axis; a represents the clock drift coefficient, which is a positive real number close to 1, and is estimated by weighted least squares on the sample pair set; b represents the clock offset, which is a real number, and is obtained by solving together with ; the mapping principle is to synchronize the clocks at both ends through affine transformation, so that the same physical time is aligned on the unified time axis, thereby providing a unified time reference for cross-modal alignment, and the point cloud time label corrected by affine mapping and the video time label corrected by affine mapping are output as the input of jitter smoothing.

[0037] After obtaining the time label corrected by affine mapping, in order to suppress the short-period jitter introduced by the network and the decoder, first-order exponential smoothing is performed on the two types of corrected time labels respectively, and the smoothing update is:

[0038] Among them, represents the smoothing time label of the th frame; represents the smoothing coefficient, which is a real number in the interval, and is selected by statistical jitter variance; represents the corrected time label of the th frame; the update principle is recursive low-pass filtering to weaken high-frequency jitter and maintain the monotonicity of the time label, and the smoothed point cloud time label and the smoothed video time label are output as the input of the unified time axis sampling.

[0039] A sequence of sampling instants is generated according to a target frame rate and a start time on a unified time axis, and two types of frames are aligned to the same sampling instant by a method combining nearest neighbor plus threshold and linear interpolation of small gap, the sequence of sampling instants is:

[0040] wherein, denotes the th unified sampling instant; denotes a start time of the unified time axis; denotes a target frame rate, which is a positive real number and is selected by downstream parallel processing constraints; the alignment principle is to use nearest neighbor matching within a tolerance , if , then direct pairing, if there is no direct pairing within the small gap threshold, then construct a placeholder frame by linear interpolation of the adjacent two frames, if it exceeds the small gap threshold, then register a placeholder marker for missing frames, output the point cloud frame sequence aligned according to and the video frame sequence aligned according to as the input of data format standardization.

[0041] In video data format standardization, first, the video frame sequence aligned according to is completed decoding uniformization and color space uniformization, unified to a preset color space and bit depth, and alignment padding of resolution normalization and aspect ratio preservation is performed, then the time baseline is written to the of the unified time axis, and the frame header field is rewritten with source identification and frame sequence number, forming standardized video data. To ensure the stability of subsequent texture and edge statistics, contrast-limited histogram equalization of the gray or luminance channel is performed on each frame, and the equivalent gain factor is recorded as a quality metric; finally, the field order is source identification, correction timing label, frame sequence number, color space identification, resolution, pixel matrix and quality metric, as the video side unified input of distributed parallel feature extraction.

[0042] In laser point cloud data format standardization, first, the point cloud frame sequence aligned according to is spatially uniformized, mapping each frame of point cloud coordinates to a global reference coordinate system according to the calibration extrinsic parameters, and performing uniform resampling with a voxel grid to stabilize the point density; then the surface normal vector and curvature are estimated within a fixed radius neighborhood, the normal vector and curvature are obtained by eigenvalue decomposition of the neighborhood covariance, the corresponding estimation is:

[0043] wherein, n denotes a surface normal vector, which is a unit vector; denotes a feature vector corresponding to the minimum eigenvalue of the neighborhood covariance; denotes a normalized measure of curvature, which is an interval real number; λmin represents the minimum eigenvalue of the neighborhood covariance, , , λi represents the eigenvalue sorted in ascending order; the principle is to depict the local geometric principal axis in the principal component direction, the minimum eigenvector is perpendicular to the local fitting plane, and the eigenvalue ratio measures the strength of local curvature; for each voxel, the number of unit volume points is taken as the point density, and the echo intensity is robustly normalized to suppress quantization bias; the final output field order is source identification, correction timing label, frame sequence number, spatial coordinates in the global reference coordinate system, surface normal vector, point density, curvature, and normalized echo intensity at the available time, forming standardized laser point cloud data as a unified input on the point cloud side of distributed parallel feature extraction.

[0044] In the cross-modal consistency check, the same video frame and point cloud frame under the same source identification, correction timing label and frame sequence number are checked for bidirectional mapping relationship, if there is a missing frame placeholder, the placeholder and time tolerance are registered in the standardized video data and standardized laser point cloud data, and the corresponding quality metric is transmitted to the subsequent calculation link; after the check is completed, the paired standardized laser point cloud data and standardized video data are output, and the unified time axis, unified field order and unified source identification are maintained, so that the subsequent cross-node parallel feature extraction can be directly consumed without further timing or structure shaping.

[0045] S130, the standardized laser point cloud data and video data are distributed to multiple computing nodes, and each computing node performs feature extraction operation in parallel.

[0046] In one possible implementation, the standardized laser point cloud data and video data are distributed to multiple computing nodes, and each computing node performs feature extraction operation in parallel, specifically including: generating a time window batch containing the standardized laser point cloud data and video data, and distributing the time window batch to multiple computing nodes; performing image preprocessing, texture enhancement and motion estimation on the video data in each computing node, extracting texture statistics, edge direction histogram and motion vector field, and splicing to form a multi-dimensional feature vector sequence with timing label, frame sequence number and source identification; performing spatial resampling and geometry estimation on the laser point cloud data in each computing node, calculating spatial coordinates, surface normal vector and point density, and combining curvature, flatness and linearity to form a three-dimensional geometric feature vector sequence with timing label, frame sequence number and source identification; performing timing aggregation and dimensionality reduction processing within the time window on the multi-dimensional feature vector sequence and the three-dimensional geometric feature vector sequence in each computing node, forming a node-level feature package with integrity flag and quality metric, and outputting through a distributed message queue.

[0047] Specifically, time window batches are generated with fixed time window width and sliding step under unified time axis, and the standardized laser point cloud data and the standardized video data are packaged in the same window by the task scheduler, each time window batch is bound with source identifier, time sequence label range and frame sequence number interval, and integrity flag is registered to indicate missing frame and placeholder, then load balancing is performed according to the computing power, memory and queue depth of the computing node, and the standardized laser point cloud data and the standardized video data in the same time window are allocated to the target computing node to ensure the time sequence consistency of the cross-modal. In this process, the time window width and the sliding step Quota constraints are used to maintain the balance between node throughput and end-to-end delay, and the frame count in the time window satisfies:

[0048] wherein, denotes the target frame rate; denotes the floor operation; the above constraint is used to ensure that the number of frames contained in each batch is stable and matches the parallel pipeline beat.

[0049] Image preprocessing, texture enhancement and motion estimation are performed on the video data in the computing node, first, denoising and color space consistency are performed without changing the source identifier and time sequence label, and contrast limited histogram equalization is performed on the brightness channel to stabilize the texture statistics; then the edge response is calculated and quantized as an edge direction histogram, and the first direction component of the edge direction histogram is defined as:

[0050] wherein, denotes the pixel set of the current video frame; denotes the pixel gradient direction; denotes the first direction interval; is an indicator function, true for 1 and false for 0; denotes the pixel gradient amplitude; and then based on pyramid optical flow estimation, a dense motion vector field is obtained, and the speed amplitude histogram and the main direction are counted on the grid. The splicing vectors of the texture statistics, the edge direction histogram and the motion vector statistics are constructed, and the time sequence label, the frame sequence number and the source identifier are connected to form a multi-dimensional feature vector sequence, which is used for subsequent cross-modal correlation. The above "multi-dimensional feature vector" refers to a set of statistical descriptions of a single video in texture, edge and motion dimensions; and the "motion vector field" refers to a pixel-level displacement vector field obtained by solving the optical flow between adjacent frames.

[0051] Spatial resampling and geometric estimation are performed on the laser point cloud data in the computing node, first, a voxel grid is used with a voxel edge length Uniform resampling is performed on the point cloud, and the voxel representative point sampling strategy ensures stable point density; then the surface normal and curvature are obtained by calculating the covariance matrix and performing eigenvalue decomposition in a fixed radius neighborhood The covariance matrix is calculated and eigenvalue decomposition is performed to obtain the surface normal and curvature, and the geometric estimation is defined as:

[0052] wherein, denotes the neighborhood covariance matrix; denotes the neighborhood centroid; denotes the eigenvalue of ; the corresponding eigenvector and as the surface normal; is the curvature measure; on this basis, the planarity and linearity are defined to characterize the local structure:

[0053] wherein, the larger the value, the more consistent with the local planar structure, the larger the value, the more consistent with the local line structure; at the same time, the point density is obtained by counting the number of points in a unit volume . The spatial coordinates, surface normal, point density, curvature, planarity and linearity are combined with the time sequence label, frame number and source identification to form a three-dimensional geometric feature vector sequence. The above "spatial resampling" refers to reducing the influence of uneven sampling on the statistical quantity by voxelization; "surface normal" refers to the unit normal in the direction of the smallest eigenvalue, which is used to describe the local surface orientation; "point density" refers to the point count in a unit volume.

[0054] In the computing node, the multi-dimensional feature vector sequence and the three-dimensional geometric feature vector sequence are executed in the time window for time sequence aggregation and dimension reduction processing. First, the frames in the same time window are registered according to the time sequence label using a bounded hysteresis buffer, and the placeholder marker and integrity flag bit are registered at the missing frame position; then, the statistical quantities are aggregated in the frame dimension to obtain the mean vector and the diagonal approximation of the covariance , and the linearly interpretable mapping matrix is used for fidelity dimension reduction, and the node-level aggregation is defined as:

[0055] wherein, denotes the node-level embedding vector; denotes vector splicing; the energy of the key statistics is maintained through offline least squares fitting or online recursive least squares; the quality measure integrates the bad frame ratio , occupancy ratio with signal-to-noise index and give:

[0056] wherein, are non-negative weights and satisfy normalization; denotes the proportion of frames judged as bad frames; denotes the proportion of interpolation occupancy frames; denotes the data signal-to-noise index reflected based on the gradient amplitude stability and the local covariance condition number. Finally, the node-level feature package containing , integrity flag and quality metric is output through a distributed message queue, and the message header carries the time window boundary, source identification and frame sequence interval to support downstream consistency checking. The above-mentioned "temporal aggregation" refers to the aggregation of frame-level statistics in the time direction within a fixed time window; "dimension reduction processing" refers to compressing redundant dimensions and retaining task-related energy through linear mapping; "distributed message queue" refers to middleware that provides an ordered, reusable and traceable data transmission channel in a distributed environment, which is used to decouple computing nodes and fusion nodes and ensure data arrival order and recoverability.

[0057] In S140, the multi-dimensional feature vectors output by each computing node are cross-modal correlation calculated with the three-dimensional geometric feature vectors, and the multi-dimensional feature vectors are matched with the three-dimensional geometric feature vectors by using the timestamp alignment result, so as to generate a fusion feature set.

[0058] In one possible implementation, the multi-dimensional feature vectors output by each computing node are cross-modal correlation calculated with the three-dimensional geometric feature vectors, and the multi-dimensional feature vectors are matched with the three-dimensional geometric feature vectors by using the timestamp alignment result, so as to generate a fusion feature set, specifically including: based on the unified time sequence label, performing bounded lag alignment and frame loss interpolation compensation on the multi-dimensional feature vector sequence and the three-dimensional geometric feature vector sequence, forming the time sequence aligned multi-dimensional feature vector sequence and the time sequence aligned three-dimensional geometric feature vector sequence; using the extrinsic parameter matrix and the intrinsic parameter matrix to perform pixel plane projection on the spatial coordinates in the time sequence aligned three-dimensional geometric feature vector, and combining the texture matrix and the edge response map in the time sequence aligned multi-dimensional feature vector to establish a candidate cross-modal pairing set; performing matching scoring on the candidate cross-modal pairing set according to time sequence consistency, spatial proximity and feature similarity, and constructing a bipartite graph to perform maximum weight matching solving, obtaining a cross-modal matching result set; performing confidence weighted synthesis on the multi-dimensional feature vectors and the three-dimensional geometric feature vectors based on the matching score and the quality metric, outputting a fusion feature vector, and in the scene of one-to-many or many-to-one pairing, performing robust pooling and uncertainty screening on the same cluster fusion features, and finally forming a fusion feature set as the input of the multi-modal three-dimensional scene representation construction.

[0059] Specifically, the bounded delay alignment is established with a unified time label in the distributed fusion unit, the multi-dimensional feature vector sequence and the three-dimensional geometric feature vector sequence are simultaneously put into a windowed time buffer, the window alignment is completed within the maximum delay threshold, and the small gap caused by network jitter or decoding overhead is compensated by frame interpolation. For the video side, when the time labels of two adjacent valid frames fall on both sides of the same unified sampling time, the placeholder feature is generated by linear interpolation in the feature space; for the point cloud side, the three-dimensional geometric feature vector is linearly interpolated or the nearest neighbor placeholder is maintained according to the same unified sampling time, so that the two types of sequences after time alignment are obtained. The interpolation is as follows, where the video side and the point cloud side are calculated independently:

[0060] wherein, represents the placeholder feature vector at the unified sampling time . and represent the adjacent valid feature vectors registered with the time labels and . represents the linear interpolation coefficient, which is a real number in the interval; "bounded delay alignment" refers to the rearrangement of the buffer within the fixed maximum delay threshold to ensure the consistency of the cross-modal time; "frame interpolation compensation" refers to maintaining the time axis integrity by linear interpolation or nearest neighbor placeholder in the feature space when there is short-time data missing.

[0061] After completing the time alignment, the spatial coordinates in the three-dimensional geometric feature vector are projected onto the pixel plane according to the extrinsic parameter matrix and the intrinsic parameter matrix obtained by calibration, and the pixel domain correspondence is established with the texture matrix and the edge response map in the multi-dimensional feature vector, forming a candidate cross-modal pairing set. The pixel plane projection process is expressed as:

[0062] wherein, represents the three-dimensional point in the global reference coordinate system; and represent the rotation and translation of the extrinsic parameter matrix, which are obtained by offline calibration or online calibration; represents the intrinsic parameter matrix, which contains the focal length and principal point parameters; represents the perspective division of the homogeneous coordinates to the pixel plane; represents the pixel coordinates. The local pixel neighborhood is defined in the texture matrix and the edge response map with the projection obtained as the center. If a feature vector falls into the neighborhood and satisfies the same temporal label and source identifier, then the 3D geometric feature vector and the corresponding multi-dimensional feature vector are paired to form a candidate cross-modal pair. The "extrinsic parameter matrix" refers to the rigid body transformation between cross-modal coordinate systems. The "intrinsic parameter matrix" refers to the internal parameters of the imaging geometry. The "pixel plane projection" refers to the mapping of a 3D point to a 2D pixel coordinate through the imaging model. The "candidate cross-modal pair set" refers to the set of cross-modal feature pairs that satisfy the temporal and spatial neighborhood consistency conditions.

[0063] The matching score is calculated for the candidate cross-modal pairing set, and a maximum weight matching of the bipartite graph is performed. The multidimensional feature vector and the three-dimensional geometric feature vector are respectively used as two vertex sets of the bipartite graph. The optimal one-to-one correspondence is solved using the matching score as the edge weight. The matching score is defined as:

[0064] in, This represents the time-series label difference between cross-modal pairings, and its value is a non-negative real number. This represents the time-series scale parameter, and its value is a positive real number. This represents the Euclidean distance from the projection point on the pixel plane to the center of the local pixel neighborhood, and its value is a non-negative real number. This represents a spatial proximity scale parameter, and its value is a positive real number. and These represent the same-dimensional aligned feature sub-vectors derived from multidimensional feature vectors and three-dimensional geometric feature vectors, respectively; Represents the inner product operator; Represents the L2 norm; Describes non-negative weight coefficients that satisfy... Maximum weight matching with bipartite graph decision variables. The expression is as follows:

[0065] in, Indicates the first The multidimensional feature vector and the first Matching score of three-dimensional geometric feature vectors; The matching selection variable takes a Boolean value; the two sets of inequality constraints each restrict each vertex to be matched at most once; "bipartite graph" refers to a graph in which the vertices are divided into two non-overlapping classes and the edges cross only the two classes; "maximum weight matching" refers to the matching selection that maximizes the sum of the edge weights under the matching constraints.

[0066] After obtaining the cross-modal matching result set, confidence-weighted synthesis is performed on the multi-dimensional feature vectors and three-dimensional geometric feature vectors of each matching pair to output a fused feature vector. In cases of one-to-many or many-to-one relationships, robust pooling and uncertainty filtering are applied to clustered fused features to suppress anomalies. The fusion follows a linear normalization rule.

[0067] wherein, with denotes the feature sub-vector after homodimensional mapping; with denotes the confidence weight, taking non-negative real values, which can be given by a monotonic combination of the matching score and the quality metrics on both sides, for example,

[0068] wherein, denotes the weight assignment coefficient, taking interval real values; the "quality metric" refers to the credibility index aggregated from the bad frame ratio, occupancy ratio and signal-to-noise index carried by the node-level feature bag. The robust pooling of multiple pairings in the same cluster can adopt the weighted truncated mean:

[0069] wherein, denotes the intra-cluster matching set; denotes the fusion candidate in the th cluster; denotes the intra-cluster median estimate; denotes the truncation threshold, taking positive real values. The weight of the fusion candidate is adaptively reduced according to its degree of outlying, and the low-confidence candidates are screened out according to the upper bound of covariance or equivalent uncertainty index, and the set of fusion feature vectors that pass the screening is reserved as the input for the construction of the multi-modal three-dimensional scene representation; "robust pooling" refers to the statistical synthesis of multiple samples in the cluster under the premise of suppressing the influence of outliers; "uncertainty screening" refers to the elimination of fusion results with insufficient confidence according to the weight and dispersion estimate threshold.

[0070] S150, based on the fusion feature set, constructing a multi-modal three-dimensional scene representation, and intelligently and automatically surveying and analyzing the work site, outputting the work site analysis result.

[0071] ​​In a possible implementation, based on the fusion feature set, a multi-modal three-dimensional scene representation is constructed, and an intelligent and automated survey and analysis of the work site is performed, and a work site analysis result is output. Specifically, a voxel grid is constructed using the spatial coordinates in the fusion feature vector and the surface normal vector in a global reference coordinate system, and a voxel-level fusion update is performed based on a truncated signed distance function to form an attribute triangle mesh containing geometric attributes and visual attributes; terrain and topography segmentation is performed on the attribute triangle mesh in combination with elevation, normal vector inclination, point density, and motion vector statistics to generate terrain and topography labels, obstacle initial labels, and column structure labels; boundary compactness, voxel occupancy consistency, and texture boundary consistency calculations are performed based on the connected pieces of the obstacle initial labels, and dynamic discriminant probability of motion vector statistics is introduced to update the obstacle probability to obtain a multi-modal three-dimensional scene representation with obstacle probability and geometric circumscribed parameters; the crossing objects are identified from the multi-modal three-dimensional scene representation, and the minimum static clearance and the minimum dynamic clearance are calculated, the crossing object clearance parameters and the obstacle probability, terrain and topography slope form a risk index vector, and the spatial boundaries of the work prohibited area, the work warning area, and the work permitted area are generated based on the weighted risk score to form a multi-modal three-dimensional scene representation with work safety boundaries; according to the multi-modal three-dimensional scene representation with work safety boundaries, the connectivity of the workable corridor, the equipment placement site, and the work path in the work permitted area is calculated, and a work site analysis result containing terrain and topography distribution, obstacle position and size parameters, crossing and crossing situation, work safety boundary, and work path suggestion is output.

[0072] Specifically, a three-dimensional voxel grid is constructed using the spatial coordinates in the fusion feature vector in a global reference coordinate system, and the surface normal vector is used as a geometric observation. Each fusion feature vector is projected along the line-of-sight direction to the voxel grid and the truncated distance is calculated. The truncated signed distance function is used to incrementally update the geometric state of the voxel. The voxel update follows the weighted average rule of weight normalization. The truncated signed distance of the voxel and the weight are iterated as follows:

[0073] wherein, represents the historical truncated signed distance of the voxel, and is a bounded real number; represents the observed truncated signed distance calculated from the fusion feature vector and the projection geometry, and is a bounded real number; represents the historical weight of the voxel, and is a non-negative real number; denotes the observation weight, taking non-negative real values, which is obtained by monotonically mapping the quality measure of the fused feature vector and the incident angle consistency. The isosurface extraction is performed on the updated voxel grid to obtain a triangular mesh. The surface normal and the point density in the fused feature vector are interpolated to the vertices of the mesh to propagate as the geometric attributes. The texture statistics and the edge direction histogram in the fused feature vector are mapped to the triangular mesh by key frame back-projection and are combined with confidence weighting in the overlapping area to form the visual attributes, thereby obtaining an attribute triangular mesh containing geometric attributes and visual attributes.

[0074] A vertex feature vector composed of the elevation, the normal vector inclination, the point density and the motion vector statistics is constructed for each mesh vertex on the attribute triangular mesh. The graph structure segmentation is performed on the vertex feature vector to output the terrain and landscape label, the obstacle initial label and the column-like structure label. The vertex feature vector is denoted as wherein is the elevation in the global reference coordinate system, taking real values; is the angle between the normal vector and the gravity direction, taking interval real values; is the point density of the unit volume point number, taking non-negative real values; is the motion vector amplitude statistics, taking non-negative real values. The Markov random field energy is taken as the segmentation criterion, and the energy function is

[0075] wherein, is the vertex label set; is the th vertex label, taking one of the terrain and landscape label, the obstacle initial label or the column-like structure label; is the single-point cost based on , taking non-negative real values, which is obtained by logical mapping of the elevation threshold, the inclination threshold and the point density threshold; is the adjacent vertex consistency cost, taking non-negative real values, which is represented by the function of the mesh edge weight and the normal vector difference; is the mesh adjacent edge set. The energy function is minimized to obtain the segmentation result and to assign the corresponding label.

[0076] The boundary compactness, the voxel occupancy consistency and the texture boundary consistency are calculated on the connected piece of the obstacle initial label, and the dynamic discrimination probability is obtained by combining the motion vector statistics to perform Bayesian update on the obstacle probability. The boundary compactness is represented by the perimeter-to-area ratio, which is defined as

[0077] wherein, is the boundary perimeter of the connected piece, taking non-negative real values; is the projection area of the connected component, taking values in the non-negative real numbers, The smaller the value, the tighter the compactness. The voxel occupancy consistency is characterized by the stability of the signed histogram of voxel signed distance, denoted as , taking values in the interval real numbers. The texture boundary consistency is characterized by the overlap of the edge response map and the boundary of the connected component, denoted as , taking values in the interval real numbers. The three are combined into the observation likelihood , which is updated with the historical obstacle probability

[0078] where is the updated obstacle probability, taking values in the interval real numbers; is the historical obstacle probability, taking values in the interval real numbers; is the observation likelihood obtained by combining , and , taking values in the interval real numbers. The geometric circumscription parameters, including the center, size and orientation of the minimum circumscribed rectangle or oriented bounding box, are calculated for each connected component, taking values in real numbers and unit vectors, respectively, for subsequent gap and risk assessment.

[0079] The crossing objects are identified on the multi-modal three-dimensional scene representation with obstacle probability and geometric circumscription parameters. The approximately linear or overhanging components are filtered as crossing objects according to geometric morphology and connectivity, and the minimum static gap and minimum dynamic gap are calculated, respectively. The minimum static gap is defined as the minimum distance from the crossing object to the terrain surface and the obstacle enclosing geometry, denoted as

[0080] where is the point set of the crossing object; is the point set of the environment surface, both taking values in the global reference coordinate system. The minimum dynamic gap considers relative motion, defined as

[0081] where and are the crossing object and environment position sets at time The crossing object gap parameters, obstacle probability and terrain slope are combined into a risk indicator vector, and the spatial boundary is generated by a weighted risk score:

[0082] where is the weighted risk score, taking values in the non-negative real numbers; and​ is the minimum gap, taking a non-negative real value; is the stability constant, taking a positive real value and much smaller than the regular gap scale; is the obstacle probability, taking an interval real value; is the normalized measure of terrain slope, taking an interval real value; , , , are non-negative weights and satisfy the normalization. According to the threshold, the space boundary of the operation prohibited zone, the operation alert zone and the operation permitted zone is obtained.

[0083] In the multi-modal three-dimensional scene representation with the operation safety boundary, the operation permitted zone is used to construct the passable corridor graph, the node set is generated for the equipment placement site and the key operation site, the edge set is generated for the passable corridor, and the edge cost is defined by the distance, the slope and the risk score. The connectivity is evaluated and the operation path suggestion is generated by using the constraint shortest path. The edge cost is defined as

[0084] wherein, is the cost of edge , taking a non-negative real value; is the edge length, taking a positive real value; is the average value of the slope along the edge, taking a non-negative real value; is the average value of the weighted risk score along the edge, taking a non-negative real value; , , are non-negative weights and satisfy the normalization. The path search is performed on the whole graph under the constraints of the operation safety boundary and the equipment operation radius, the connectivity of the passable corridor, the accessibility of the equipment placement site and the cost sorting of multiple candidate operation paths are obtained, and the operation site analysis result containing the terrain distribution, the obstacle position and size parameters, the cross-over situation, the operation safety boundary and the operation path suggestion is output.

[0085] S160, based on the multi-modal three-dimensional scene representation and the operation site analysis result, an operation scheme with spatial geometric constraints and operation safety boundary is automatically generated.

[0086] In one possible implementation, based on the multi-modal three-dimensional scene representation and the operation site analysis result, an operation scheme with spatial geometric constraints and operation safety boundary is automatically generated, specifically including: ​A task constraint graph containing geometric entities, passable corridors, equipment placement sites and suggested operation paths is constructed in a global reference coordinate system, and operation prohibited areas, operation alert areas and operation permitted areas are generated by combining operation safety boundaries in the task constraint graph; a path and process joint optimization model is established on the task constraint graph, path segment selection, equipment placement site selection and operation unit time schedule are taken as decision variables, a joint optimization problem containing path objective function and scheduling objective function is constructed, and spatial geometric constraints, operation safety boundary constraints and process sequence constraints are applied; the joint optimization problem is solved iteratively to generate path geometry, time schedule curve and resource allocation that meet the operation safety boundary, and dynamic gap monitoring instructions, emergency fallback corridors and speed and residence time limits of alert area boundaries are added to the candidate path; the path geometry, time schedule curve and resource allocation are instantiated into an executable instruction set, and real-time re-planning trigger conditions and rollback points are set in the risk area to form a final operation plan with spatial geometric constraints and operation safety boundaries.

[0087] Specifically, in the global reference coordinate system, the multi-modal three-dimensional scene representation and the operation site analysis result are mapped into a task constraint graph. First, the geometric entities are extracted with attribute triangle mesh and the directed graph of passable corridors is generated. The equipment placement sites and key operation sites are registered as nodes, the candidate operation path segments are registered as edges, and the operation safety boundary is interpreted as a three-partition set. Then, a comprehensive cost and a crossing permission flag are assigned to each edge. The comprehensive cost is a weighted sum of distance, slope and risk score, defined as:

[0088] wherein, represents the comprehensive cost of edge ; represents the edge length; represents the average slope along the edge; represents the average risk score along the edge; is a non-negative weight and satisfies normalization. Based on the operation safety boundary, the edges falling into the operation prohibited area are marked as unavailable, the edges crossing the operation alert area boundary are recorded with speed and residence upper limits, and the edges located in the operation permitted area are set as available and the redundant corridor information is retained, thereby obtaining a task constraint graph containing geometric entities, passable corridors, equipment placement sites and suggested operation paths, and embedding operation prohibited areas, operation alert areas and operation permitted areas.

[0089] A path and process joint optimization model is established on the task constraint graph, path segment selection, equipment placement site selection and operation unit time schedule are taken as decision variables, a weighted sum multi-objective form is used to minimize the path objective function and the scheduling objective function at the same time, and the joint objective and the main constraints are defined as:

[0090]

[0091] where, denotes path segment selection variables; denotes device placement site selection variables; denotes job unit schedule; is an edge cost; is a target weight, taking values in the interval; is a dispatch weight, taking non-negative and normalized values; is a maximum completion time; is a job unit a risk score at pose trajectory ; is a set of flow conservation and start-end point constraints; is a set of device placement site reachability and viewshed constraints; is a set of process precedence, resource non-overlapping and time window constraints; and are the feasible and infeasible subsets of job permission zone and job forbidden zone in the mission constraint graph, respectively. To suppress risk exposure in the job alert zone, velocity and dwell constraints are introduced for edges crossing the job alert zone:

[0092] where, is the execution velocity along edge ; is the dwell time along edge ; and are the upper bounds of velocity and dwell in the alert zone.

[0093] The joint optimization problem is solved iteratively, adopting a hierarchical decomposition strategy to satisfy spatial geometric constraints, job safety boundary constraints and process precedence constraints. The first layer solves a bounded shortest path subject to and constraints on the passable corridor subgraph, obtaining an initial path geometry; the second layer implements curvature-constrained smoothing on the initial path geometry in continuous space and fine-tunes the path with a gap gain term, making it feasible while reducing ; the third layer optimizes schedule curves and resource allocation in with mixed-integer programming or priority-constrained constraint propagation, outputting a The parallelism configuration is configured. For the path segment close to the object crossing and the high obstacle probability, the dynamic gap monitoring instruction and the emergency retreat corridor are automatically attached, and the speed upper limit and the residence upper limit are written for the segment crossing the operation warning area, so that the iterative solution simultaneously checks the operation safety boundary and the risk exposure threshold after each round of update until the target and the constraint converge, and finally the path geometry, the time curve and the resource allocation meeting the operation safety boundary are obtained.

[0094] The path geometry, the time curve and the resource allocation are instantiated as an executable instruction set, a task-oriented instruction template is used to generate a time-sequenced sequence of position instructions, speed instructions, attitude instructions, residence instructions and sensor trigger instructions, and each instruction is bound with a source identifier, a time sequence label and a quality metric; for the path segment marked as high risk, write real-time re-planning trigger conditions including dynamic gap threshold, obstacle probability threshold and attitude drift threshold, and configure rollback point and fallback corridor for each trigger condition, so that the execution end can unambiguously fallback to a safe state when the threshold is triggered. To ensure online robustness, check instructions for rotation key checking and data consistency checking are appended at the end of the instruction set, and version number and hash digest are recorded for each operation unit to realize end-to-end traceability and auditability of the operation scheme, and finally an operation scheme with spatial geometry constraints and operation safety boundary is formed and can be directly issued for execution.

[0095] The embodiment also discloses a device for fusing laser point cloud and video based on a 3D control ball, referring to Figure 2 , comprising an acquisition module 201, a processing module 202 and an output module 203203, and the device is used for executing any one of the above-mentioned methods for fusing laser point cloud and video based on a 3D control ball, wherein: The acquisition module 201 is used for encrypting the data in the data transmission process after the 3D control ball collects laser point cloud data and video data.

[0096] The processing module 202 is used for decrypting the laser point cloud data and the video data at the receiving end, and performing timestamp alignment and data format standardization processing on the laser point cloud data and the video data respectively.

[0097] The processing module 202 is used for distributing the standardized laser point cloud data and the video data to a plurality of computing nodes, and each computing node performs feature extraction operation in parallel, wherein a multidimensional feature vector based on image texture, edge and motion vector is extracted for the video data, and a three-dimensional geometric feature vector based on spatial coordinates, surface normal vector and point density is extracted for the laser point cloud data.

[0098] The processing module 202 is configured to perform cross-modal correlation calculation on the multi-dimensional feature vector output by each computing node and the three-dimensional geometric feature vector, and match the multi-dimensional feature vector and the three-dimensional geometric feature vector by using the timestamp alignment result, so as to generate a fusion feature set.

[0099] The processing module 202 is configured to construct a multi-modal three-dimensional scene representation based on the fusion feature set, and perform intelligent and automatic survey and analysis on the work site, and output a work site analysis result.

[0100] The output module 203 is configured to automatically generate a work scheme with spatial geometric constraints and work safety boundaries based on the multi-modal three-dimensional scene representation and the work site analysis result.

[0101] It should be noted that, in the implementation of the device provided in the above embodiment, only the division of the above functional modules is exemplified, and in actual application, the above functions can be completed by different functional modules according to needs, that is, the internal structure of the device is divided into different functional modules to complete all or part of the above-described functions. In addition, the device and method embodiments provided in the above embodiment belong to the same concept, and the specific implementation process is detailed in the method embodiment, which will not be described here.

[0102] The embodiment also discloses an electronic device, which refers to Figure 3 The electronic device can include at least one processor 301, at least one communication bus 302, a user interface 303, a network interface 304, and at least one memory 305.

[0103] The communication bus 302 is configured to realize the connection and communication between the components.

[0104] The user interface 303 can include a display screen (Display) and a camera (Camera), and the optional user interface 303 can further include a standard wired interface and a wireless interface.

[0105] The network interface 304 can optionally include a standard wired interface and a wireless interface (such as a WI-FI interface).

[0106] The processor 301 can include one or more processing cores. The processor 301 connects various parts within the server through various interfaces and lines, performs various functions of the server and processes data by running or executing instructions, programs, code sets or instruction sets stored in the memory 305, and calling data stored in the memory 305. Alternatively, the processor 301 can be implemented in at least one of a hardware form of a digital signal processing (DSP), a field-programmable gate array (FPGA), and a programmable logic array (PLA). The processor 301 can integrate a combination of one or more of a central processing unit (CPU), a graphics processing unit (GPU), and a modem. Among them, the CPU mainly processes operating systems, user interfaces, and application programs. The GPU is responsible for rendering and drawing the content to be displayed on the display screen. The modem is used to process wireless communication. It can be understood that the above-mentioned modem can also not be integrated into the processor 301, but can be realized by a separate chip.

[0107] The memory 305 can include a random access memory (RAM) and a read-only memory (ROM). Alternatively, the memory includes a non-transitory computer-readable storage medium. The memory 305 can be used to store instructions, programs, codes, code sets or instruction sets. The memory 305 can include a program storage area and a data storage area, wherein the program storage area can store instructions for implementing an operating system, instructions for at least one function (such as a touch function, a sound playing function, an image playing function, etc.), instructions for implementing the above-mentioned various method embodiments, etc. The data storage area can store data related to the above-mentioned various method embodiments, etc. The memory 305 can also be at least one storage device located away from the aforementioned processor 301. As a computer storage medium, the memory 305 can include an operating system, a network communication module, a user interface 303 module, and an application program of a method of fusing a laser point cloud and a video based on a 3D control ball.

[0108] In Figure 3In the electronic device shown, the user interface 303 is mainly used to provide an interface for the user to input, and obtain data input by the user. The processor 301 can be used to call an application program stored in the memory 305, which is a method for fusing a 3D control ball-based laser point cloud and a video, and when executed by one or more processors 301, causes the electronic device to perform the method of one or more of the above-described embodiments.

[0109] It should be noted that, for the foregoing method embodiments, in order to simply describe, they are all described as a series of action combinations, but those skilled in the art should know that the present application is not limited to the order of the actions described, because according to the present application, certain steps can be performed in other order or at the same time. Secondly, those skilled in the art should know that the embodiments described in the specification all belong to preferred embodiments, and the actions and modules involved are not necessarily essential to the present application.

[0110] In the above embodiments, the description of each embodiment has its own focus, and the parts not described in detail in a certain embodiment can be referred to the related description of other embodiments.

[0111] In the several embodiments provided by the present application, it should be understood that the disclosed device can be implemented in other ways. For example, the device embodiments described above are only schematic. The division of the units is only a logical function division. There can be another division manner for actual implementation, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the displayed or discussed mutual couplings or direct couplings or communication connections between the units can be indirect couplings or communication connections through some interfaces, devices or units, and can be electrical or other forms.

[0112] The units described as separate components can or can not be physically separate, and the components displayed as units can or can not be physical units, that is, they can be located in one place, or can be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the embodiment scheme.

[0113] In addition, each functional unit in the various embodiments of the present application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be implemented in the form of hardware, or in the form of a software functional unit.

[0114] The integrated unit, if implemented in the form of a software function unit and sold or used as an independent product, can be stored in a computer readable memory. Based on such understanding, the technical solutions of the present application, essentially or in other words, the part that contributes to the prior art or the whole or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a memory 305 and includes a plurality of instructions for causing a computer device (which can be a personal computer, a server or a network device, etc.) to execute all or part of the steps of the embodiments of the present application. The aforementioned memory 305 includes: a U disk, a mobile hard disk, a magnetic disk or an optical disk, and various media that can store program codes.

[0115] The present application also discloses a computer readable storage medium, which stores instructions. When executed by one or more processors 301, the instructions cause an electronic device to perform one or more methods as described in the above embodiments.

[0116] The above are only exemplary embodiments of the present disclosure, and cannot limit the scope of the present disclosure. That is, any equivalent changes and modifications made in accordance with the teachings of the present disclosure are still within the scope of the present disclosure. Other embodiments of the present disclosure will be readily apparent to those skilled in the art upon considering the specification and practicing the true principles of the present disclosure. The present application is intended to cover any variations, uses or adaptive changes of the present disclosure that follow the general principles of the present disclosure and include common knowledge or conventional technical means in the art that are not described in the present disclosure. The specification and examples are only considered as exemplary, and the scope and spirit of the present disclosure are defined by the claims.

Claims

1. A method for fusing laser point cloud and video based on 3D control sphere, characterized in that, The method comprises: After collecting laser point cloud data and video data in the 3D control sphere, encryption processing is performed in the data transmission process; In the receiving end, the laser point cloud data and the video data are decrypted, and timestamp alignment and data format standardization processing are performed on the laser point cloud data and the video data respectively; The laser point cloud data and the video data after standardization processing are distributed to multiple computing nodes, and each computing node performs feature extraction operation in parallel, wherein multi-dimensional feature vectors based on image texture, edge and motion vector are extracted for the video data, and three-dimensional geometric feature vectors based on spatial coordinates, surface normal vector and point density are extracted for the laser point cloud data; The multi-dimensional feature vectors and the three-dimensional geometric feature vectors output by each computing node are cross-modal correlation calculated, and the multi-dimensional feature vectors and the three-dimensional geometric feature vectors are matched by using the timestamp alignment result, so as to generate a fusion feature set; Based on the fusion feature set, a multi-modal three-dimensional scene representation is constructed, and intelligent and automatic survey and analysis are performed on the work site, and a work site analysis result is output; Based on the multi-modal three-dimensional scene representation and the work site analysis result, a work scheme with spatial geometric constraints and work safety boundaries is automatically generated.

2. The method of claim 1, wherein, The distribution of the laser point cloud data and the video data after standardization processing to multiple computing nodes and the parallel execution of feature extraction operation by each computing node specifically comprises: A time window batch containing the laser point cloud data and the video data after standardization processing is generated, and the time window batch is distributed to multiple computing nodes; In each computing node, image preprocessing, texture enhancement and motion estimation are performed on the video data, texture statistics, edge direction histogram and motion vector field are extracted, and multi-dimensional feature vector sequences with time sequence labels, frame sequence numbers and source identifiers are formed by splicing; In each computing node, spatial resampling and geometric estimation are performed on the laser point cloud data, spatial coordinates, surface normal vector and point density are calculated, and three-dimensional geometric feature vector sequences with time sequence labels, frame sequence numbers and source identifiers are formed by combining curvature, flatness and linearity; In each computing node, time sequence aggregation and dimension reduction processing within the time window are performed on the multi-dimensional feature vector sequences and the three-dimensional geometric feature vector sequences, node-level feature packages with integrity flag and quality metric are formed, and are output through a distributed message queue.

3. The method of claim 1, wherein, The cross-modal correlation calculation of the multi-dimensional feature vectors and the three-dimensional geometric feature vectors output by each computing node, and the matching of the multi-dimensional feature vectors and the three-dimensional geometric feature vectors by using the timestamp alignment result, so as to generate a fusion feature set, specifically comprises: Based on the unified time sequence label, bounded delay alignment and missing frame interpolation compensation are performed on the multi-dimensional feature vector sequences and the three-dimensional geometric feature vector sequences, and time sequence aligned multi-dimensional feature vector sequences and three-dimensional geometric feature vector sequences are formed; project the spatial coordinates in the time-sequentially aligned three-dimensional geometric feature vector into a pixel plane using an extrinsic matrix and an intrinsic matrix, and combine a texture matrix and an edge response map in the time-sequentially aligned multi-dimensional feature vector to establish a candidate cross-modal pairing set; perform matching scoring on the candidate cross-modal pairing set according to time sequence compatibility, spatial proximity, and feature similarity, and construct a bipartite graph to perform maximum weight matching to obtain a cross-modal matching result set; perform confidence weighting on the multi-dimensional feature vector and the three-dimensional geometric feature vector based on matching scoring and quality measurement, output a fused feature vector, and in the presence of one-to-many or many-to-one pairing, perform robust pooling and uncertainty screening on the same cluster fused features to finally form a fused feature set as an input for multi-modal three-dimensional scene representation construction.

4. The method of claim 3, wherein, based on the fused feature set, construct a multi-modal three-dimensional scene representation, and intelligently and automatically survey and analyze the work site to output a work site analysis result, specifically including: construct a voxel grid using the spatial coordinates and surface normal vectors in the fused feature vector in a global reference coordinate system, and perform voxel-level fusion update based on a truncated signed distance function to form an attribute triangle mesh containing geometric and visual attributes; perform terrain and landscape segmentation on the attribute triangle mesh in combination with elevation, normal vector inclination, point density, and motion vector statistics to generate terrain and landscape labels, obstacle initial labels, and column-like structure labels; perform boundary compactness, voxel occupancy consistency, and texture boundary consistency calculation based on the connected pieces of obstacle initial labels, and introduce dynamic discriminant probability of motion vector statistics to update obstacle probability to obtain a multi-modal three-dimensional scene representation with obstacle probability and geometric circumscribed parameters; identify crossing objects from the multi-modal three-dimensional scene representation and calculate minimum static clearance and minimum dynamic clearance, and construct a risk indicator vector from the crossing object clearance parameters, obstacle probability, and terrain and landscape slope, generate spatial boundaries of work prohibited areas, work alert areas, and work permitted areas based on weighted risk scoring to form a multi-modal three-dimensional scene representation with work safety boundaries; based on the multi-modal three-dimensional scene representation with work safety boundaries, calculate the connectivity of the work permitted area, equipment placement site, and work path, and output a work site analysis result containing terrain and landscape distribution, obstacle position and size parameters, crossing and crossing situation, work safety boundary, and work path suggestion.

5. The method of claim 1, wherein, based on the multi-modal three-dimensional scene representation and the work site analysis result, automatically generate a work scheme with spatial geometric constraints and work safety boundaries, specifically including: construct a task constraint graph containing geometric entities, passable corridors, equipment placement sites, and work path suggestions in a global reference coordinate system, and generate work prohibited areas, work alert areas, and work permitted areas in the task constraint graph with work safety boundaries; A path and process joint optimization model is established on the task constraint graph, path segment selection, equipment layout site selection and operation unit time schedule are taken as decision variables, a joint optimization problem including path objective function and scheduling objective function is constructed, and space geometry constraint, operation safety boundary constraint and process sequence constraint are applied; An iterative solution is performed on the joint optimization problem to generate path geometry, time schedule curve and resource allocation that meet the operation safety boundary, and dynamic gap monitoring instructions, emergency fallback corridor and speed and residence time limit of warning area boundary are added to the candidate path; The path geometry, time schedule curve and resource allocation are instantiated into an executable instruction set, real-time re-planning trigger conditions and rollback points are set in the risk area, and a final operation scheme with space geometry constraint and operation safety boundary is formed.

6. The method of claim 1, wherein, After collecting laser point cloud data and video data on the 3D control ball, encryption processing is performed during data transmission, specifically including: After the 3D control ball generates a device identity certificate, the device identity certificate and the hardware root key in the security chip are bound, and the device identity public key and the device identity certificate are output; The device identity certificate is verified and an access token and a negotiation parameter set are issued, wherein the 3D control ball generates a session key set based on the negotiation parameter set and the device identity public key, the session key set includes a point cloud session key for the laser point cloud data and a video session key for the video data; A frame-level protection structure is established for the laser point cloud data using the point cloud session key, and a frame-level protection structure is established for the video data using the video session key, a time label, a frame sequence number and a source identifier are injected into the frame header, and encryption processing with authentication is performed on the frame payload to generate a point cloud ciphertext frame sequence and a video ciphertext frame sequence; During transmission, a replay bitmap and an accumulated counter of the rolling window are added to the point cloud ciphertext frame sequence and the video ciphertext frame sequence, and a connection-oriented encrypted channel is established based on the access token, and the point cloud ciphertext frame sequence and the video ciphertext frame sequence are transmitted in real time as the payload.

7. The method of claim 6, wherein, The decryption processing of laser point cloud data and video data at the receiving end, specifically including: After the access token verification and positioning of the corresponding session key set are completed, the point cloud session key and the video session key are imported, the decryption context is initialized, and the replay window, the accumulated counter and the time label buffer are reset; The point cloud ciphertext frame sequence and the video ciphertext frame sequence are input into the corresponding decryption context, replay detection and sequential rearrangement are performed based on the time label, the frame sequence number and the source identifier, and illegal ciphertext frames are removed using the replay bitmap and the accumulated counter of the rolling window; The ciphertext frames that pass the replay detection are subjected to integrity verification with authentication, and the ciphertext frames that pass the verification are decrypted using the point cloud session key and the video session key, respectively, to output point cloud decrypted frame objects and video decrypted frame objects, and the time label, the frame sequence number and the source identifier are retained in the decrypted frame objects; Perform forward error correction recovery and frame compensation on the point cloud decryption frame object and the video decryption frame object, and complete time label correction and cross-modal time delay compensation based on the clock mapping relationship constructed based on the negotiated parameter set, and output a time-aligned point cloud decryption frame object sequence and a time-aligned video decryption frame object sequence; Parse the time-aligned point cloud decryption frame object sequence into standardized laser point cloud data, and parse the time-aligned video decryption frame object sequence into standardized video data.

8. An apparatus for fusing laser point cloud and video based on 3D control sphere, characterized in that, The device is used to perform a method of laser point cloud and video fusion based on a 3D control ball as claimed in any one of claims 1-7, and the device comprises an acquisition module, a processing module, and an output module, wherein: The acquisition module is configured to encrypt the laser point cloud data and the video data during data transmission after the 3D control ball collects the laser point cloud data and the video data; The processing module is configured to decrypt the laser point cloud data and the video data at a receiving end, and perform timestamp alignment and data format standardization on the laser point cloud data and the video data, respectively; The processing module is configured to distribute the standardized laser point cloud data and the standardized video data to multiple computing nodes, and each computing node performs feature extraction in parallel, wherein a multi-dimensional feature vector based on image texture, edge, and motion vector is extracted for the video data, and a three-dimensional geometric feature vector based on spatial coordinates, surface normal vector, and point density is extracted for the laser point cloud data; The processing module is configured to perform cross-modal correlation calculation on the multi-dimensional feature vector and the three-dimensional geometric feature vector output by each computing node, and match the multi-dimensional feature vector and the three-dimensional geometric feature vector using the timestamp alignment result, thereby generating a fusion feature set; The processing module is configured to construct a multi-modal three-dimensional scene representation based on the fusion feature set, and perform intelligent and automated investigation and analysis on a work site, and output a work site analysis result; The output module is configured to automatically generate a work scheme with spatial geometric constraints and work safety boundaries based on the multi-modal three-dimensional scene representation and the work site analysis result.

9. An electronic device, comprising: The electronic device comprises a processor, a communication bus, a user interface, a network interface, and a memory, the memory is configured to store instructions, the user interface and the network interface are configured to communicate with other devices, the communication bus is configured to realize connection and communication between components in the electronic device, and the processor is configured to execute the instructions stored in the memory, so that the electronic device performs the method as claimed in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores instructions, and when the instructions are executed, the method as claimed in any one of claims 1-7 is performed.

Images