Multi-source 5g message video cooperative monitoring method and system

Abstract

The application discloses a kind of multi-source 5G message video cooperative monitoring method and system, specifically related to video monitoring technical field, for solving the problem of multi-camera calibration parameter misalignment, video fusion misplacement caused by thermal deformation asynchrony in prior art;By deploying temperature sensor in industrial camera support structure, calculate the thermal deformation three-dimensional displacement compensation vector, according to adjacent camera displacement gradient and material thermal conductivity coefficient threshold value mark priority transmission object, transmit displacement vector to edge computing node by 5G message according to priority, dynamically correct spatial calibration parameters at node end by fusing all displacement data, generate cooperative calibration matrix, finally utilize the matrix to real-time registration of multiple video streams, execute workpiece defect detection, guarantee the geometric consistency of multi-camera system in thermal deformation environment, improve visual detection precision and stability.

Description

Technical Field

[0001] This invention relates to the field of video surveillance technology, and more specifically, to a multi-source 5G message video collaborative monitoring method and system. Background Technology

[0002] In industrial automation visual inspection, high-precision production quality inspection can be achieved through multi-camera collaborative monitoring. By deploying multiple industrial cameras to synchronously acquire workpiece images from different angles, the images are transmitted to the processing terminal via closed-circuit television network. Multi-source video fusion technology is used to realize three-dimensional reconstruction and defect analysis. Existing technologies usually rely on preset spatial calibration parameters to establish the coordinate mapping relationship of each camera's perspective, thereby ensuring the geometric consistency of multi-view image data.

[0003] However, the monitoring equipment that operates continuously in industrial sites for a long time causes thermal expansion of metal structural components due to temperature changes, resulting in slight changes in the physical pose of each camera. Due to the differences in the position, material, and heat dissipation conditions of different cameras in the production line, their thermal deformation degree is significantly asynchronous, causing the spatial calibration parameters of the multi-camera system to gradually become inaccurate. Ultimately, this leads to spatial misalignment when multi-source video streams are collaboratively fused, causing the visual inspection accuracy to continuously deteriorate over time. Summary of the Invention

[0004] In order to overcome the above-mentioned defects of the prior art, the present invention provides a multi-source 5G message video collaborative monitoring method and system to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution:

[0006] A method for collaborative monitoring of multi-source 5G message video includes the following steps:

[0007] S1. Deploy temperature sensors on the support structure of each industrial camera in the industrial site, and continuously collect temperature data of the structural components of the corresponding industrial camera through the temperature sensors.

[0008] S2. Based on the thermal expansion coefficient of the material of each industrial camera support structure and the collected temperature data of the structural components, calculate the three-dimensional displacement compensation vector of each industrial camera caused by thermal deformation.

[0009] S3. Calculate the three-dimensional displacement compensation vector gradient of adjacent cameras based on the spatial position of the industrial camera to mark the priority transmission object;

[0010] S4. The three-dimensional displacement compensation vector of each industrial camera is transmitted to the edge computing node in real time via 5G messages, and the objects to be transmitted first are given the highest transmission priority.

[0011] S5. Receive the three-dimensional displacement compensation vectors of all industrial cameras at the edge computing node, correct the spatial calibration parameters of the multi-camera system based on the three-dimensional displacement compensation vectors, and generate a collaborative calibration matrix.

[0012] S6. Real-time spatial registration of video streams acquired by multiple industrial cameras is performed using a collaborative calibration matrix to perform visual defect detection on industrial workpieces.

[0013] Furthermore, deploying temperature sensors on the support structure of each industrial camera in the industrial field includes:

[0014] The temperature sensor is attached to the deformation-sensitive area of ​​the metal bracket of an industrial camera;

[0015] Temperature data of structural components are continuously collected using a temperature sensor at a fixed sampling frequency.

[0016] The collected temperature data of structural components is stored in a local cache queue.

[0017] Furthermore, the calculation of the three-dimensional displacement compensation vector for each industrial camera caused by thermal deformation includes:

[0018] Query the material property database based on the coefficient of thermal expansion of the material supporting structure of the industrial camera;

[0019] The linear expansion of the structural component in a three-dimensional spatial coordinate system is calculated based on the thermal expansion displacement formula.

[0020] The linear dilatation is converted into a three-dimensional displacement compensation vector of the optical center point of the industrial camera.

[0021] Furthermore, calculating the gradient of the three-dimensional displacement compensation vector between adjacent cameras based on the spatial position of the industrial camera includes:

[0022] Obtain the installation position coordinates of the industrial camera in the production line coordinate system;

[0023] Calculate the spatial distance vector between adjacent industrial cameras;

[0024] Calculate the vector difference between the three-dimensional displacement compensation vectors of adjacent industrial cameras;

[0025] Dividing the vector difference by the magnitude of the spatial distance vector yields the gradient of the three-dimensional displacement compensation vector.

[0026] Furthermore, the objects marked for priority transmission include:

[0027] The gradient of the three-dimensional displacement compensation vector is compared with the threshold of the thermal conductivity of the supporting structure material;

[0028] When the gradient of the three-dimensional displacement compensation vector exceeds the thermal conductivity threshold, a priority transfer object marker is generated.

[0029] Associate the priority transmission object tag with the corresponding industrial camera's device identifier.

[0030] Furthermore, transmitting the three-dimensional displacement compensation vector to the edge computing node in real time via 5G messaging includes:

[0031] Encapsulate the three-dimensional displacement compensation vector into a JSON format data packet;

[0032] Add a highest transmission priority identifier to the data packets of the priority transmission object;

[0033] Data packets are routed to edge computing nodes via the 5G user plane functional unit.

[0034] Furthermore, the spatial calibration parameters of the multi-camera system are corrected based on the three-dimensional displacement compensation vector, including:

[0035] Read the initial spatial calibration parameter matrix of the multi-camera system;

[0036] The 3D displacement compensation vector of each industrial camera is converted into a homogeneous coordinate transformation matrix.

[0037] Multiplying the homogeneous coordinate transformation matrix by the initial spatial calibration parameter matrix yields the corrected spatial calibration parameters.

[0038] Furthermore, generating the collaborative calibration matrix includes:

[0039] Input the corrected spatial calibration parameters into the multi-view geometry fusion engine;

[0040] The projection matrix of each industrial camera is optimized using a bundle adjustment algorithm.

[0041] Output a collaborative calibration matrix that fuses the perspectives of all industrial cameras.

[0042] Furthermore, real-time spatial registration of the video stream using a co-calibration matrix includes:

[0043] Load the co-calibration matrix into the graphics processor;

[0044] Perform perspective transformation operations on video streams captured by multiple industrial cameras;

[0045] The transformed video stream is mapped to a unified world coordinate system.

[0046] On the other hand, the present invention provides a multi-source 5G message video collaborative monitoring system, comprising the following modules:

[0047] The temperature acquisition module is used to deploy temperature sensors on the support structure of each industrial camera in the industrial field, and continuously collect temperature data of the structural components of the corresponding industrial camera through the temperature sensors.

[0048] The compensation calculation module is used to calculate the three-dimensional displacement compensation vector of each industrial camera caused by thermal deformation based on the material thermal expansion coefficient of each industrial camera support structure and the collected temperature data of the structural components.

[0049] The gradient calculation module is used to calculate the three-dimensional displacement compensation vector gradient of adjacent cameras based on the spatial position of the industrial camera to mark the priority transmission objects.

[0050] The real-time transmission module is used to transmit the three-dimensional displacement compensation vector of each industrial camera to the edge computing node in real time via 5G messages, with priority given to the objects to be transmitted.

[0051] The collaborative calibration module is used to receive the three-dimensional displacement compensation vectors of all industrial cameras at the edge computing node, correct the spatial calibration parameters of the multi-camera system based on the three-dimensional displacement compensation vectors, and generate a collaborative calibration matrix.

[0052] The defect detection module is used to perform real-time spatial registration of video streams acquired by multiple industrial cameras using a collaborative calibration matrix, thereby performing visual defect detection on industrial workpieces.

[0053] Compared with the prior art, the present invention has the following beneficial effects:

[0054] 1. A temperature sensor is used to monitor the thermal deformation of the industrial camera support structure in real time. Based on the thermal expansion characteristics of the material, the three-dimensional displacement compensation vector of each camera is accurately calculated, effectively quantifying the microscopic impact of thermal deformation on camera pose. By analyzing the gradient changes of the displacement vectors of adjacent cameras and dynamically prioritizing transmission based on the thermal conductivity of the support material, the asynchronous nature of multi-camera thermal deformation is specifically addressed. 5G messaging is used to transmit displacement data to edge nodes according to priority, ensuring the real-time data availability in areas of severe deformation and preventing calibration parameter lag caused by transmission delays from the source. The displacement compensation amounts of all cameras are fused at the edge, dynamically correcting the spatial calibration parameters of the multi-camera system and generating a collaborative calibration matrix, significantly improving the geometric consistency of multi-view video streams under high-temperature conditions. Finally, video spatial registration is performed based on the real-time updated calibration matrix, eliminating accumulated errors from thermal deformation and ensuring the long-term stability and robustness of visual defect detection for industrial workpieces.

[0055] 2. By deeply coupling material physical properties (coefficient of thermal expansion, thermal conductivity) with 5G transmission strategies and dynamic calibration technology, a differentiated transmission of key data is achieved through a displacement gradient threshold decision mechanism, overcoming the limitations of traditional fixed calibration parameters. Edge nodes directly reconstruct camera projection relationships using displacement compensation vectors, avoiding complex global recalibration processes and significantly reducing computational latency. The collaborative calibration matrix continuously adapts to the actual pose changes of the camera, enabling pixel-level spatial alignment of multiple video streams in a unified coordinate system. This fundamentally solves the problem of misalignment in multi-source video fusion caused by the asynchronous nature of thermal deformation, ensuring the sustainable operation of high-precision visual inspection systems in harsh industrial environments. Attached Figure Description

[0056] Figure 1 This is a flowchart of a multi-source 5G message video collaborative monitoring method according to the present invention;

[0057] Figure 2 This is a schematic diagram of the structure of a multi-source 5G message video collaborative monitoring system according to the present invention. Detailed Implementation

[0058] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of the present invention.

[0059] Example 1: Figure 1 This invention presents a multi-source 5G message video collaborative monitoring method, which includes the following steps:

[0060] S1. Deploy temperature sensors on the support structure of each industrial camera in the industrial site, and continuously collect temperature data of the structural components of the corresponding industrial camera through the temperature sensors.

[0061] S2. Based on the thermal expansion coefficient of the material of each industrial camera support structure and the collected temperature data of the structural components, calculate the three-dimensional displacement compensation vector of each industrial camera caused by thermal deformation.

[0062] S3. Calculate the three-dimensional displacement compensation vector gradient of adjacent cameras based on the spatial position of the industrial camera to mark the priority transmission object;

[0063] S4. The three-dimensional displacement compensation vector of each industrial camera is transmitted to the edge computing node in real time via 5G messages, and the objects to be transmitted first are given the highest transmission priority.

[0064] S5. Receive the three-dimensional displacement compensation vectors of all industrial cameras at the edge computing node, correct the spatial calibration parameters of the multi-camera system based on the three-dimensional displacement compensation vectors, and generate a collaborative calibration matrix.

[0065] S6. Real-time spatial registration of video streams acquired by multiple industrial cameras is performed using a collaborative calibration matrix to perform visual defect detection on industrial workpieces.

[0066] Deploying temperature sensors on the support structure of each industrial camera in an industrial setting involves the following steps: First, determine the location of the deformation-sensitive area of ​​the industrial camera's metal bracket. This location is determined through thermal stress simulation analysis, specifically selecting the area with the largest coefficient of thermal expansion and closest to the optical center of the industrial camera. Next, attach the temperature sensor to the surface of this deformation-sensitive area using high-temperature thermally conductive adhesive, ensuring complete contact between the sensor and the metal bracket surface. Continuously collect temperature data of the structural components using the temperature sensor at a fixed sampling frequency of once per second. This sampling frequency is determined based on the thermal response time of the metal material to ensure the capture of rapid temperature fluctuations. Store the collected temperature data of the structural components in a local cache queue in chronological order. This cache queue uses a first-in, first-out (FIFO) storage mechanism, and its capacity is set to store temperature data generated during continuous operation, ensuring no data loss occurs during data transmission interruptions.

[0067] Calculating the three-dimensional displacement compensation vector of each industrial camera due to thermal deformation includes the following operations: Querying a pre-set material property library based on the material type of the industrial camera support structure. This library contains reference values ​​for the thermal expansion coefficients of commonly used metallic materials. Calculating the linear expansion of the structural component in the three-dimensional coordinate system based on the thermal expansion displacement formula, where the linear expansion equals the material's thermal expansion coefficient multiplied by the original length of the structural component multiplied by the temperature change. Converting the linear expansion into a three-dimensional displacement compensation vector for the optical center point of the industrial camera. This conversion is achieved by establishing a geometric mapping relationship between the deformation of the metal support and the displacement of the optical center point, specifically by establishing a displacement transfer model based on the mechanical mounting structure of the industrial camera.

[0068] When querying the material property database, the material property database is stored in the non-volatile memory of the edge computing node. It contains a mapping table between material names and coefficients of thermal expansion, which is constructed by referencing the national standard material properties database. For the support structure of composite materials, the weighted average of the coefficients of thermal expansion of each component material is taken, and the weight is determined according to the volume ratio of each material in the structure.

[0069] When calculating the linear expansion of structural components, the original length of the structural component refers to the straight-line distance from the installation position of the temperature sensor to the center of the fixing bolt of the industrial camera. This distance is obtained through the industrial camera installation drawings. The temperature change is calculated by subtracting the preset reference temperature value from the currently collected temperature data of the structural component. The reference temperature value is set as the initial temperature when the industrial ambient temperature is stable.

[0070] When transforming the three-dimensional displacement compensation vector, the displacement transfer model is established based on the principle of rigid body kinematics. Specifically, the metal bracket is regarded as a rigid body, and its thermal deformation causes the industrial camera to translate and rotate as a whole. The three-dimensional displacement compensation vector of the optical center point is calculated through homogeneous coordinate transformation. The input parameters include linear expansion, bracket geometry, and the installation position coordinates of the industrial camera's optical center relative to the bracket.

[0071] The specific process of establishing the displacement transfer model includes: defining the center of the industrial camera fixing bolt as the origin of the coordinate system; determining the coordinates of the temperature sensor installation position in the coordinate system based on the geometric dimensions of the bracket; calculating the displacement of the key points of the bracket based on the linear expansion; fitting the rigid body transformation matrix using the least squares method; applying the transformation matrix to the coordinates of the optical center point to output a three-dimensional displacement compensation vector.

[0072] In homogeneous coordinate transformation calculation, the transformation matrix contains translation and rotation components. The translation components are directly taken from the linear expansion components on the three coordinate axes. The rotation components are simplified to micro-rotations around a specific axis according to the support constraint conditions. The rotation angle is calculated by the cross product of the vectors before and after the support deformation.

[0073] Each component of the three-dimensional displacement compensation vector is in millimeters, with an accuracy retained to three decimal places. This accuracy meets the registration requirements of industrial vision inspection. The calculation process is executed in real time in the embedded processor of the industrial camera, and the three-dimensional displacement compensation vector is output once per temperature sampling cycle.

[0074] The local cache queue is implemented using a circular buffer, with the buffer size being the storage space, which can store temperature data generated during continuous operation. When the queue is full, it automatically overwrites the oldest temperature data and triggers a data transmission alarm. Queue read operations are performed in batches in chronological order during data transmission, and the corresponding storage space is released after each read.

[0075] The temperature sensor is a platinum resistance temperature sensor, which measures the temperature variation range in the industrial environment and meets the position compensation requirements of industrial cameras. The sensor signal is converted into a digital temperature value through an analog-to-digital converter, and the conversion resolution is set to meet the position compensation accuracy requirements of industrial cameras.

[0076] The fixed sampling frequency is set based on the thermal response time constant of the metallic material, which is determined through material thermal conduction experiments; for aluminum alloy brackets, a sampling frequency of once per second can capture typical temperature change processes in industrial settings.

[0077] The thermal stress simulation analysis of the deformation-sensitive area was performed using the finite element method. A three-dimensional model of the metal support was established in computer-aided engineering software, and a thermal load of the industrial site temperature variation range was applied to analyze and obtain the maximum thermal stress distribution area. The center point of this area was determined as the installation location of the temperature sensor.

[0078] The optical center point of an industrial camera is defined as the principal point on the image side of the industrial camera lens. The coordinates of this point are obtained through the factory calibration parameters of the industrial camera. In the displacement transfer model, it participates in coordinate transformation as a fixed point on a rigid body.

[0079] The maintenance of the material property library includes regularly updating new material data. When updating, the latest material parameter table is downloaded from the national standard material property database, converted into a new format, and then imported into the non-volatile memory of the edge computing node.

[0080] The calculation process of the three-dimensional displacement compensation vector is equipped with an anomaly handling mechanism. When the input temperature data changes abnormally, the weighted average of the temperature data of the previous three cycles is used to replace the current value, and the weight coefficient is dynamically adjusted according to the temperature change trend.

[0081] The installation position coordinates of the industrial camera in the production line coordinate system are obtained by reading the production line layout database, which stores the precise installation position coordinate data of the industrial camera in three-dimensional space. The production line coordinate system takes the starting end of the production line as the origin, and the three coordinate axes are parallel to the length, width, and height directions of the production line, respectively. The installation position coordinates are obtained by on-site surveying using a total station during the installation of the industrial camera, and the surveying accuracy meets the spatial positioning accuracy required by the industrial vision inspection system. The installation position coordinates of each industrial camera contain three coordinate axis components, which represent the position coordinates of the optical center point of the industrial camera in the production line coordinate system. The coordinate values ​​are stored in meters and retain four decimal places.

[0082] When calculating the spatial distance vector between adjacent industrial cameras, the pairing list of adjacent industrial cameras is first determined based on the production line layout topology. For each pair of adjacent industrial cameras, the installation position coordinates of the two industrial cameras are read from the production line layout database. The installation position coordinates of the two industrial cameras are then subtracted to obtain the spatial distance vector from the first industrial camera to the second industrial camera. The three components of the spatial distance vector represent the distance difference between the two industrial cameras in the three coordinate axes of the production line coordinate system, with the distance difference in meters and four decimal places.

[0083] When calculating the vector difference of the three-dimensional displacement compensation vectors of adjacent industrial cameras, the three-dimensional displacement compensation vector data of the adjacent industrial cameras in the current calculation cycle are obtained from the local cache. The three-dimensional displacement compensation vector contains three components, which represent the displacement compensation amount of the optical center point of the industrial camera in the three-dimensional spatial coordinate system in the three-dimensional coordinate system. The three-dimensional displacement compensation vectors of the two industrial cameras are subtracted to obtain the vector difference of the three-dimensional displacement compensation vectors. The three components of the vector difference represent the difference of the displacement compensation amount of the two industrial cameras in the three coordinate axis directions. The unit of the displacement compensation difference is millimeters, and three significant figures are retained after the decimal point.

[0084] When obtaining the gradient of the three-dimensional displacement compensation vector by dividing the vector difference by the magnitude of the spatial distance vector, the magnitude of the spatial distance vector is first calculated, which is the square root of the sum of the squares of the three components of the spatial distance vector. Then, the magnitude of the difference of the three-dimensional displacement compensation vector is calculated, which is the square root of the sum of the squares of the three components of the vector difference. The magnitude of the difference of the three-dimensional displacement compensation vector is divided by the magnitude of the spatial distance vector to obtain the gradient value of the three-dimensional displacement compensation vector. The gradient of the three-dimensional displacement compensation vector is dimensionless and represents the rate of change of displacement per unit length. The calculation result is retained to five decimal places.

[0085] When comparing the gradient of the three-dimensional displacement compensation vector with the thermal conductivity threshold of the supporting structure material, the main material type of the industrial camera supporting structure is first queried; the corresponding thermal conductivity reference value is obtained from the material property library according to the material type; the thermal conductivity reference value is multiplied by a preset proportional coefficient to obtain the thermal conductivity threshold, which is determined based on industrial field heat conduction characteristic experiments; the comparison operation is implemented through a numerical comparator, and when the gradient value of the three-dimensional displacement compensation vector is greater than the thermal conductivity threshold, a logical true value is output, otherwise a logical false value is output.

[0086] When the gradient of the three-dimensional displacement compensation vector exceeds the thermal conductivity threshold, a priority transmission object marker is generated. The marker generation operation is triggered when the comparison operation outputs a logical truth value. The priority transmission object marker is a Boolean type data, and a truth value indicates that the industrial camera needs to be transmitted first. The marker generation operation is executed in the marker management unit of the edge computing node, and each marker corresponds to one industrial camera device. The marker state is maintained for one calculation cycle and is automatically reset at the beginning of a new calculation cycle.

[0087] When associating a priority transmission object tag with the corresponding industrial camera's device identifier, the first step is to obtain the unique device identifier of the current industrial camera, which is the media access control address of the industrial camera in network communication. A mapping relationship between the priority transmission object tag and the media access control address is established, and this mapping relationship is stored in the tag mapping table of the edge computing node. During the data transmission phase, the transmission priority of each industrial camera data packet is determined by querying the tag mapping table.

[0088] The three-dimensional displacement compensation vector gradient calculation process is executed in the vector processing unit of the edge computing node. Each calculation cycle processes all adjacent industrial camera pairs. For each pair of adjacent industrial cameras, the calculated three-dimensional displacement compensation vector gradient value is stored in the gradient buffer. The buffer data is arranged in time series. The gradient buffer adopts a circular storage structure, and the storage depth meets the data storage requirements of the largest temperature fluctuation cycle in the industrial site.

[0089] The preset proportional coefficient of the thermal conductivity threshold is determined based on the thermal conduction characteristics of the material. For metallic materials, the value is between 0.7 and 0.9, and for non-metallic materials, the value is between 0.5 and 0.7. The specific value is calibrated through thermal conduction experiments. The experimental method is to measure the actual thermal deformation gradient distribution within a typical temperature change range in an industrial setting, so that the threshold covers 95% of the abnormal gradient values.

[0090] The tag management unit is implemented using a state machine mechanism, which includes an idle state, a tag generation state, and a tag association state. When a gradient comparison result is received, it enters the tag generation state, generates a tag for the priority transmission object, and then enters the tag association state. After the tag association operation is completed, it returns to the idle state and waits for the next calculation cycle to be triggered.

[0091] The adjacent industrial camera pairing list is automatically generated based on the spatial location of the industrial cameras. The pairing rule is that the two industrial cameras that are closest to each other in the production line coordinate system form a pair. When there are multiple industrial cameras that are close to each other, the three industrial cameras with the smallest distance are selected to establish a fully connected pairing relationship. The pairing list is generated and stored in the configuration memory during system initialization, and is recalculated and generated when the production line layout changes.

[0092] The spatial distance vector magnitude calculation uses a square root algorithm, and Newton's iteration method is used for square root operations to ensure real-time calculation. The number of iterations is set to three to meet the accuracy requirements of the industrial vision inspection system. The same algorithm is used to calculate the magnitude of the difference in the three-dimensional displacement compensation vector to ensure consistency in the calculation process.

[0093] The device identifier association operation is performed during the data transmission preparation phase. When it is necessary to transmit the three-dimensional displacement compensation vector data of the industrial camera, the marker mapping table is queried to obtain the marker status of the current priority transmission object. The data transmission priority flag is set according to the marker status, and this flag is embedded in the header control field of the data transmission protocol.

[0094] The numerical comparator for the gradient comparison operation uses a comparator circuit design with hysteresis characteristics to prevent frequent state switching near the threshold. The hysteresis interval is set to ±5% of the thermal conductivity threshold. When the gradient value of the three-dimensional displacement compensation vector enters the hysteresis interval, the previous comparison result remains unchanged.

[0095] The storage of priority transfer object tags adopts a non-volatile memory backup mechanism, which automatically saves the current tag state when the system is powered off; after the system is powered on again, the tag state is restored from the non-volatile memory to maintain the continuity of the compensation strategy.

[0096] An outlier detection mechanism is set up during the three-dimensional displacement compensation vector gradient calculation process. When the magnitude of the spatial distance vector is less than the minimum installation spacing of the industrial camera, it is judged as a calculation anomaly, the current calculation result is discarded and the gradient value of the previous cycle is used to replace it. The minimum installation spacing of the industrial camera is determined according to the production line layout design specifications.

[0097] Encapsulating the 3D displacement compensation vector into a JSON format data packet involves the following operations: First, a JSON object data structure is created, containing a device identifier field, a timestamp field, and a 3D displacement compensation vector field. The device identifier field is written with the media access control address string of the industrial camera. The timestamp field is written with the Coordinated Universal Time (UTC) time value at the time of data acquisition. The 3D displacement compensation vector field is created as an array containing three numerical elements, corresponding to the displacement compensation amounts in the three coordinate axes. The three component values ​​of the 3D displacement compensation vector are then sequentially filled into the array elements. A data verification field is added, and the cyclic redundancy check (CRC) code of the JSON object content is calculated and stored in a fixed location. Finally, the complete JSON object is serialized into a UTF-8 encoded binary data stream to form a data packet. The data packet size is controlled within the maximum transmission unit allowed by the 5G message transmission protocol to ensure the integrity of a single transmission.

[0098] Adding a highest transmission priority identifier to data packets of priority transmission objects includes the following operations: adding a custom control field to the JSON data packet header; this control field contains a four-bit priority flag, which is set to the highest priority encoded value when the industrial camera is marked as a priority transmission object; the highest priority encoded value corresponds to the guaranteed bit rate bearer level in the 5G Quality of Service (QoS) level; adding a digital signature field to the data packet tail, using the industrial camera's device private key to perform a hash operation on the data packet content and then encrypting it to generate a digital signature; and assigning a dedicated QoS flow identifier to the data packets of priority transmission objects, which is pre-configured as the highest priority transmission channel in the 5G core network.

[0099] The routing of data packets to edge computing nodes via the 5G user plane functional unit includes the following operations: the communication module of the industrial camera establishes a wireless bearer connection with the 5G base station; the encapsulated data packet is submitted to the application layer of the 5G protocol stack; the application layer selects the corresponding protocol data unit session based on the quality of service flow identifier; the user plane functional unit detects the priority flag bit in the data packet header, and when the highest priority encoding value is identified, the data packet is inserted into the high-priority transmission queue; the user plane functional unit forwards the data packet to the user plane gateway of the edge computing node through the N3 interface of the 5G core network; after receiving the data packet, the edge computing node verifies the validity of the digital signature and confirms the legality of the data source.

[0100] Reading the initial spatial calibration parameter matrix of the multi-camera system involves the following operations: loading the initial spatial calibration parameter file from the non-volatile memory of the edge computing node; this file stores the calibration parameter matrix of the multi-camera system at a reference temperature; the calibration parameter matrix is ​​a fourth-order square matrix containing twelve independent parameter elements; parsing the file content into a matrix data structure and storing it in a memory buffer; verifying the integrity of the matrix data, checking whether the matrix determinant value is within a valid range, and ensuring that the matrix is ​​invertible; initializing the calibration parameters and updating the working area, and copying the initial matrix to the working area as the input reference for correction operations.

[0101] Converting the 3D displacement compensation vector of each industrial camera into a homogeneous coordinate transformation matrix involves the following operations: creating a fourth-order identity matrix for each industrial camera as the basis of the transformation matrix; taking the three component values ​​of the 3D displacement compensation vector as the translation components of the transformation matrix; setting the translation components in the first three elements of the fourth column of the transformation matrix; determining the rotation components based on the constraints of the industrial camera support structure, ignoring the influence of rotation under small deformation conditions, and setting the rotation components to zero; setting the scaling components to a unit scaling factor; finally forming a homogeneous coordinate transformation matrix containing translation components but without rotation scaling; storing the matrix elements in single-precision floating-point format to meet the accuracy requirements of industrial vision inspection.

[0102] The process of multiplying the homogeneous coordinate transformation matrix with the initial spatial calibration parameter matrix to obtain the corrected spatial calibration parameters involves the following operations: traversing all industrial cameras in the order of their identification; for each industrial camera, reading the corresponding homogeneous coordinate transformation matrix and initial spatial calibration parameter matrix from memory; performing a multiplication operation on the two matrices, with the order of multiplication being left-multiplying the homogeneous coordinate transformation matrix by the initial spatial calibration parameter matrix; using the standard algorithm of row-to-column multiplication for matrix multiplication, calculating the sum of each element through accumulation; writing the calculation result into the corrected spatial calibration parameter matrix; obtaining the global corrected spatial calibration parameter matrix after traversal; and performing an orthogonality check on the corrected matrix to ensure that it maintains its rigid transformation characteristics.

[0103] During the encapsulation of the JSON format data packet, the value of the three-dimensional displacement compensation vector field is in millimeters and is represented using the IEEE 754 single-precision floating-point format; the timestamp field adopts the Unix timestamp format, accurate to the millisecond level; the cyclic redundancy check code is generated using the CRC-32 standard algorithm, and the check polynomial is a polynomial commonly used in industrial communication standards.

[0104] The priority flag setting rule is as follows: when an industrial camera is marked as a priority transmission object, the flag is set to a binary value; this value is mapped to the service quality level indicator value in the 5G communication protocol, corresponding to the guaranteed bit rate service bearing level; the dedicated service quality flow identifier is allocated and configured to the industrial camera terminal by the 5G core network during system initialization.

[0105] During packet routing, the user plane functional unit maintains two transmission queues: a high-priority queue and a normal-priority queue. The high-priority queue adopts a strict priority scheduling strategy to ensure that packets in the queue are sent first. When packets are transmitted through the N3 interface, they are encapsulated in a GTP-U protocol tunnel, and the tunnel header is marked with a quality of service level indicator value.

[0106] The initial spatial calibration parameter matrix is ​​stored in binary file format, with the file header containing matrix dimension and data type information. During loading, the file header is read first to verify format compatibility, and then the matrix elements are parsed in row-major order. Integrity verification includes calculating the matrix determinant value and checking whether it is within the valid range of the rigid transformation matrix.

[0107] When constructing the homogeneous coordinate transformation matrix, the translation components are directly obtained from the three-dimensional displacement compensation vector. For special installation structures with rotational effects, the rotation angle is calculated based on the deformation characteristics of the bracket, but the maximum rotation angle is limited to the range allowed by the industrial vision inspection system to avoid over-correction.

[0108] The matrix multiplication operation employs an optimized algorithm, utilizing a single instruction multiple data stream instruction set to process the matrix element multiplication and accumulation in parallel. The calculation process for each output element is as follows: take the dot product of the row vector of the left matrix and the column vector of the right matrix; the dot product operation is implemented through multiplication and accumulation instructions; after all elements are calculated, a matrix transpose storage operation is performed to maintain the continuity of memory access.

[0109] The data packet receiving end processing includes: decapsulating the GTP-U tunnel protocol at the user plane gateway of the edge computing node; extracting the JSON data packet and verifying the cyclic redundancy check code; after successful verification, parsing the JSON object, extracting the three-dimensional displacement compensation vector and storing it in the compensation vector database; and simultaneously reading the priority flag status and updating the industrial camera transmission status record.

[0110] Before applying the corrected spatial calibration parameter matrix, a validity check is performed: whether the rotation matrix part of the calculation matrix is ​​orthogonal is checked by checking whether the transpose and inverse matrix of the matrix are equal; whether the translation component is within a reasonable range for industrial scenarios; if the check fails, an error handling process is triggered to revert to the previous valid parameter version.

[0111] During the 3D displacement compensation vector conversion process, a displacement threshold is set. When the magnitude of the 3D displacement compensation vector exceeds the maximum allowable displacement of the industrial camera, it is determined to be a sensor malfunction, the current data is discarded and replaced with the historical average value. The maximum allowable displacement of the industrial camera is calculated and determined based on the mechanical structure constraints.

[0112] Matrix multiplication operations employ a block-based computation strategy, decomposing large matrices into sub-matrix blocks and leveraging the principle of locality of reference in processor caches to improve computational efficiency. The block size is optimized based on the processor architecture of edge computing nodes to balance computational performance and memory usage.

[0113] Inputting the corrected spatial calibration parameters into the multi-view geometric fusion engine involves the following operations: Initiating the multi-view geometric fusion process at the edge computing node; this process creating a data input interface to receive the corrected spatial calibration parameter matrix; transmitting the parameter matrix in a fourth-order square matrix format to the fusion engine's memory workspace; dividing the memory workspace into a parameter buffer and a computation area; storing the corrected spatial calibration parameter matrices for all industrial cameras in the parameter buffer; allocating iterative computation space for the bundle adjustment algorithm in the computation area; performing format validation when inputting parameters to ensure that each matrix satisfies the rigid transformation constraints; validation includes checking the orthogonality of the rotation matrix and the range of determinant values; and arranging the validated parameter matrices in the parameter buffer according to the industrial camera identification order.

[0114] Optimizing the projection matrix of each industrial camera using the bundle adjustment algorithm involves the following operations: reading the initial projection matrix of all industrial cameras from the parameter buffer; setting a virtual control point grid in 3D space, with the grid density determined according to the accuracy requirements of industrial field inspection; projecting the control points onto the image plane of each industrial camera and calculating the reprojection error; establishing an error minimization objective function, which includes the sum of squared reprojection errors and a regularization term; iteratively solving for the optimal projection matrix using the Levenberg-Marquardt optimization algorithm; updating the projection matrix parameters in each iteration until the change in the objective function is less than the convergence threshold; the convergence threshold is set according to the accuracy requirements of the industrial vision inspection system; and outputting the optimized projection matrix set after iteration.

[0115] Outputting a collaborative calibration matrix that fuses the views of all industrial cameras involves the following operations: performing matrix fusion on the optimized set of projection matrices; establishing geometric constraints between cameras based on their spatial relationships; solving for the globally optimal collaborative calibration matrix using singular value decomposition; the collaborative calibration matrix is ​​a fourth-order square matrix containing the unified coordinate transformation relationship of all industrial camera views; verifying the invertibility of the collaborative calibration matrix to ensure that its determinant value is within a valid range; serializing the verified collaborative calibration matrix into a binary file and storing it in the non-volatile memory of the edge computing node; and simultaneously sending a matrix ready notification signal.

[0116] Loading the co-calibration matrix into the graphics processor includes the following operations: upon receiving a matrix ready notification, reading the co-calibration matrix file from non-volatile memory; parsing the file content to recover the matrix data structure; allocating a dedicated storage area in the graphics processor's constant memory; transferring the co-calibration matrix data to the graphics processor's constant memory in row-major order; accelerating the transfer process using direct memory access channels; triggering an interrupt notification in the graphics processor upon completion of the transfer; and configuring the parameter register of the texture sampling unit in the graphics processor to set the co-calibration matrix as the reference transformation matrix for perspective transformation.

[0117] Performing perspective transformation on video streams acquired by multiple industrial cameras involves the following operations: the graphics processor's video input interface receives video stream data from multiple industrial cameras; each video stream is decomposed into consecutive video frames; a texture object is created for each video frame and loaded into the texture memory; the graphics processor's shader program is called to perform perspective transformation; the shader program applies a co-calibration matrix to each texture pixel for coordinate transformation; the coordinate transformation calculation uses homogeneous coordinate operations, raising the two-dimensional image coordinates to three-dimensional homogeneous coordinates before multiplying with the matrix; the calculation result is back-projected onto a two-dimensional plane to obtain the transformed coordinates; and a bilinear interpolation algorithm is used to calculate the transformed pixel values ​​to eliminate aliasing.

[0118] Mapping the transformed video stream to a unified world coordinate system involves the following operations: creating a unified world coordinate system rendering target in the graphics processor; aligning the world coordinate system with the production line reference point as the origin and the three coordinate axes aligned with the production line direction; mapping each perspective-transformed video frame to its corresponding area in the world coordinate system; determining the position and orientation of the video frame in the world coordinate system based on the installation location of the industrial camera; performing pixel-level fusion on multiple video frames in overlapping areas, using a weighted average algorithm to mix pixel values; and finally generating a seamlessly stitched global scene video stream.

[0119] The initialization of the multi-view geometric fusion engine includes: configuring the iteration parameters of the bundle adjustment algorithm, setting the maximum number of iterations to the value required for the real-time performance of the industrial vision inspection system; setting the virtual control point grid spacing to the value of the minimum feature size of the industrial workpiece; and adaptively adjusting the regularization term weight coefficient based on historical data of the industrial camera's position stability.

[0120] The execution process of the Levenberg-Marquardt optimization algorithm is as follows: calculate the Jacobian matrix of the objective function with respect to the projection matrix parameters; construct the Hessian matrix approximation based on the Jacobian matrix; add damping factors to the diagonal elements of the Hessian matrix; solve the linear equation system to obtain the parameter increments; decrease the damping factor when the objective function value decreases and increase the damping factor when it increases; iterate until the convergence condition is met.

[0121] Verification of the invertibility of the co-calibration matrix includes: calculating the determinant of the matrix and checking whether it is within the effective range of the rigid transformation matrix; performing eigenvalue decomposition and verifying whether the magnitudes of all eigenvalues ​​are close to unit values; and performing singular value decomposition and checking whether the singular values ​​are within a reasonable range.

[0122] The configuration of a graphics processor's shader program includes: writing a vertex shader to implement coordinate transformations; writing a fragment shader to implement pixel value interpolation; compiling the shader program into executable binary code of the graphics processor instruction set; and loading it into the shader core through the graphics processor driver.

[0123] Video stream fusion processing includes: establishing a mapping relationship from the world coordinate system to the screen coordinate system; creating a perspective transformation mesh for each video frame from the industrial camera; calculating the mesh vertex positions based on the co-calibration matrix; processing multiple video streams simultaneously using instantiation rendering technology; and performing a view frustum clipping operation in the geometry shader to remove invisible areas.

[0124] The bilinear interpolation algorithm is implemented as follows: for the transformed target pixel position, find the four nearest pixels in the source image; calculate the weight coefficients based on the relative positions of the target pixel and the four source pixels; take the weighted average of the color values ​​of the four source pixels as the target pixel value; and perform interpolation calculations on the luminance component and chrominance component respectively.

[0125] The convergence threshold is set based on the positioning accuracy requirements of the industrial vision inspection system. Convergence is determined when the projection error is less than the value of the feature size of the industrial workpiece. The maximum number of iterations is determined based on the real-time computing capability of the edge computing node to ensure that the optimization is completed within the video frame processing cycle.

[0126] The collaborative calibration matrix is ​​stored in binary format. The file header contains matrix dimension and data type information; the file body stores matrix elements in row-major order; and the file tail stores a cyclic redundancy check code for verifying data integrity.

[0127] The graphics processor's constant memory is configured as a read-only memory area, and the co-calibration matrix remains constant during video processing; each shader core can access the constant memory in parallel, ensuring the efficiency of large-scale pixel processing.

[0128] During the world coordinate system mapping process, a distance-weighted algorithm is used for pixel fusion in the overlapping area: the weight coefficient is calculated based on the distance from the pixel position to the optical center of each industrial camera, and the closer the distance, the greater the weight; the weight coefficient is normalized to ensure a smooth transition of pixel values ​​in the overlapping area.

[0129] Real-time performance assurance measures include: setting up multiple computing queues in the graphics processor to process multiple video streams in parallel; allocating dedicated computing resources to video streams from high-priority industrial cameras; and performing synchronization processing based on the timestamps of video frames to ensure time alignment of multiple video streams.

[0130] The anomaly handling mechanism includes: triggering a calibration anomaly alarm when a large blank area appears after perspective transformation; replacing illegal pixel values ​​with corresponding pixels from adjacent frames when pixel interpolation occurs; and automatically switching to a backup calibration matrix version when matrix loading fails.

[0131] Example 2: Figure 2 A schematic diagram of a multi-source 5G message video collaborative monitoring system according to the present invention is provided. The multi-source 5G message video collaborative monitoring system includes the following modules:

[0132] The temperature acquisition module is used to deploy temperature sensors on the support structure of each industrial camera in the industrial field, and continuously collect temperature data of the structural components of the corresponding industrial camera through the temperature sensors.

[0133] The compensation calculation module is used to calculate the three-dimensional displacement compensation vector of each industrial camera caused by thermal deformation based on the material thermal expansion coefficient of each industrial camera support structure and the collected temperature data of the structural components.

[0134] The gradient calculation module is used to calculate the three-dimensional displacement compensation vector gradient of adjacent cameras based on the spatial position of the industrial camera to mark the priority transmission objects.

[0135] The real-time transmission module is used to transmit the three-dimensional displacement compensation vector of each industrial camera to the edge computing node in real time via 5G messages, with priority given to the objects to be transmitted.

[0136] The collaborative calibration module is used to receive the three-dimensional displacement compensation vectors of all industrial cameras at the edge computing node, correct the spatial calibration parameters of the multi-camera system based on the three-dimensional displacement compensation vectors, and generate a collaborative calibration matrix.

[0137] The defect detection module is used to perform real-time spatial registration of video streams acquired by multiple industrial cameras using a collaborative calibration matrix, thereby performing visual defect detection on industrial workpieces.

[0138] All calculations involved in the embodiments are dimensionless numerical calculations, and the preset parameters and thresholds in the calculations are set by those skilled in the art according to the actual situation.

[0139] The above embodiments can be implemented, in whole or in part, by software, hardware, firmware, or any other combination thereof. When implemented using software, the above embodiments can be implemented, in whole or in part, in the form of a computer program product.

[0140] Those skilled in the art will recognize that the modules and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and inventive constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0141] In addition, the functional modules in the various embodiments of this application can be integrated into one processing module, or each module can exist physically separately, or two or more modules can be integrated into one module.

[0142] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or modules may be electrical, mechanical, or other forms.

[0143] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

[0144] In conclusion, the above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A multi-source 5G message video collaborative monitoring method, characterized in that, Includes the following steps: S1. Deploy temperature sensors on the support structure of each industrial camera in the industrial site, and continuously collect temperature data of the structural components of the corresponding industrial camera through the temperature sensors. S2. Based on the thermal expansion coefficient of the material of each industrial camera support structure and the collected temperature data of the structural components, calculate the three-dimensional displacement compensation vector of each industrial camera caused by thermal deformation. S3. Calculate the three-dimensional displacement compensation vector gradient of adjacent cameras based on the spatial position of the industrial camera to mark the priority transmission object; S4. The three-dimensional displacement compensation vector of each industrial camera is transmitted to the edge computing node in real time via 5G messages, and the objects to be transmitted first are given the highest transmission priority. S5. Receive the three-dimensional displacement compensation vectors of all industrial cameras at the edge computing node, correct the spatial calibration parameters of the multi-camera system based on the three-dimensional displacement compensation vectors, and generate a collaborative calibration matrix. S6. Real-time spatial registration of video streams acquired by multiple industrial cameras is performed using a collaborative calibration matrix to perform visual defect detection on industrial workpieces.

2. The multi-source 5G message video collaborative monitoring method according to claim 1, characterized in that, Temperature sensors are deployed on the support structure of each industrial camera in the industrial field, including: The temperature sensor is attached to the deformation-sensitive area of ​​the metal bracket of an industrial camera; Temperature data of structural components are continuously collected using a temperature sensor at a fixed sampling frequency. The collected temperature data of structural components is stored in a local cache queue.

3. The multi-source 5G message video collaborative monitoring method according to claim 2, characterized in that, The calculation of the three-dimensional displacement compensation vector for each industrial camera due to thermal deformation includes: Query the material property database based on the coefficient of thermal expansion of the material supporting structure of the industrial camera; The linear expansion of the structural component in a three-dimensional spatial coordinate system is calculated based on the thermal expansion displacement formula. The linear dilatation is converted into a three-dimensional displacement compensation vector of the optical center point of the industrial camera.

4. The multi-source 5G message video collaborative monitoring method according to claim 3, characterized in that, The calculation of the 3D displacement compensation vector gradient between adjacent cameras based on the spatial position of the industrial camera includes: Obtain the installation position coordinates of the industrial camera in the production line coordinate system; Calculate the spatial distance vector between adjacent industrial cameras; Calculate the vector difference between the three-dimensional displacement compensation vectors of adjacent industrial cameras; Dividing the vector difference by the magnitude of the spatial distance vector yields the gradient of the three-dimensional displacement compensation vector.

5. The multi-source 5G message video collaborative monitoring method according to claim 4, characterized in that, The objects marked for priority transmission include: The gradient of the three-dimensional displacement compensation vector is compared with the threshold of the thermal conductivity of the supporting structure material; When the gradient of the three-dimensional displacement compensation vector exceeds the thermal conductivity threshold, a priority transfer object marker is generated. Associate the priority transmission object tag with the corresponding industrial camera's device identifier.

6. The multi-source 5G message video collaborative monitoring method according to claim 5, characterized in that, Transmitting the three-dimensional displacement compensation vector to edge computing nodes in real time via 5G messaging includes: Encapsulate the three-dimensional displacement compensation vector into a JSON format data packet; Add a highest transmission priority identifier to the data packets of the priority transmission object; Data packets are routed to edge computing nodes via the 5G user plane functional unit.

7. The multi-source 5G message video collaborative monitoring method according to claim 6, characterized in that, The spatial calibration parameters of a multi-camera system based on 3D displacement compensation vector correction include: Read the initial spatial calibration parameter matrix of the multi-camera system; The 3D displacement compensation vector of each industrial camera is converted into a homogeneous coordinate transformation matrix. Multiplying the homogeneous coordinate transformation matrix by the initial spatial calibration parameter matrix yields the corrected spatial calibration parameters.

8. A multi-source 5G message video collaborative monitoring method according to claim 7, characterized in that, Generating the co-calibration matrix includes: Input the corrected spatial calibration parameters into the multi-view geometry fusion engine; The projection matrix of each industrial camera is optimized using a bundle adjustment algorithm. Output a collaborative calibration matrix that fuses the perspectives of all industrial cameras.

9. A multi-source 5G message video collaborative monitoring method according to claim 8, characterized in that, Real-time spatial registration of video streams using a co-calibration matrix includes: Load the co-calibration matrix into the graphics processor; Perform perspective transformation operations on video streams captured by multiple industrial cameras; The transformed video stream is mapped to a unified world coordinate system.

10. A multi-source 5G message video collaborative monitoring system, used to implement the multi-source 5G message video collaborative monitoring method according to any one of claims 1-9, characterized in that, Includes the following modules: The temperature acquisition module is used to deploy temperature sensors on the support structure of each industrial camera in the industrial field, and continuously collect temperature data of the structural components of the corresponding industrial camera through the temperature sensors. The compensation calculation module is used to calculate the three-dimensional displacement compensation vector of each industrial camera caused by thermal deformation based on the material thermal expansion coefficient of each industrial camera support structure and the collected temperature data of the structural components. The gradient calculation module is used to calculate the three-dimensional displacement compensation vector gradient of adjacent cameras based on the spatial position of the industrial camera to mark the priority transmission objects. The real-time transmission module is used to transmit the three-dimensional displacement compensation vector of each industrial camera to the edge computing node in real time via 5G messages, with priority given to the objects to be transmitted. The collaborative calibration module is used to receive the three-dimensional displacement compensation vectors of all industrial cameras at the edge computing node, correct the spatial calibration parameters of the multi-camera system based on the three-dimensional displacement compensation vectors, and generate a collaborative calibration matrix. The defect detection module is used to perform real-time spatial registration of video streams acquired by multiple industrial cameras using a collaborative calibration matrix, thereby performing visual defect detection on industrial workpieces.

Images