A 5G-based multi-sensor inspection system and method

Abstract

The invention discloses a multi-sensor inspection system and method based on 5G, comprising a dual-camera video punch-in module (1), an audio punch-in module (2), a video tag calibration module (3), an audio tag calibration module (4), a multi-sensor inspection point library module (5), a multi-sensor hybrid inspection point identification module (6), a Beidou / GPS positioning punch-in module (7), a 5G base station positioning punch-in module (8), a tag punch-in module (9), a 5G upload module (10), and an inspection chain storage module (11). The multi-sensor inspection method based on 5G fills the defects of existing tags and Beidou / GPS inspection punch-in, the tags need to be set in advance, and the Beidou / GPS no signal problem, and uses video object depth information and sound frequency characteristics to perform inspection punch-in. The multi-sensor inspection method based on 5G uses multiple sensors to increase the credibility of the inspection and avoids the problem that a single sensor inspection is easy to falsify.

Description

Technical Field

[0001] The present invention relates to the field of inspection, and in particular to a 5G-based multi-sensor inspection system and method. Background Art

[0002] Currently, patrol and inspection systems are widely used in offices, commercial areas, factories, residential areas, hotels, warehouses, and other places. There are two main methods for patrol and inspection: tag-based patrol, which requires pre-placed tags. Beidou / GPS-based patrol, which is flexible but can be difficult to locate due to weak Beidou / GPS signals in areas with obstructions, limiting its range.

[0003] In the prior art, there is a GPS- and RFID-based power equipment inspection system, published as CN103714586A. This system includes a handheld terminal and a local management server. The handheld terminal includes a GPS antenna, a GPS receiver circuit, an RFID identification circuit, and a communication interface. This system utilizes a combined GPS and RFID technology, enabling easy access to equipment information and accurate positioning without contact or aiming, making it easy to operate. However, its inspection method is limited, and clocking in may fail in the presence of obstructions or when the GPS signal is weak. Summary of the Invention

[0004] The purpose of the present invention is to provide a 5G-based multi-sensor inspection system and method to fill the defects of existing tags and Beidou / GPS inspection check-in, use multiple sensors to increase the credibility of inspection, and avoid the problem of easy fraud in single sensor inspection.

[0005] To achieve the above objectives, the present invention provides the following technical solutions:

[0006] A 5G-based multi-sensor inspection system, including:

[0007] Dual-camera video clocking module: uses two cameras to sample video, sends the inspection point ID and the obtained image object list to the video label calibration module for video calibration, and sends the image object list to the multi-sensor hybrid inspection point recognition module for video clocking;

[0008] Audio clocking module: used to collect surrounding sounds, generate PCM audio data for data slicing, send the obtained frequency domain data slices to the audio tag calibration module for audio calibration, and send the frequency domain data slices to the multi-sensor hybrid inspection point identification module for audio clocking;

[0009] Video label calibration module: Receives the image object list sent by the dual-camera video punch-in module, obtains the image object list data of all inspection points from the multi-sensor inspection point library module, calibrates it, and then sends the image object list and inspection point ID to the multi-sensor inspection point library module;

[0010] Audio tag calibration module: Receives the frequency domain data slices sent by the audio punch module, obtains the frequency domain data slices of all inspection points from the multi-sensor inspection point library module, calibrates them, and then sends the frequency domain data slices and inspection point IDs to the multi-sensor inspection point library module;

[0011] Multi-sensor inspection point library module: saves and queries inspection point information, which includes inspection point ID, tag ID, location coordinates, frequency domain data slices, and image object list data;

[0012] Multi-sensor hybrid inspection point identification module: reads the information of all inspection points in the multi-sensor inspection point library module, combines multiple sensors to identify the inspection points, and notifies the 5G upload module of the check-in information;

[0013] Beidou / GPS positioning check-in module: Receives radio signals with time and location information continuously sent by Beidou / GPS satellites in the air, obtains the positioning information and sends it to the multi-sensor hybrid inspection point identification module;

[0014] 5G base station positioning punch-in module: receives 5G base station positioning signals to obtain positioning information, and sends the positioning data to the multi-sensor hybrid inspection point identification module;

[0015] Tag punching module: integrated with RFID tag reader, the tag punching module reads the RFID tag ID value and sends the tag ID value to the multi-sensor hybrid inspection point identification module;

[0016] 5G upload module: Receives and caches the clock-in information. After checking that the 5G network connection is successful, it transmits the clock-in information of the inspection point to the inspection chain storage module.

[0017] Inspection chain storage module: Receives the inspection point punch-in information from the 5G upload module, generates an inspection chain according to the inspector ID, and stores the punch-in information in the inspection chain in the order of the punch-in time. The inspector's historical records can be retrieved according to the inspector ID and opening time.

[0018] The present invention also includes a 5G-based multi-sensor inspection method, which uses the above-mentioned 5G-based multi-sensor inspection system and includes the following steps:

[0019] S1: The dual-camera video clocking module uses two cameras to sample video, sends the inspection point ID and the obtained image object list to the video label calibration module for video calibration, and sends the image object list to the multi-sensor hybrid inspection point recognition module for video clocking;

[0020] S2: The audio clocking module collects surrounding sounds, generates PCM audio data for data slicing, sends the obtained frequency domain data slicing to the audio tag calibration module for audio calibration, and sends the frequency domain data slicing to the multi-sensor hybrid inspection point identification module for audio clocking;

[0021] S3: The Beidou / GPS positioning punch-in module receives the radio signals with time and location information continuously sent by Beidou / GPS satellites in the air, obtains the positioning information, and sends the positioning information to the multi-sensor hybrid inspection point identification module;

[0022] S4: The 5G base station positioning punch-in module receives the 5G base station positioning signal to obtain positioning information and sends the positioning data to the multi-sensor hybrid inspection point identification module;

[0023] S5: The tag punching module integrates an RFID tag reader to read the RFID tag ID value. The tag punching module sends the tag ID value to the multi-sensor hybrid inspection point identification module;

[0024] S6: The video label calibration module receives the image object list sent by the dual-camera video punch-in module, obtains the image object list data of all inspection points from the multi-sensor inspection point library module, performs calibration, and then sends the image object list and inspection point ID to the multi-sensor inspection point library module;

[0025] S7: Audio tag calibration module: Receives the frequency domain data slices sent by the audio punch module, obtains the frequency domain data slices of all inspection points from the multi-sensor inspection point library module, calibrates them, and then sends the frequency domain data slices and inspection point IDs to the multi-sensor inspection point library module;

[0026] S8: The multi-sensor inspection point library module stores and queries the inspection point information, which includes the inspection point ID, tag ID, location coordinates, frequency domain data slices, and image object list data;

[0027] S9: Multi-sensor hybrid inspection point identification module: reads the information of all inspection points in the multi-sensor inspection point library module, combines multiple sensors to identify inspection points for clocking in, and notifies the 5G upload module of the clocking in information;

[0028] S10: The 5G upload module receives the clock-in information from the multi-sensor hybrid inspection point identification module and caches the clock-in information. After checking that the 5G network connection is successful, the 5G upload module transmits the inspection point clock-in information to the inspection chain storage module.

[0029] S11: The inspection chain storage module receives the inspection point punch-in information from the 5G upload module. The inspection chain storage module generates an inspection chain according to the inspector ID. The punch-in information is stored in the inspection chain in the order of punch-in time. The punch-in information includes the punch-in inspection point ID. The punch-in information includes: tag ID, location coordinates, frequency domain data fragmentation, image object list data, and the inspection chain storage module can retrieve the inspector's historical records according to the inspector ID and opening time.

[0030] The step S1 comprises the following steps:

[0031] 1.1 The dual-camera video punch-in module uses two cameras to sample videos and generate YUV video frames of the two cameras;

[0032] 1.2 The dual-camera video check-in module uses the YOLO target detection algorithm to identify objects in the YUV video frames of the two cameras and generates a list of image objects captured by the two cameras. The objects in the image object list include type, size and coordinates;

[0033] 1.3 The dual-camera video check-in module uses the Harris corner detection method to extract the feature points of the Y component of the YUV video frames of the two cameras;

[0034] 1.4 The dual-camera video check-in module obtains the corner point coordinates and the type, size, and coordinates of the object on the YUV video frames of the two cameras, and determines the corner points of the object on the YUV video frames of the two cameras based on the object size, coordinates, and corner point coordinates;

[0035] 1.5 The dual-camera video check-in module uses the sequential similarity detection (SSDA) fusion algorithm to calculate the same feature point for the same type of objects in the YUV video frames of the two cameras, with the corner point as the center. The objects in the YUV video frames of the two cameras are considered to be the same object, and the parallax distance calculation method is used to calculate the distance between the cameras of the object, and then merge them into an image object list with distance information;

[0036] 1.6 The dual-camera video check-in module sorts the image object list with distance information from near to far, and calculates the relative distance of each object. The relative distance of each object is the object's distance minus the distance of the first object in the image object list. The resulting image object list contains the relative distances of the objects and is sorted from near to far. The objects in the image object list include the object type and the relative distance of the object.

[0037] 1.7 The dual-camera video punch-in module performs calibration operations. The dual-camera video punch-in module sends the inspection point ID and the relative distance of the object, sorted from near to far, to the video label calibration module. The inspection point ID is manually entered by the user.

[0038] 1.8 The dual-camera video punch-in module performs inspection punch-in operations. The dual-camera video punch-in module sends a list of image objects with relative distances to objects and sorted from near to far to the multi-sensor hybrid inspection point recognition module.

[0039] The step S2 comprises the following steps:

[0040] 2.1 The audio clock-in module collects surrounding sounds and generates PCM audio data;

[0041] 2.2 The audio punch-in module slices the PCM audio data into 4 millisecond time domain data slices;

[0042] 2.3 The audio clock-in module calculates the 4-millisecond time domain data fragmentation and adds the absolute value of each sampling point to obtain the fragmentation capability value. If the fragmentation capability value is greater than the set threshold, the next step is entered;

[0043] 2.4 The audio punch-in module performs FFT Fourier transform to convert the time domain data slices into frequency domain data slices. The frequency domain data slices are the energy values of each frequency in the time domain slices.

[0044] 2.5 The audio tag calibration module sorts the frequency domain data in descending order of energy value and standardizes it according to the energy value. The standard value of the first frequency is 100, and the standard values of other frequencies are 100*energy value / energy value of the first frequency.

[0045] 2.6 The audio punch-in module obtains standardized frequency domain data slices in descending order of energy value;

[0046] 2.7 The audio punch-in module performs calibration operations and sends the standardized frequency data slices in descending order of energy value to the audio tag calibration module;

[0047] 2.8 The audio clock-in module performs the inspection clock-in operation and sends the standardized frequency domain data slices in descending order of energy value to the multi-sensor hybrid inspection point identification module.

[0048] The step S3 comprises the following steps:

[0049] 3.1 The BeiDou / GPS positioning check-in module receives radio signals continuously transmitted by BeiDou / GPS satellites in the air, which contain time and location information. The satellites and the BeiDou / GPS positioning check-in module simultaneously generate the same pseudo-random code. When the two codes are synchronized, the time delay is measured and multiplied by the speed of light to obtain the pseudo-range.

[0050] 3.2 The BeiDou / GPS positioning punch-in module synchronizes the signals of four satellites and finally obtains the three-dimensional position of the BeiDou / GPS positioning punch-in module based on pseudo-range measurement. The BeiDou / GPS positioning punch-in module sends the positioning information to the multi-sensor hybrid inspection point identification module;

[0051] 3.3 When the Beidou / GPS positioning clock-in module cannot synchronize to the signals of the four satellites, the multi-sensor hybrid inspection point identification module will be fed back that the Beidou / GPS positioning has failed.

[0052] The step S4 comprises the following steps:

[0053] 4.15G base station positioning punch-in module receives 5G base station positioning signal for positioning;

[0054] The 4.25G base station transmits a signal to the 5G base station positioning clocking-in module. The 5G base station positioning clocking-in module transmits the signal back. The 5G base station calculates the distance from the base station to the mobile phone based on the round-trip time. It then locates the location of the 5G base station positioning clocking-in module using the angle of arrival and angle of departure. The 5G base station sends the positioning information to the 5G base station positioning clocking-in module, which then sends the positioning data to the multi-sensor hybrid inspection point identification module.

[0055] 4.3 When the 5G base station positioning clock-in module cannot communicate with any 5G base station, the 5G base station positioning clock-in module fails to clock in and notifies the multi-sensor hybrid inspection point identification module.

[0056] The step S6 comprises the following steps:

[0057] 6.1 The video label calibration module receives an image object list with relative distances to objects and sorted from near to far, along with the inspection point ID. The object list includes object types and relative distances to objects.

[0058] 6.2 The video label calibration module obtains the image object list data of all inspection points from the multi-sensor inspection point library module;

[0059] 6.3 The video label calibration module compares the image object list with the image object list of all inspection points from near to far according to the object type and relative distance of the object. If the relative position objects of the two image object lists are different, the module stops. If the same ratio exceeds 80%, the image object list is considered to match the inspection point, and the video label calibration module fails.

[0060] 6.4 If the video label calibration module and the inspection point image object list have the same ratio of less than 80%, the video label calibration is successful;

[0061] 6.5 The video label calibration module sends the image object list with the relative distance of the objects and sorted from near to far and the inspection point ID to the multi-sensor inspection point library module.

[0062] The step S7 comprises the following steps:

[0063] 7.1 The audio tag calibration module receives the standardized frequency domain data fragments and inspection point IDs from the audio punch-in module in descending order of energy value;

[0064] 7.2 The audio tag calibration module obtains the frequency domain data fragments of all inspection points from the multi-sensor inspection point library module;

[0065] 7.3 The audio tag calibration module compares the frequencies of the frequency domain data slices of the inspection points in the multi-sensor inspection point library module in descending order of capability value. If the frequencies are different, the calibration stops. If the matching ratio exceeds 80%, the frequency domain data slice is considered to match the inspection point, and the audio tag calibration module fails.

[0066] 7.4 The audio tag calibration module has the same frequency domain data fragments as all inspection points, and the calibration is successful if the ratio does not exceed 80%. The audio tag calibration module sends the standardized frequency domain data fragments and inspection point IDs in the order of energy value from large to small to the multi-sensor inspection point library module.

[0067] The step S8 comprises the following steps:

[0068] 8.1 The multi-sensor inspection point library module receives the image object list with relative distances to objects and sorted from near to far and the inspection point ID from the video label calibration module. The multi-sensor inspection point library module saves the image object list data to the inspection point according to the inspection point ID.

[0069] 8.2 The multi-sensor inspection point library module receives the energy values of the audio tag calibration module in descending order and in a standardized manner, and the inspection point ID. The multi-sensor inspection point library module saves the frequency domain data slices to the inspection point according to the inspection point ID;

[0070] The step S9 comprises the following steps:

[0071] 9.1 The multi-sensor hybrid inspection point identification module reads the information of all inspection points in the multi-sensor inspection point library module. The inspection point information includes inspection point ID, tag ID, location coordinates, frequency domain data slices, and image object list data;

[0072] 9.2 The multi-sensor hybrid inspection point identification module uses each sensor to identify the clock-in information according to the preset priority;

[0073] 9.3 The multi-sensor hybrid inspection point identification module performs tag punching. The multi-sensor hybrid inspection point identification module receives the tag ID from the tag punching module, matches the tag ID of the inspection point according to the received tag ID, and determines that the inspection point ID of the inspection point is the punching inspection point ID. After the multi-sensor hybrid inspection point identification module successfully punches in the tag, it determines whether there are other sensors at the inspection point and selects other sensors for punching in according to priority. Otherwise, it considers the identification successful and notifies the 5G upload module of the punching information.

[0074] 9.4 The multi-sensor hybrid inspection point identification module notifies the Beidou / GPS positioning punching module to perform location punching or receives the location of the Beidou / GPS positioning punching module based on the priority. If the multi-sensor hybrid inspection point identification module receives the location positioning of the Beidou / GPS positioning punching module successfully, it will determine the location punching. If the multi-sensor hybrid inspection point identification module fails to receive the location positioning of the Beidou / GPS positioning punching module, it will notify the 5G base station positioning punching module to perform location punching;

[0075] 9.5 The multi-sensor hybrid inspection point identification module receives the position of the 5G base station positioning punch-in module according to the priority notification. If the positioning is successful, it determines the position punch-in. The multi-sensor hybrid inspection point identification module receives the position of the 5G base station positioning punch-in module. If the positioning fails, it is considered that the punch-in failed. The process of judging the position punch-in: the multi-sensor hybrid inspection point identification module determines whether the punch-in inspection point ID has been confirmed. If the punch-in inspection point ID has been confirmed, the distance calculation is performed with the position coordinates of the inspection point with the punch-in inspection point ID. If the distance difference is less than the threshold value, the position punch-in is considered successful. If the punch-in inspection point ID is not confirmed, the distance calculation is performed with the position coordinates of all inspection points. If the distance difference is less than the threshold value, the position punch-in is considered successful, and the inspection point ID is the punch-in inspection point ID;

[0076] 9.6 After the multi-sensor hybrid inspection point identification module successfully clocks in, it determines if there are other sensors at the inspection point and selects other sensors for clocking in according to priority. Otherwise, it considers the identification successful and notifies the 5G upload module of the clocking in information.

[0077] 9.7 The multi-sensor hybrid inspection point identification module notifies the dual-camera video clocking module to perform video clocking in or receives video clocking in from the dual-camera video clocking module based on priority;

[0078] 9.8 The multi-sensor hybrid inspection point identification module receives the image object list with the relative distance of the objects and sorted from near to far from the dual-camera video punch-in module. The multi-sensor hybrid inspection point identification module determines whether the punch-in inspection point ID has been confirmed;

[0079] 9.9 If the ID of the checkpoint has been confirmed, the multi-sensor hybrid checkpoint identification module compares the image object list from near to far according to the object type and the relative distance of the object with the image object list of the checkpoint corresponding to the checkpoint ID. If the relative position objects of the two image object lists are different, the module stops. If the same ratio exceeds 80%, the video check-in is considered successful. Otherwise, the check-in is considered unsuccessful.

[0080] 9.10 If the Multi-sensor Hybrid Inspection Point Identification Module fails to confirm the punch-in inspection point ID, the Multi-sensor Hybrid Inspection Point Identification Module compares the image object list with the image object list of all inspection points from near to far, based on object type and relative distance. If the relative position objects in the two image object lists are different, the module stops. If the same ratio exceeds 80%, the image object list is considered to match the inspection point, and the inspection ID of the inspection point is the punch-in inspection ID. If the same ratio of the Multi-sensor Hybrid Inspection Point Identification Module is less than 80%, the video punch-in fails.

[0081] 9.11 After the multi-sensor hybrid inspection point identification module successfully clocks in, it determines if there are other sensors at the inspection point and selects other sensors for clocking in according to priority. Otherwise, it considers the identification successful and notifies the 5G upload module of the clocking in information.

[0082] 9.12 The multi-sensor hybrid inspection point identification module notifies the audio punch module to perform audio punch or receive audio punch according to the priority. The multi-sensor hybrid inspection point identification module receives the standardized frequency domain data slices in descending order of energy value from the audio punch module;

[0083] 9.13 The multi-sensor hybrid inspection point identification module determines whether the punch-in inspection point ID has been confirmed. If the punch-in inspection point ID has been confirmed, the multi-sensor hybrid inspection point identification module compares the frequency domain data of the inspection point corresponding to the punch-in inspection point ID in descending order of capability value. If the frequencies are different, the module stops. If the frequency is the same for more than 80%, the audio punch-in is considered successful. Otherwise, the punch-in fails.

[0084] 9.14 If the multi-sensor hybrid inspection point identification module fails to confirm the punch-in inspection point ID, it will compare the frequency data of all inspection points in descending order of capability value. If the frequencies are different, it will stop. If the matching ratio exceeds 80%, it is considered a match with the inspection point, the audio punch-in is successful, and the inspection point ID of the inspection point is used as the punch-in inspection point ID. If the matching ratio is less than 80%, the punch-in is considered unsuccessful.

[0085] 9.15 After the Multi-Sensor Hybrid Inspection Point Identification Module successfully clocks in via audio, it notifies the 5G Upload Module of the clock-in information. This information must include the inspection point ID, inspector ID, and clock-in time. It may also include the tag ID, location coordinates, frequency domain data fragments, and image object list data. The inspector ID is pre-entered by the Multi-Sensor Hybrid Inspection Point Identification Module.

[0086] The invention provides a 5G-based multi-sensor inspection system and method with the following beneficial effects:

[0087] 1. The present invention fills the defects of existing tags and Beidou / GPS inspection and punching, the tags need to be set in advance, and the Beidou / GPS no signal problem, and uses video object depth information and sound frequency characteristics for inspection and punching.

[0088] 2. The present invention uses multiple sensors to increase the credibility of inspections and avoid the problem of easy fraud in inspections with a single sensor.

[0089] 3. 5G base station large-scale antenna technology has higher-resolution beams and can also achieve higher-precision ranging and angle measurement characteristics. BRIEF DESCRIPTION OF THE DRAWINGS

[0090] In order to more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the drawings required for use in the specific embodiments or the description of the prior art. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0091] Figure 1 This is a flow chart of a 5G-based multi-sensor inspection system and method used in the present invention. DETAILED DESCRIPTION

[0092] The technical solution of the present invention is further described below with reference to the accompanying drawings and embodiments.

[0093] like Figure 1 As shown, a 5G-based multi-sensor inspection system includes: a dual-camera video clocking module 1, an audio clocking module 2, a video tag calibration module 3, an audio tag calibration module 4, a multi-sensor inspection point library module 5, a multi-sensor hybrid inspection point identification module 6, a Beidou / GPS positioning clocking module 7, a 5G base station positioning clocking module 8, a tag clocking module 9, a 5G upload module 10, and an inspection chain storage module 11.

[0094] Dual-camera video punch-in module 1: Dual-camera video punch-in module 1 implements video calibration and video punch-in. Dual-camera video punch-in module 1 uses two cameras to sample videos and generates YUV video frames of the two cameras. The YUV video frames of the two cameras are separately identified using the YOLO target detection algorithm to obtain a list of image objects captured by the two cameras. The objects in the image object list contain type, size and coordinates. Dual-camera video punch-in module 1 uses the Harris corner detection method to extract the feature points of the Y component of the YUV video frames of the two cameras, and obtains the coordinates of the corner points and the type, size and coordinates of the objects on the YUV video frames of the two cameras. Dual-camera video punch-in module 1 determines the corner points of the objects on the YUV video frames of the two cameras based on the object size, coordinates and corner coordinates, and compares the corner points of the same type of objects in the YUV video frames of the two cameras respectively. The SSDA fusion algorithm is used to calculate and confirm the identity of the feature points. The objects in the YUV video frames of the two cameras are identified as the same object. The disparity distance calculation method is used to calculate the camera distance of the object, and the distance information is merged into a list of image objects with distance information. The dual-camera video check-in module 1 sorts the list of image objects with distance information from near to far, and calculates the relative distance of each object. The relative distance is the object's distance minus the distance of the first object in the image object list. This results in a list of image objects with relative distances, sorted from near to far. The objects in the image object list contain object types and relative distances. The dual-camera video check-in module 1 performs calibration operations and sends the inspection point ID and the list of image objects with relative distances, sorted from near to far, to the video tag calibration module 3. The inspection point ID is manually entered by the user. The dual-camera video check-in module 1 performs the inspection check-in operation and sends the list of image objects with relative distances, sorted from near to far, to the multi-sensor hybrid inspection point recognition module 6.

[0095] Audio Punch-in Module 2: Audio Punch-in Module 2 collects surrounding sounds and generates PCM audio data. Audio Punch-in Module 2 slices the PCM audio data to obtain 4-millisecond time-domain data slices. Audio Punch-in Module 2 performs an FFT Fourier transform to convert the time-domain data slices into frequency-domain data slices. The frequency-domain data slices are the energy values of each frequency in the time-domain slices. Audio Tag Calibration Module 4 sorts the frequency-domain data in descending order of energy value. Audio Tag Calibration Module 4 standardizes the energy values. The first frequency has a standard value of 100, and the other frequencies have a standard value of 100*energy value / energy value of the first frequency. Audio Punch-in Module 2 obtains the frequency-domain data slices, which are standardized in descending order of energy value. Audio Punch-in Module 2 performs the calibration operation and sends the standardized frequency-domain data slices, which are standardized in descending order of energy value, to Audio Tag Calibration Module 4. The audio punching module 2 performs the inspection punching operation. The audio punching module 2 sends the standardized frequency domain data slices in the order of energy value from large to small to the multi-sensor hybrid inspection point identification module 6.

[0096] Video label calibration module 3: The video label calibration module 3 receives a list of image objects with relative distances and sorted from near to far, and a patrol point ID, including object type and relative distance. The video label calibration module 3 obtains the image object list data of all patrol points from the multi-sensor patrol point library module 5, and compares the image object list with the image object list of all patrol points according to object type and relative distance from near to far. If the relative position objects of the two image object lists are different, the module stops. If the same ratio exceeds 80%, the image object list is considered to match the patrol point, and the video label calibration module 3 fails. If the same ratio between the video label calibration module 3 and the image object list of the patrol point is less than 80%, the video label calibration is successful. The video label calibration module 3 sends the image object list with relative distances and sorted from near to far, and the patrol point ID to the multi-sensor patrol point library module 5.

[0097] Audio tag calibration module 4: The audio tag calibration module 4 receives the frequency domain data slices and patrol point IDs that are standardized in descending order of energy values from the audio punch-in module 2. The audio tag calibration module 4 obtains the frequency domain data slices of all patrol points from the multi-sensor patrol point library module 5. The audio tag calibration module 4 compares the frequencies with the frequency domain data slices of the patrol points of the multi-sensor patrol point library module 5 in descending order of energy values. If the frequencies are different, the module stops. If the same ratio exceeds 80%, it is considered that the image object list matches the patrol point and the calibration of the audio tag calibration module 4 fails. If the same ratio of the audio tag calibration module 4 with the frequency domain data slices of all patrol points is less than 80%, the audio tag calibration module 4 is successfully calibrated. The audio tag calibration module 4 sends the frequency domain data slices and patrol point IDs that are standardized in descending order of energy values to the multi-sensor patrol point library module 5.

[0098] Multi-sensor inspection point library module 5: The multi-sensor inspection point library module 5 stores inspection point information. Inspection point information includes the inspection point ID, tag ID, location coordinates, frequency domain data slices, and image object list data. The multi-sensor inspection point library module 5 receives the image object list with relative distances and sorted from near to far, along with the inspection point ID, from the video tag calibration module 3. The multi-sensor inspection point library module 5 stores the image object list data at the inspection point according to the inspection point ID. The multi-sensor inspection point library module 5 receives the frequency domain data slices with energy values sorted from large to small and standardized, along with the inspection point ID, from the audio tag calibration module 4. The multi-sensor inspection point library module 5 stores the frequency domain data slices at the inspection point according to the inspection point ID. The multi-sensor inspection point library module 5 receives inspection information queries from the video tag calibration module 3, the audio tag calibration module 4, and the multi-sensor hybrid inspection point identification module 6.

[0099] Multi-sensor Hybrid Inspection Point Identification Module 6: This module uses multiple sensors to identify inspection points. It reads all inspection point information from the Multi-sensor Inspection Point Library Module 5. This information includes the inspection point ID, tag ID, location coordinates, frequency domain data fragments, and image object list data.

[0100] The multi-sensor hybrid inspection point identification module 6 uses various sensors to identify check-in information according to preset priorities, such as tag check-in, Beidou / GPS location check-in, 5G base station positioning check-in, video check-in, and audio check-in. The multi-sensor hybrid inspection point identification module 6 performs tag check-in. The multi-sensor hybrid inspection point identification module 6 receives the tag ID from the tag check-in module 9, matches the tag ID of the inspection point based on the received tag ID, and determines that the inspection point ID of the inspection point is the check-in inspection point ID. After the multi-sensor hybrid inspection point identification module 6 successfully checks in with the tag, it determines if other sensors are available at the inspection point and selects other sensors for check-in based on priority. Otherwise, it deems the identification successful and notifies the 5G upload module 10 of the check-in information. The multi-sensor hybrid inspection point identification module 6 notifies the Beidou / GPS positioning check-in module 7 to perform location check-in or receive the location of the Beidou / GPS positioning check-in module 7 based on priority. The multi-sensor hybrid inspection point identification module 6 successfully receives the location of the Beidou / GPS positioning check-in module 7 and determines the location check-in. If the multi-sensor hybrid inspection point identification module 6 fails to receive the positioning from the Beidou / GPS positioning punch-in module 7, it notifies the 5G base station positioning punch-in module 8 to perform the location punch-in. If the multi-sensor hybrid inspection point identification module 6 successfully receives the location positioning from the 5G base station positioning punch-in module 8, it determines the location punch-in. If the multi-sensor hybrid inspection point identification module 6 fails to receive the location positioning from the 5G base station positioning punch-in module 8, it is considered that the punch-in failed. Location punch-in determination process: The multi-sensor hybrid inspection point identification module 6 determines whether the punch-in inspection point ID has been confirmed. If the punch-in inspection point ID has been confirmed, it calculates the distance with the location coordinates of the inspection point with the punch-in inspection point ID. If the distance difference is less than the threshold value, the location punch-in is considered successful. If the punch-in inspection point ID has not been confirmed, it calculates the distance with the location coordinates of all inspection points. If the distance difference is less than the threshold value, the location punch-in is considered successful, and the inspection point ID is the punch-in inspection point ID. After successfully clocking in at a location, the multi-sensor hybrid inspection point identification module 6 determines if other sensors are available at that inspection point and selects them for clocking in based on priority. Otherwise, the identification is considered successful and the clocking-in information is notified to the 5G upload module 10. Based on the priority, the multi-sensor hybrid inspection point identification module 6 notifies the dual-camera video clocking module 1 to perform a video clocking-in or receives a video clocking-in from the dual-camera video clocking module 1. The multi-sensor hybrid inspection point identification module 6 receives a list of image objects with relative distances, sorted from near to far, from the dual-camera video clocking module 1. The multi-sensor hybrid inspection point identification module 6 determines whether the clocking-in inspection point ID has been confirmed. If the clocking-in inspection point ID has been confirmed, the multi-sensor hybrid inspection point identification module 6 compares the object type in the image object list with the image object list of the inspection point corresponding to the clocking-in inspection point ID, sorted by relative distance, from near to far. If the relative position objects in the two image object lists differ, the module stops. If the matching ratio exceeds 80%, the video clocking-in is considered successful. Otherwise, the clocking-in is considered unsuccessful.If the multi-sensor hybrid inspection point identification module 6 fails to confirm the check-in point ID, it compares the image object list with the image object lists of all inspection points, from nearest to farthest, by object type and relative distance. If the relative position objects in the two image object lists differ, the process stops. If the matching ratio exceeds 80%, the image object list is considered a match with the inspection point, and the inspection ID of the inspection point is used as the check-in ID. If the matching ratio is less than 80%, the video check-in fails. After the multi-sensor hybrid inspection point identification module 6 successfully checks the location, it determines if other sensors are available at the inspection point and selects them for check-in based on priority. Otherwise, the identification is considered successful and the check-in information is notified to the 5G upload module 10. Based on the priority, the multi-sensor hybrid inspection point identification module 6 notifies the audio check-in module 2 to perform an audio check-in or receive the audio check-in information. The multi-sensor hybrid inspection point identification module 6 receives the frequency domain data from the audio check-in module 2 in descending order of energy value and in a standardized manner. The multi-sensor hybrid inspection point identification module 6 determines whether the punch-in inspection point ID has been confirmed. If the punch-in inspection point ID has been confirmed, the multi-sensor hybrid inspection point identification module 6 compares the frequencies with the frequency domain data slices of the inspection point corresponding to the punch-in inspection point ID in descending order of capability value. If the frequencies are not the same, the module stops. If the same ratio exceeds 80%, the audio punch-in is considered successful; otherwise, the punch-in fails. If the multi-sensor hybrid inspection point identification module 6 has not confirmed the punch-in inspection point ID, the module compares the frequencies with the frequency domain data slices of all inspection points in descending order of capability value. If the frequencies are not the same, the module stops. If the same ratio exceeds 80%, the module considers that the checkpoint matches the checkpoint, the audio punch-in is successful, and the inspection point ID of the checkpoint is the punch-in inspection point ID. If the same ratio is less than 80%, the punch-in is considered unsuccessful. After the multi-sensor hybrid inspection point identification module 6 successfully clocks in via audio, it notifies the 5G upload module 10 of the clock-in information. The clock-in information must include the inspection point ID, the inspector ID, and the clock-in time. The clock-in information may also include the tag ID, location coordinates, frequency domain data fragments, and image object list data. The inspector ID is pre-entered by the multi-sensor hybrid inspection point identification module 6.

[0101] Beidou / GPS Positioning Module 7: Beidou / GPS Positioning Module 7 receives radio signals continuously transmitted by Beidou / GPS satellites in the air, which contain time and location information. The satellites and Beidou / GPS Positioning Module 7 simultaneously generate the same pseudo-random code. Once the two codes are time-synchronized, Beidou / GPS Positioning Module 7 can measure the time delay. Multiplying the time delay by the speed of light yields the pseudo-range. Beidou / GPS Positioning Module 7 synchronizes the signals of four satellites and ultimately determines its three-dimensional position based on the pseudo-range measurement. Beidou / GPS Positioning Module 7 then transmits this positioning information to the Multi-Sensor Hybrid Inspection Point Identification Module 6. If Beidou / GPS Positioning Module 7 cannot synchronize to the signals of the four satellites, it reports a Beidou / GPS positioning failure to the Multi-Sensor Hybrid Inspection Point Identification Module 6.

[0102] 5G base station positioning clock-in module 8: The 5G base station positioning clock-in module 8 receives the 5G base station positioning signal for positioning. The 5G base station large-scale antenna technology has a higher-resolution beam and can also achieve higher-precision ranging and angle measurement characteristics. As long as there is a 5G base station transmitting a signal to the 5G base station positioning clock-in module 8, the 5G base station positioning clock-in module 8 will immediately transmit the signal back. The 5G base station calculates the distance from the base station to the mobile phone based on the round-trip time, and then locates the location of the 5G base station positioning clock-in module 8 through the arrival angle and departure angle. The 5G base station sends the positioning information to the 5G base station positioning clock-in module 8. The 5G base station positioning clock-in module 8 sends the positioning data to the multi-sensor hybrid inspection point identification module 6. If the 5G base station positioning clock-in module 8 cannot communicate with any 5G base station, it is considered that the 5G base station positioning clock-in module 8 has failed to clock in, and the multi-sensor hybrid inspection point identification module 6 is notified.

[0103] Tag punching module 9: The tag punching module 9 integrates an RFID tag reader, the tag punching module 9 reads the RFID tag ID value, and the tag punching module 9 sends the tag ID value to the multi-sensor hybrid inspection point identification module 6.

[0104] 5G upload module 10: The 5G upload module 10 receives the check-in information from the multi-sensor hybrid inspection point identification module 6 and caches the check-in information. After checking that the 5G network connection is successful, the 5G upload module 10 transmits the inspection point check-in information to the inspection chain storage module 11.

[0105] Inspection Chain Storage Module 11: This module receives inspection point clock-in information from 5G Upload Module 10 and generates an inspection chain based on inspector ID. The clock-in information is stored in the inspection chain in chronological order. The clock-in information includes the inspection point ID and may include: tag ID, location coordinates, frequency domain data fragments, and image object list data. The inspection chain storage module 11 can retrieve the inspector's history records based on the inspector ID and access time.

[0106] The present invention also includes a 5G-based multi-sensor inspection method, which uses the above-mentioned 5G-based multi-sensor inspection system and includes the following steps:

[0107] S1: The dual-camera video clocking module 1 uses two cameras to sample video, sends the inspection point ID and the obtained image object list to the video label calibration module 3 for video calibration, and sends the image object list to the multi-sensor hybrid inspection point recognition module 6 for video clocking;

[0108] 1.1 Dual-camera video punch-in module 1 uses two cameras to sample videos and generate YUV video frames of the two cameras;

[0109] 1.2 Dual-camera video check-in module 1: Use the YOLO target detection algorithm to identify objects in the YUV video frames of the two cameras and generate a list of image objects captured by the two cameras. The objects in the image object list include type, size and coordinates;

[0110] 1.3 Dual-camera video check-in module 1 uses Harris corner detection method to extract feature points of the Y component of the YUV video frames of the two cameras;

[0111] 1.4 Dual-camera video check-in module 1 obtains the corner point coordinates and the type, size, and coordinates of the object on the YUV video frames of the two cameras, and determines the corner points of the object on the YUV video frames of the two cameras based on the object size, coordinates, and corner point coordinates;

[0112] 1.5 Dual-camera video check-in module 1 For the same type of objects in the YUV video frames of the two cameras, the sequential similarity detection (SSDA) fusion algorithm is used to calculate the same feature point with the corner point as the center. The objects in the YUV video frames of the two cameras are considered to be the same object. The parallax distance calculation method is used to calculate the distance between the cameras of the object and merge them into an image object list with distance information;

[0113] 1.6 Dual-Camera Video Check-in Module 1 sorts the image object list with distance information from near to far according to the distance of the object, and calculates the relative distance of each object. The relative distance of the object is the object's distance minus the distance of the first object in the image object list. Finally, a list of image objects with relative distances and sorted from near to far is obtained. The objects in the image object list include the object type and the relative distance of the object;

[0114] 1.7 The dual-camera video punching module 1 performs a calibration operation. The dual-camera video punching module 1 sends a list of image objects with inspection point IDs and relative distances to objects in order from near to far to the video label calibration module 3, where the inspection point IDs are manually entered by the user.

[0115] 1.8 The dual-camera video punch-in module 1 performs the inspection punch-in operation. The dual-camera video punch-in module 1 sends a list of image objects with relative distances to objects and sorted from near to far to the multi-sensor hybrid inspection point recognition module 6.

[0116] S2: The audio clocking module 2 collects surrounding sounds, generates PCM audio data for data slicing, sends the obtained frequency domain data slicing to the audio tag calibration module 4 for audio calibration, and sends the frequency domain data slicing to the multi-sensor hybrid inspection point identification module 6 for audio clocking;

[0117] 2.1 Audio punch-in module 2 collects surrounding sounds and generates PCM audio data;

[0118] 2.2 Audio punch-in module 2 slices the PCM audio data into 4 millisecond time domain data slices;

[0119] 2.3 Audio clock-in module 2 calculates the 4-millisecond time domain data slices, adds the absolute values of each sampling point to obtain the slice capability value, and if the slice capability value is greater than the set threshold, the next step is entered;

[0120] 2.4 Audio punch-in module 2 performs FFT Fourier transform to convert the time domain data slices into frequency domain data slices. The frequency domain data slices are the energy values of each frequency in the time domain slices.

[0121] 2.5 Audio Tag Calibration Module 4 sorts the frequency domain data in descending order of energy value and standardizes them according to the energy value. The standard value of the first frequency is 100, and the standard values of other frequencies are 100*energy value / energy value of the first frequency.

[0122] 2.6 The audio punch-in module 2 obtains the frequency domain data slices in descending order of energy value and in a standardized manner;

[0123] 2.7 The audio punch-in module 2 performs a calibration operation and sends the standardized frequency domain data slices in descending order of energy value to the audio tag calibration module 4;

[0124] 2.8 The audio clocking module 2 performs the inspection clocking operation and sends the standardized frequency domain data slices in descending order of energy value to the multi-sensor hybrid inspection point identification module 6.

[0125] S3: Beidou / GPS positioning punch-in module 7 receives the radio signals with time and location information continuously sent by Beidou / GPS satellites in the air, obtains the positioning information and sends the positioning information to the multi-sensor hybrid inspection point identification module 6;

[0126] 3.1 The BeiDou / GPS positioning clock-in module 7 receives radio signals continuously transmitted by BeiDou / GPS satellites in the air, which contain time and location information. The satellites and the BeiDou / GPS positioning clock-in module 7 simultaneously generate the same pseudo-random code. When the two codes are synchronized, the time delay is measured and multiplied by the speed of light to obtain the pseudo-range.

[0127] 3.2 The BeiDou / GPS positioning punch-in module 7 synchronizes the signals of the four satellites and finally obtains the three-dimensional position of the BeiDou / GPS positioning punch-in module 7 based on the pseudo-range measurement. The BeiDou / GPS positioning punch-in module 7 sends the positioning information to the multi-sensor hybrid inspection point identification module 6;

[0128] 3.3 When the Beidou / GPS positioning clocking-in module 7 cannot synchronize to the signals of the four satellites, the multi-sensor hybrid inspection point identification module 6 will be fed back that the Beidou / GPS positioning has failed.

[0129] S4: The 5G base station positioning punch-in module 8 receives the 5G base station positioning signal to obtain positioning information, and sends the positioning data to the multi-sensor hybrid inspection point identification module 6;

[0130] 4.15G base station positioning punch-in module 8 receives 5G base station positioning signals for positioning. 5G base station large-scale antenna technology has higher resolution beams and can also achieve higher precision ranging and angle measurement characteristics;

[0131] 4. The 25G base station transmits a signal to the 5G base station positioning clocking-in module 8. The 5G base station positioning clocking-in module 8 transmits the signal back. The 5G base station calculates the distance from the base station to the mobile phone based on the round-trip time. Then, it locates the location of the 5G base station positioning clocking-in module 8 using the arrival angle and departure angle. The 5G base station sends the positioning information to the 5G base station positioning clocking-in module 8. The 5G base station positioning clocking-in module 8 sends the positioning data to the multi-sensor hybrid inspection point identification module 6.

[0132] 4.3 When the 5G base station positioning clocking-in module 8 cannot communicate with any 5G base station, the 5G base station positioning clocking-in module 8 fails to clock in and notifies the multi-sensor hybrid inspection point identification module 6.

[0133] S5: The tag punching module 9 integrates an RFID tag reader to read the RFID tag ID value, and the tag punching module 9 sends the tag ID value to the multi-sensor hybrid inspection point identification module 6;

[0134] S6: The video tag calibration module 3 receives the image object list sent by the dual-camera video punch-in module 1, obtains the image object list data of all inspection points from the multi-sensor inspection point library module 5 for calibration, and sends the image object list and inspection point ID to the multi-sensor inspection point library module 5 after successful calibration.

[0135] 6.1 Video Label Calibration Module 3 receives an image object list with relative distances to objects and sorted from near to far, and an inspection point ID. The object list includes object types and relative distances to objects.

[0136] 6.2 The video label calibration module 3 obtains the image object list data of all inspection points from the multi-sensor inspection point library module 5;

[0137] 6.3 Video label calibration module 3 compares the image object list with the image object list of all inspection points from near to far according to the object type and relative distance of the object. If the relative position objects of the two image object lists are different, the process stops. If the same ratio exceeds 80%, the image object list is considered to match the inspection point, and the video label calibration module 3 video label calibration fails;

[0138] 6.4 Video Label Calibration Module 3 If the same ratio of the image object list of the inspection point is less than 80%, the video label calibration is successful;

[0139] 6.5 The video tag calibration module 3 sends the image object list with the relative distance of the objects and sorted from near to far and the inspection point ID to the multi-sensor inspection point library module 5.

[0140] S7: Audio tag calibration module 4: Receives the frequency domain data slices sent by audio punch module 2, obtains the frequency domain data slices of all inspection points from multi-sensor inspection point library module 5 for calibration, and sends the frequency domain data slices and inspection point ID to multi-sensor inspection point library module 5 after successful calibration.

[0141] 7.1 The audio tag calibration module 4 receives the frequency domain data segments and inspection point IDs in descending order of energy values from the audio punch-in module 2;

[0142] 7.2 The audio tag calibration module 4 obtains the frequency domain data fragments of all inspection points from the multi-sensor inspection point library module 5;

[0143] 7.3 The audio tag calibration module 4 compares the frequencies of the frequency domain data slices of the inspection points in the multi-sensor inspection point library module 5 in descending order of capability value. If the frequencies are different, the calibration stops. If the matching ratio exceeds 80%, the frequency domain data slice is considered to match the inspection point, and the audio tag calibration module 4 fails the calibration.

[0144] 7.4 The audio tag calibration module 4 successfully calibrates the frequency domain data fragments of all inspection points in the same proportion but no more than 80%. The audio tag calibration module 4 sends the standardized frequency domain data fragments and inspection point IDs in the order of energy value from large to small to the multi-sensor inspection point library module 5.

[0145] S8: The multi-sensor inspection point library module 5 stores and queries the inspection point information, which includes the inspection point ID, tag ID, location coordinates, frequency domain data slices, and image object list data.

[0146] 8.1 The multi-sensor inspection point library module 5 receives the image object list with relative distances to objects and sorted from near to far and the inspection point ID from the video label calibration module 3. The multi-sensor inspection point library module 5 saves the image object list data at the inspection point according to the inspection point ID;

[0147] 8.2 The multi-sensor inspection point library module 5 receives the energy values of the audio tag calibration module 4 in descending order and in a standardized manner, and the inspection point ID, and stores the frequency domain data fragments of the inspection point according to the inspection point ID;

[0148] 8.3 The multi-sensor inspection point library module 5 receives the inspection information query from the video tag calibration module 3, the audio tag calibration module 4 and the multi-sensor hybrid inspection point identification module 6.

[0149] S9: Multi-sensor hybrid inspection point identification module 6: reads the information of all inspection points of the multi-sensor inspection point library module 5, combines multiple sensors to identify the inspection points, and notifies the 5G upload module 10 of the punch-in information;

[0150] 9.1 The multi-sensor hybrid inspection point identification module 6 reads the information of all inspection points of the multi-sensor inspection point library module 5. The inspection point information includes the inspection point ID, tag ID, location coordinates, frequency domain data slices, and image object list data;

[0151] 9.2 Multi-sensor hybrid inspection point identification module 6 uses each sensor to identify the clock-in information according to the preset priority;

[0152] 9.3 The multi-sensor hybrid inspection point identification module 6 performs tag punching. The multi-sensor hybrid inspection point identification module 6 receives the tag ID from the tag punching module 9, matches the tag ID of the inspection point according to the received tag ID, and determines that the inspection point ID of the inspection point is the punching inspection point ID. After the multi-sensor hybrid inspection point identification module 6 successfully punches in the tag, it determines whether there are other sensors at the inspection point, and selects other sensors for punching in according to priority. Otherwise, it considers the identification successful and notifies the 5G upload module 10 of the punching information.

[0153] 9.4 The multi-sensor hybrid inspection point identification module 6 notifies the Beidou / GPS positioning punching module 7 to perform location punching or receives the location of the Beidou / GPS positioning punching module 7 according to the priority. If the multi-sensor hybrid inspection point identification module 6 receives the location positioning of the Beidou / GPS positioning punching module 7 successfully, it will determine the location punching. If the multi-sensor hybrid inspection point identification module 6 fails to receive the location positioning of the Beidou / GPS positioning punching module 7, it will notify the 5G base station positioning punching module 8 to perform location punching;

[0154] 9.5 If the multi-sensor hybrid inspection point identification module 6 receives the position positioning from the 5G base station positioning punch-in module 8 successfully, it determines the position punch-in. If the multi-sensor hybrid inspection point identification module 6 fails to receive the position positioning from the 5G base station positioning punch-in module 8, it is considered that the punch-in failed. Process for determining the position punch-in: the multi-sensor hybrid inspection point identification module 6 determines whether the punch-in inspection point ID has been confirmed. If the punch-in inspection point ID has been confirmed, the distance calculation is performed with the position coordinates of the inspection point with the punch-in inspection point ID. If the distance difference is less than the threshold value, the position punch-in is considered successful. If the punch-in inspection point ID has not been confirmed, the distance calculation is performed with the position coordinates of all inspection points. If the distance difference is less than the threshold value, the position punch-in is considered successful, and the inspection point ID is the punch-in inspection point ID.

[0155] 9.6 After the multi-sensor hybrid inspection point identification module 6 successfully clocks in at the location, it determines whether there are other sensors at the inspection point and selects other sensors for clocking in according to priority. Otherwise, it considers the identification successful and notifies the 5G upload module 10 of the clocking in information;

[0156] 9.7 The multi-sensor hybrid inspection point identification module 6 notifies the dual-camera video clocking module 1 to clock in or receives the video clocking from the dual-camera video clocking module 1 according to the priority;

[0157] 9.8 The multi-sensor hybrid inspection point identification module 6 receives the image object list with relative distances to objects and sorted from near to far from the dual-camera video clocking module 1, and determines whether the clocking inspection point ID has been confirmed;

[0158] 9.9 If the ID of the checkpoint has been confirmed, the multi-sensor hybrid checkpoint identification module 6 compares the image object list with the image object list of the checkpoint corresponding to the checkpoint ID according to the object type and relative distance from the nearest to the farthest object. If the relative position objects of the two image object lists are different, the module stops. If the same ratio exceeds 80%, the video check-in is considered successful. Otherwise, the check-in is considered unsuccessful.

[0159] 9.10 If the Multi-sensor Hybrid Inspection Point Identification Module 6 fails to confirm the punch-in inspection point ID, it compares the image object list with the image object lists of all inspection points from near to far, based on object type and relative distance. If the relative position objects in the two image object lists are different, the module stops. If the matching ratio exceeds 80%, the image object list is considered to match the inspection point, and the inspection ID of the inspection point is the punch-in inspection ID. If the matching ratio is less than 80%, the video punch-in fails.

[0160] 9.11 After the multi-sensor hybrid inspection point identification module 6 successfully clocks in at the location, it determines if there are other sensors at the inspection point and selects other sensors for clocking in according to priority. Otherwise, it considers the identification successful and notifies the 5G upload module 10 of the clocking in information.

[0161] 9.12 The multi-sensor hybrid inspection point identification module 6 notifies the audio punch module 2 to perform audio punching or receive audio punching according to the priority. The multi-sensor hybrid inspection point identification module 6 receives the frequency domain data slices from the audio punch module 2 in descending order of energy value and in a standardized manner;

[0162] 9.13 The multi-sensor hybrid inspection point identification module 6 determines whether the punch-in inspection point ID has been confirmed. If the punch-in inspection point ID has been confirmed, the multi-sensor hybrid inspection point identification module 6 compares the frequency domain data slices of the inspection point corresponding to the punch-in inspection point ID in descending order of capability value. If the frequencies are different, the module stops. If the frequency is the same for more than 80%, the audio punch-in is considered successful. Otherwise, the punch-in fails.

[0163] 9.14 If the multi-sensor hybrid inspection point identification module 6 fails to confirm the punch-in inspection point ID, it will compare the frequencies with the frequency domain data of all inspection points in descending order of capability value. If the frequencies are different, it will stop. If the matching ratio exceeds 80%, it is considered a match with the inspection point, the audio punch-in is successful, and the inspection point ID of the inspection point is used as the punch-in inspection point ID. If the matching ratio is less than 80%, the punch-in is considered unsuccessful.

[0164] 9.15 After the multi-sensor hybrid inspection point identification module 6 successfully clocks in via audio, it notifies the 5G upload module 10 of the clock-in information. The clock-in information must include the inspection point ID, inspector ID, and clock-in time. The clock-in information may also include the tag ID, location coordinates, frequency domain data fragments, and image object list data. The inspector ID is pre-entered by the multi-sensor hybrid inspection point identification module 6.

[0165] S10: The 5G upload module 10 receives the check-in information from the multi-sensor hybrid inspection point identification module 6 and caches the check-in information. After checking that the 5G network connection is successful, the 5G upload module 10 transmits the inspection point check-in information to the inspection chain storage module 11.

[0166] S11: The inspection chain storage module 11 receives the inspection point check-in information from the 5G upload module 10. The inspection chain storage module 11 generates an inspection chain based on the inspector ID. The check-in information is stored in the inspection chain in chronological order. The check-in information includes the inspection point ID. The check-in information may include: tag ID, location coordinates, frequency domain data fragments, and image object list data. The inspection chain storage module 11 can retrieve the inspector's historical records based on the inspector ID and the opening time.

[0167] The specific embodiments described above further illustrate the technical problems, technical solutions and beneficial effects solved by the present invention. It should be understood that the above are only specific embodiments of the present invention and are not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the scope of protection of the present invention.

Claims

1. A 5G-based multi-sensor inspection system, characterized in that: include: Dual-camera video check-in module (1): uses two cameras to sample video, sends the inspection point ID and the obtained image object list to the video label calibration module (3) for video calibration, and sends the image object list to the multi-sensor hybrid inspection point recognition module (6) for video check-in; Audio clocking module (2): used to collect surrounding sounds, generate PCM audio data for data slicing, send the obtained frequency domain data slicing to audio tag calibration module (4) for audio calibration, and send the frequency domain data slicing to multi-sensor hybrid inspection point identification module (6) for audio clocking; The video tag calibration module (3) receives the image object list sent by the dual-camera video punch-in module (1), obtains the image object list data of all inspection points from the multi-sensor inspection point library module (5), calibrates them, and then sends the image object list and the inspection point ID to the multi-sensor inspection point library module (5); The audio tag calibration module (4) receives the frequency domain data slices sent by the audio punch module (2), obtains the frequency domain data slices of all inspection points from the multi-sensor inspection point library module (5), calibrates them, and then sends the frequency domain data slices and the inspection point ID to the multi-sensor inspection point library module (5); Multi-sensor inspection point library module (5): saves and queries the inspection point information, which includes the inspection point ID, tag ID, location coordinates, frequency domain data slices, and image object list data; Multi-sensor hybrid inspection point identification module (6): reads the information of all inspection points in the multi-sensor inspection point library module (5), combines multiple sensors to identify the inspection points, and notifies the 5G upload module (10) of the check-in information; Beidou / GPS positioning check-in module (7): receives radio signals continuously sent by Beidou / GPS satellites in the air with time and location information, obtains the positioning information and sends the positioning information to the multi-sensor hybrid inspection point identification module (6); 5G base station positioning punch-in module (8): receives the 5G base station positioning signal to obtain positioning information, and sends the positioning data to the multi-sensor hybrid inspection point identification module (6); Tag punching module (9): integrated with RFID tag reader, the tag punching module (9) reads the RFID tag ID value, and the tag punching module (9) sends the tag ID value to the multi-sensor hybrid inspection point identification module (6); 5G upload module (10): receives and caches the check-in information, and after checking that the 5G network connection is successful, transmits the check-in information of the inspection point to the inspection chain storage module (11); Inspection chain storage module (11): Receives the inspection point punch-in information from the 5G upload module (10), generates an inspection chain according to the inspector ID, and stores the punch-in information in the inspection chain in the order of the punch-in time. The inspector's historical record can be retrieved according to the inspector ID and the opening time.

2. A 5G-based multi-sensor inspection method, using a 5G-based multi-sensor inspection system according to claim 1, characterized in that: The method comprises the following steps: S1: The dual-camera video punch-in module (1) uses two cameras to sample video, sends the inspection point ID and the obtained image object list to the video label calibration module (3) for video calibration, and sends the image object list to the multi-sensor hybrid inspection point recognition module (6) for video punch-in; S2: The audio clocking module (2) collects surrounding sounds, generates PCM audio data for data slicing, sends the obtained frequency domain data slicing to the audio tag calibration module (4) for audio calibration, and sends the frequency domain data slicing to the multi-sensor hybrid inspection point identification module (6) for audio clocking; S3: Beidou / GPS positioning punch-in module (7) receives the radio signals with time and location information continuously sent by Beidou / GPS satellites in the air, obtains the positioning information and sends the positioning information to the multi-sensor hybrid inspection point identification module (6); S4: The 5G base station positioning punch-in module (8) receives the 5G base station positioning signal to obtain positioning information, and sends the positioning data to the multi-sensor hybrid inspection point identification module (6); S5: The tag punching module (9) integrates an RFID tag reader to read the RFID tag ID value, and the tag punching module (9) sends the tag ID value to the multi-sensor hybrid inspection point identification module (6); S6: The video tag calibration module (3) receives the image object list sent by the dual-camera video punch-in module (1), obtains the image object list data of all inspection points from the multi-sensor inspection point library module (5), performs calibration, and then sends the image object list and the inspection point ID to the multi-sensor inspection point library module (5); S7: Audio tag calibration module (4): Receives the frequency domain data slices sent by the audio punch module (2), obtains the frequency domain data slices of all inspection points from the multi-sensor inspection point library module (5), calibrates them, and then sends the frequency domain data slices and the inspection point ID to the multi-sensor inspection point library module (5); S8: The multi-sensor inspection point library module (5) stores and queries the inspection point information, which includes the inspection point ID, tag ID, location coordinates, frequency domain data slices, and image object list data; S9: The multi-sensor hybrid inspection point identification module (6) reads the information of all inspection points of the multi-sensor inspection point library module (5), combines the multi-sensor identification inspection points to punch in, and notifies the 5G upload module (10) of the punch-in information; S10: The 5G upload module (10) receives the check-in information from the multi-sensor hybrid inspection point identification module (6), caches the check-in information, and after checking that the 5G network connection is successful, the 5G upload module (10) transmits the inspection point check-in information to the inspection chain storage module (11); S11: The inspection chain storage module (11) receives the inspection point punch-in information from the 5G upload module (10), and the inspection chain storage module (11) generates an inspection chain according to the inspector ID. The punch-in information is stored in the inspection chain in the order of the punch-in time. The punch-in information includes the punch-in inspection point ID. The punch-in information includes: tag ID, location coordinates, frequency domain data fragmentation, image object list data, and the inspection chain storage module (11) can retrieve the inspector's historical record according to the inspector ID and the opening time.

3. The 5G-based multi-sensor inspection method according to claim 2, characterized in that: The step S1 comprises the following steps: 1.1 Dual-camera video punch-in module (1) uses two cameras to sample videos and generate YUV video frames of the two cameras; 1.2 Dual-camera video check-in module (1) Use the YOLO target detection algorithm to identify objects in the YUV video frames of the two cameras and generate a list of image objects captured by the two cameras. The objects in the image object list include type, size and coordinates; 1.3 Dual-camera video check-in module (1) Harris corner detection method is used to extract the feature points of the Y component of the YUV video frames of the two cameras; 1.4 Dual-camera video punch-in module (1) Obtain the coordinates of the corner points on the YUV video frames of the two cameras and the type, size and coordinates of the object. According to the object size, coordinates and corner point coordinates, obtain the corner points of the object on the YUV video frames of the two cameras; 1.5 Dual-camera video check-in module (1) The sequential similarity detection (SSDA) fusion algorithm is used to calculate the same feature point for the same type of objects in the YUV video frames of the two cameras, with the corner point as the center. The objects in the YUV video frames of the two cameras are considered to be the same object, and the distance between the cameras of the object is calculated using the parallax distance calculation method, and then merged into a list of image objects with distance information; 1.6 Dual-camera video check-in module (1) Sort the image object list with distance information according to the distance of the object from near to far, calculate the relative distance of each object, and the relative distance of the object is the distance of the object minus the distance of the first object in the image object list, and finally obtain the image object list with the relative distance of the object and sorted from near to far. The objects in the image object list include the object type and the relative distance of the object; 1.7 The dual-camera video punching module (1) performs a calibration operation. The dual-camera video punching module (1) sends a list of image objects with inspection point IDs and relative distances to objects in order from near to far to the video label calibration module (3), wherein the inspection point IDs are manually entered by the user; 1.8 The dual-camera video punching module (1) performs the inspection punching operation. The dual-camera video punching module (1) sends the image object list with the relative distance of the object and sorted from near to far to the multi-sensor hybrid inspection point recognition module (6).

4. The 5G-based multi-sensor inspection method according to claim 2, characterized in that: The step S2 comprises the following steps: 2.1 Audio punch-in module (2) collects surrounding sounds and generates PCM audio data; 2.2 Audio punch-in module (2) slices the PCM audio data into 4 millisecond time domain data slices; 2.3 Audio punch-in module (2) calculates 4 millisecond time domain data slices, adds up the absolute value of each sampling point to get the slice energy value, and if the slice energy value is greater than the set threshold, it enters the next step of the process; 2.4 Audio punch-in module (2) performs FFT Fourier transform to convert the time domain data slices into frequency domain data slices. The frequency domain data slices are the energy values of each frequency in the time domain slices. 2.5 Audio tag calibration module (4) sorts the frequency domain data in descending order of energy value and standardizes them according to the energy value. The standard value of the first frequency is 100, and the standard values of other frequencies are 100*energy value / energy value of the first frequency. 2.6 The audio punch-in module (2) obtains the frequency domain data slices in the order of energy value from large to small and in a standardized manner; 2.7 The audio punch-in module (2) performs a calibration operation and sends the standardized frequency domain data slices in descending order of energy value to the audio tag calibration module (4); 2.8 The audio punch-in module (2) performs the inspection punch-in operation and sends the standardized frequency domain data slices in descending order of energy value to the multi-sensor hybrid inspection point identification module (6).

5. The 5G-based multi-sensor inspection method according to claim 2, characterized in that: The step S3 comprises the following steps: 3.1 The BeiDou / GPS positioning punch-in module (7) receives radio signals continuously sent by BeiDou / GPS satellites in the air with time and location information. The satellites and the BeiDou / GPS positioning punch-in module (7) simultaneously generate the same pseudo-random code. When the two codes are synchronized, the time delay is measured and the pseudo-distance is obtained by multiplying the time delay by the speed of light. 3.2 The BeiDou / GPS positioning punch-in module (7) synchronizes the signals of the four satellites and finally obtains the three-dimensional position of the BeiDou / GPS positioning punch-in module (7) based on the pseudo-range measurement. The BeiDou / GPS positioning punch-in module (7) sends the positioning information to the multi-sensor hybrid inspection point identification module (6); 3.3 When the BeiDou / GPS positioning clocking module (7) cannot synchronize to the signals of the four satellites, the BeiDou / GPS positioning failure is fed back to the multi-sensor hybrid inspection point identification module (6).

6. The 5G-based multi-sensor inspection method according to claim 2, characterized in that: The step S4 comprises the following steps: 4.

1. 5G base station positioning punch-in module (8) receives 5G base station positioning signal for positioning; 4.

2. The 5G base station transmits a signal to the 5G base station positioning punching module (8). The 5G base station positioning punching module (8) transmits the signal back. The 5G base station calculates the distance between the base station and the mobile phone based on the round-trip time. Then, the 5G base station locates the location of the 5G base station positioning punching module (8) by positioning the arrival angle and the departure angle. The 5G base station sends the positioning information to the 5G base station positioning punching module (8). The 5G base station positioning punching module (8) sends the positioning data to the multi-sensor hybrid inspection point identification module (6). 4.

3. When the 5G base station positioning clocking-in module (8) cannot communicate with any 5G base station, the 5G base station positioning clocking-in module (8) fails to clock in and notifies the multi-sensor hybrid inspection point identification module (6).

7. The 5G-based multi-sensor inspection method according to claim 2, characterized in that: The step S6 comprises the following steps: 6.1 Video Label Calibration Module (3) receives an image object list with relative distances to objects and sorted from near to far and an inspection point ID. The image object list includes object types and relative distances to objects. 6.2 The video label calibration module (3) obtains the image object list data of all inspection points from the multi-sensor inspection point library module (5); 6.3 Video label calibration module (3) compares the image object list with the image object list of all inspection points from near to far according to the object type and the relative distance of the object. If the relative position objects of the two image object lists are different, the process stops. If the same ratio exceeds 80%, the image object list is considered to match the inspection point, and the video label calibration module (3) fails. 6.4 Video Label Calibration Module (3) If the same ratio of the image object list of the inspection point is less than 80%, the video label calibration is successful; 6.5 The video tag calibration module (3) sends the image object list with the relative distance of the objects and sorted from near to far and the inspection point ID to the multi-sensor inspection point library module (5).

8. The 5G-based multi-sensor inspection method according to claim 4, characterized in that: The step S7 comprises the following steps: 7.1 The audio tag calibration module (4) receives the frequency domain data fragments and inspection point IDs in the order of energy values from large to small and standardized from the audio punching module (2); 7.2 The audio tag calibration module (4) obtains the frequency domain data fragments of all inspection points from the multi-sensor inspection point library module (5); 7.3 The audio tag calibration module (4) compares the frequency domain data slices of the inspection point of the multi-sensor inspection point library module (5) in the order of energy value from large to small, and stops the calibration if the frequencies are different. If the same ratio exceeds 80%, it is considered that the frequency domain data slice matches the inspection point and the calibration of the audio tag calibration module (4) fails. 7.4 The audio tag calibration module (4) successfully calibrates the frequency domain data segments of all inspection points with a ratio of less than 80%. The audio tag calibration module (4) sends the standardized frequency domain data segments and the inspection point ID in descending order of energy value to the multi-sensor inspection point library module (5).

9. The 5G-based multi-sensor inspection method according to claim 4, characterized in that: The step S8 comprises the following steps: 8.1 The multi-sensor inspection point library module (5) receives the image object list with relative distances of objects and sorted from near to far and the inspection point ID from the video tag calibration module (3), and the multi-sensor inspection point library module (5) saves the image object list data to the inspection point according to the inspection point ID; 8.2 The multi-sensor inspection point library module (5) receives the energy values of the audio tag calibration module (4) in descending order and in a standardized manner in the frequency domain data fragments and the inspection point ID, and the multi-sensor inspection point library module (5) saves the frequency domain data fragments of the inspection point according to the inspection point ID.

10. The 5G-based multi-sensor inspection method according to claim 4, characterized in that: The step S9 comprises the following steps: 9.1 The multi-sensor hybrid inspection point identification module (6) reads the information of all inspection points of the multi-sensor inspection point library module (5). The inspection point information includes inspection point ID, tag ID, location coordinates, frequency domain data slices, and image object list data; 9.2 Multi-sensor hybrid inspection point identification module (6) uses each sensor to identify the punch-in information according to the preset priority; 9.3 The multi-sensor hybrid inspection point identification module (6) performs tag punching. The multi-sensor hybrid inspection point identification module (6) receives the tag ID of the tag punching module (9), matches the tag ID of the inspection point according to the received tag ID, and determines that the inspection point ID of the inspection point is the punching inspection point ID. After the multi-sensor hybrid inspection point identification module (6) successfully punches in the tag, it determines whether there are other sensors at the inspection point, and selects other sensors for punching in according to priority. Otherwise, it is considered that the identification is successful and the punching information is notified to the 5G upload module (10); 9.4 The multi-sensor hybrid inspection point identification module (6) notifies the Beidou / GPS positioning punching module (7) to perform location punching or receive the location of the Beidou / GPS positioning punching module (7) according to the priority. If the multi-sensor hybrid inspection point identification module (6) receives the location of the Beidou / GPS positioning punching module (7) successfully, it will determine the location punching. If the positioning fails, other sensors will be selected according to the priority to perform punching. 9.5 The multi-sensor hybrid inspection point identification module (6) receives the position of the 5G base station positioning punching module (8) according to the priority notification. If the positioning is successful, the position punching is determined. The multi-sensor hybrid inspection point identification module (6) receives the position of the 5G base station positioning punching module (8). If the positioning fails, it is considered that the punching has failed. The position punching process is determined as follows: the multi-sensor hybrid inspection point identification module (6) determines whether the punching inspection point ID has been confirmed. If the punching inspection point ID has been confirmed, the position of the 5G base station positioning punching module (8) is calculated with the position coordinates of the inspection point of the punching inspection point ID. If the distance difference is less than the threshold value, the position punching is considered successful. If the punching inspection point ID is not confirmed, the position of the 5G base station positioning punching module (8) is calculated with the position coordinates of all inspection points. If the distance difference is less than the threshold value, the position punching is considered successful. The inspection point ID is the punching inspection point ID. 9.6 After the multi-sensor hybrid inspection point identification module (6) successfully clocks in at the location, it determines whether there are other sensors at the inspection point, and selects other sensors for clocking in according to priority. Otherwise, it considers the identification successful and notifies the 5G upload module (10) of the clocking in information. 9.7 The multi-sensor hybrid inspection point identification module (6) notifies the dual-camera video clocking module (1) to perform video clocking in or receives video clocking in from the dual-camera video clocking module (1) according to the priority; 9.8 The multi-sensor hybrid inspection point identification module (6) receives the image object list with the relative distance of the objects and sorted from near to far from the dual-camera video punch-in module (1), and the multi-sensor hybrid inspection point identification module (6) determines whether the punch-in inspection point ID has been confirmed; 9.9 If the ID of the check-in inspection point has been confirmed, the multi-sensor hybrid inspection point identification module (6) compares the image object list of the inspection point corresponding to the check-in inspection point ID with the object type and the relative distance of the object from near to far according to the object type. If the relative position objects of the two image object lists are different, the module stops. If the same ratio exceeds 80%, the video check-in is considered successful. Otherwise, the check-in is considered unsuccessful. 9.10 If the multi-sensor hybrid inspection point identification module (6) fails to confirm the punch-in inspection point ID, the multi-sensor hybrid inspection point identification module (6) compares the image object list with the image object list of all inspection points from near to far according to the object type and the relative distance of the object. If the relative position objects of the two image object lists are different, the module stops. If the same ratio exceeds 80%, the image object list is considered to match the inspection point, and the inspection ID of the inspection point is the punch-in inspection ID. If the same ratio of the multi-sensor hybrid inspection point identification module (6) is less than 80%, the video punch-in fails. 9.11 After the multi-sensor hybrid inspection point identification module (6) successfully clocks in at the location, it determines whether there are other sensors at the inspection point, and selects other sensors for clocking in according to priority. Otherwise, it considers the identification successful and notifies the 5G upload module (10) of the clocking in information; 9.12 The multi-sensor hybrid inspection point identification module (6) notifies the audio punching module (2) to perform audio punching or receive audio punching according to the priority, and the multi-sensor hybrid inspection point identification module (6) receives the frequency domain data slices in the order of energy value from large to small and standardized; 9.13 The multi-sensor hybrid inspection point identification module (6) determines whether the punch-in inspection point ID has been confirmed. If the punch-in inspection point ID has been confirmed, the multi-sensor hybrid inspection point identification module (6) compares the frequency domain data of the inspection point corresponding to the punch-in inspection point ID in descending order of energy value. If the frequencies are different, the module stops. If the same ratio exceeds 80%, the audio punch-in is considered successful, otherwise the punch-in fails. 9.14 Multi-sensor hybrid inspection point identification module (6) If the punch-in inspection point ID is not confirmed, the frequency domain data of all inspection points are sliced in descending order of energy value, and the frequency is compared. If the frequencies are not the same, the module stops. If the same ratio exceeds 80%, it is considered to match the inspection point, the audio punch-in is successful, and the inspection point ID of the inspection point is the punch-in inspection point ID. If the same ratio is less than 80%, the punch-in is considered to have failed. 9.15 After the multi-sensor hybrid inspection point identification module (6) successfully clocks in with audio, the multi-sensor hybrid inspection point identification module (6) notifies the 5G upload module (10) of the clocking-in information. The clocking-in information must include: the clocking-in inspection point ID and the inspector ID, the clocking-in time, and the clocking-in information also includes: the tag ID, the location coordinates, the frequency domain data fragmentation, the image object list data, and the inspector ID is entered in advance by the multi-sensor hybrid inspection point identification module (6).

Images