Highway field detection data multi-modal feature fusion and remote collaborative management system

Abstract

The application discloses a highway on-site detection data multi-modal feature fusion and remote collaborative management system and relates to the technical field of engineering management informatization. The system comprises the following steps: obtaining highway detection total task information, generating a subtask template, dividing a to-be-detected highway into a plurality of detection units and matching a corresponding standardized subtask for each detection unit to obtain a subtask set, encapsulating collected data, a subtask identifier, a collector, a collection time and collection environment information into a subtask data block based on each standardized subtask, establishing a mapping relationship between the subtask identifier and the data block, sending the mapping relationship to a remote management end and receiving collaborative opinions, and updating an execution state; identifying an opinion type, generating an on-site detection execution instruction in combination with detection unit information and issuing the on-site detection execution instruction, and updating a subtask execution state according to feedback. Through type identification of the collaborative opinions, the on-site detection execution instruction is automatically generated and issued to an on-site execution end, and linkage between remote auditing and on-site execution is realized.

Description

Technical Field

[0001] This invention relates to the field of information technology for engineering management, specifically to a multimodal feature fusion and remote collaborative management system for highway on-site inspection data. Background Technology

[0002] Highway inspection is a fundamental task for ensuring the safe operation of highway infrastructure and making scientific maintenance decisions. With the continuous expansion of the highway network and the development of inspection technology, highway inspection has gradually evolved from manual visual inspection to a multimodal integrated inspection mode that integrates multiple sensing methods such as laser measurement, image acquisition, and infrared imaging. At the same time, the management mode of inspection tasks is also facing the need to transform from traditional manual scheduling to information-based and remote collaborative methods.

[0003] The existing management of highway inspection tasks mainly suffers from the following problems: Current testing tasks are mostly assigned on a road segment-by-segment basis. Testing personnel arrange testing items and collection order based on their experience. Different testing personnel have different understandings and execution methods for the same type of task, which makes it difficult to ensure the consistency of data collection and the horizontal comparability of test results is insufficient. Data collected on-site needs to be brought back in a unified manner after the testing task is completed or uploaded in batches manually. Remote reviewers cannot check the data quality in a timely manner. When defects or omissions are found in the data, the testing vehicles and personnel have often left the site, requiring rescheduling and rework, resulting in a waste of time and resources. Furthermore, a closed loop cannot be formed between problem discovery and handling. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a multimodal feature fusion and remote collaborative management system for highway field inspection data.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a multimodal feature fusion and remote collaborative management system for highway on-site inspection data, specifically comprising: Data acquisition and task decomposition module: acquires the total highway inspection task information, generates sub-task templates based on the total highway inspection task information, according to the inspection type and execution standards, divides the highway to be inspected into several inspection units, and matches each inspection unit with a corresponding standardized sub-task to obtain a set of sub-tasks; Data block generation module: Based on each standardized subtask, collect multimodal detection data according to its required data types, and encapsulate the collected data with subtask identifier, collection personnel, collection time and collection environment information into subtask data blocks; Collaborative Interaction Module: Establishes a mapping relationship between subtask identifiers and subtask data blocks, sends subtask data blocks to the remote management terminal based on the mapping relationship, receives collaborative opinions returned by the remote management terminal, and updates the execution status of the corresponding subtask. Instruction generation module: Based on the returned collaborative opinions, identify the opinion type, combine the detection unit information of the corresponding subtask to generate on-site detection execution instructions and send them to the on-site execution terminal, and update the subtask execution status according to the feedback from the on-site execution terminal.

[0006] Preferably, the data acquisition and task decomposition module acquires the start and end location information of the road segment of the highway to be inspected, the structural type information of the highway to be inspected, the inspection range information of this inspection task, the inspection type information of this inspection task, the inspection item information of this inspection task, the accuracy requirement information of this inspection task, and the execution standard information. The information on the start and end points of the road section, the structural type, the inspection range, the inspection type, the inspection items, the accuracy requirements, and the implementation standards are combined into the overall highway inspection task information. Extract inspection type information and execution standard information from the overall highway inspection task information. Based on the execution standard information, determine the mandatory data type, collection order requirements, and data format requirements for each inspection item included in the inspection type information. Each test item and its corresponding mandatory data type, collection order requirements and data format requirements are combined into a standardized subtask template, and the standardized subtask templates corresponding to all test items under the test type information are combined into a subtask template set. Extract the detection range information from the overall highway inspection task information. Based on the detection range information, divide the detection range of the highway to be inspected into several connected detection units along the highway travel direction, and assign a unique detection unit identifier to each detection unit. For each detection unit obtained by division, all standardized subtask templates corresponding to the detection type information are retrieved from the subtask template set, instantiated as standardized subtasks belonging to the current detection unit, and a subtask identifier is assigned to each instantiated standardized subtask. The subtask identifier is composed of the detection unit identifier of the detection unit and the type code of the standardized subtask template. The standardized subtasks instantiated from all detection units are aggregated, and each of the aggregated standardized subtasks is marked with a first execution state. The first execution state indicates that the current standardized subtask has not yet been collected and executed, thus obtaining a set of subtasks.

[0007] Preferably, the data block generation module determines the standardized subtask to be executed according to the execution order of the standardized subtasks in the subtask set, and the execution order is arranged according to the acquisition order requirements corresponding to the standardized subtasks and the spatial order of the detection units. Extract the required data types and collection requirements from the standardized subtask templates corresponding to the currently pending standardized subtasks; According to the data collection requirements, select the data collection equipment corresponding to the data types that must be collected, and use the selected data collection equipment to collect on-site data of the detection unit to which the standardized subtask to be executed belongs, so as to obtain multimodal detection data; Record the identity information of the personnel performing this data collection, the start and end times of the data collection, as well as the ambient temperature and humidity information during the data collection, as the personnel information, data collection time information, and data collection environment information; The collected multimodal detection data, the subtask identifier of the standardized subtask to be executed, the information of the data collector, the data collection time, and the data collection environment are encapsulated to form a subtask data block; Perform data integrity verification on the subtask data block. The data integrity verification includes verifying whether each field in the subtask data block is completely filled and verifying whether the amount of multimodal detection data meets the preset range. After the verification is passed, the subtask data block is stored in the local data cache.

[0008] Preferably, the collaborative interaction module extracts the subtask identifier encapsulated in each subtask data block that has been stored in the local data cache, establishes a one-to-one mapping relationship between the subtask identifier and the storage path of the subtask data block in the local data cache, and obtains a mapping relationship table. Based on the priority order of standardized subtasks, the storage paths of the corresponding subtask data blocks are obtained sequentially from the mapping table. The corresponding subtask data blocks are read according to the storage paths, and the read subtask data blocks are sent to the remote management terminal through the network transmission channel. At the same time, the execution status of the standardized subtask corresponding to the sent subtask data blocks is marked as sent. After sending, listen for and receive the collaborative opinion data returned by the remote management terminal for the sent subtask data block. The collaborative opinion data includes the collaborative opinion content, remote reviewer information, and review timestamp. The received collaborative opinion data is bound to the subtask identifier of the standardized subtask corresponding to the sent subtask data block, and the bound collaborative opinion data is stored in the local database. The execution status of the corresponding standardized subtask is updated from the first execution status to the second execution status. The second execution status indicates that the multimodal detection data corresponding to the standardized subtask has been submitted to the remote management terminal and collaborative opinions have been received.

[0009] Preferably, the instruction generation module extracts collaborative opinion data bound to the subtask identifier of the standardized subtask from the local database for each standardized subtask in the second execution state. The content of collaborative opinions in the collaborative opinion data is identified by type, and the opinion type of the collaborative opinion content is determined to be either the first opinion type or the second opinion type. The first opinion type indicates that the multimodal detection data corresponding to the standardized sub-task has been approved and does not require supplementary collection, while the second opinion type indicates that the multimodal detection data corresponding to the standardized sub-task has defects and requires on-site supplementary collection. When the opinion type of the collaborative opinion content is the first opinion type, a first on-site detection execution instruction containing task completion confirmation information is generated. When the opinion type of the collaborative opinion content is the second opinion type, based on the supplementary collection requirements specified in the collaborative opinion content, and combined with the detection unit identifier and location information of the detection unit to which the standardized subtask belongs, a second on-site detection execution instruction containing supplementary collection operation content is generated. The generated first or second field detection execution command will be sent to the field execution terminal via wireless communication. Receive instruction execution feedback information returned by the field execution terminal, and update the execution status of the standardized subtask to the third execution status based on the instruction execution feedback information. The third execution status indicates that the collaborative opinions corresponding to the standardized subtask have been processed and the task has been closed.

[0010] This invention provides a multimodal feature fusion and remote collaborative management system for highway field inspection data, which has the following beneficial effects: This invention breaks down the overall highway inspection task into several standardized sub-tasks and binds each standardized sub-task to a specific inspection unit and multimodal data type, thereby achieving refined management of the inspection task. Each standardized sub-task includes mandatory data types, collection order, and data format requirements, enabling different inspection personnel to follow unified standards when performing similar tasks, thus improving the consistency and comparability of inspection data.

[0011] By establishing a mapping relationship between subtask identifiers and subtask data blocks, and sending data blocks to the remote management terminal based on priority, real-time or near-real-time remote collaboration of on-site detection data is realized. Remote auditors can promptly check data quality and return collaborative opinions, significantly shortening the time cycle from data collection to quality confirmation.

[0012] By identifying the types of collaborative opinions, the system automatically generates corresponding on-site inspection execution instructions and sends them to the on-site execution terminal, realizing the linkage between remote review and on-site execution. When the review finds data defects that require supplementary collection, it can immediately generate instructions containing specific supplementary collection requirements to guide on-site personnel to respond quickly. Attached Figure Description

[0013] Figure 1 This is a system flowchart of the present invention. Detailed Implementation

[0014] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0015] Please see Figure 1 This invention provides a multimodal feature fusion and remote collaborative management system for highway on-site detection data, comprising the following steps: Data acquisition and task decomposition module: acquires the total highway inspection task information, generates sub-task templates based on the total highway inspection task information, according to the inspection type and execution standards, divides the highway to be inspected into several inspection units, and matches each inspection unit with a corresponding standardized sub-task to obtain a set of sub-tasks; Furthermore, the data acquisition and task decomposition module acquires the start and end location information of the road segment to be detected, which includes the starting station number and the ending station number. Obtain the structural type information of the highway to be inspected. The structural type information includes the identifier of at least one structural type among pavement section, bridge section and tunnel section. Obtain the detection range information for this detection task. The detection range information includes the lane range information to be detected and the detection depth range information. Obtain the detection type information for this detection task. The detection type information includes at least one of the following: regular inspection, special inspection, and emergency inspection. Obtain the inspection item information for this inspection task. The inspection item information includes at least one of the following: road surface smoothness, road surface cracks, bridge deck pavement condition, and tunnel lining condition. Obtain the accuracy requirements and execution standards for this inspection task, and combine the road segment start and end location information, structure type information, inspection range information, inspection type information, inspection item information, accuracy requirements information, and execution standards information into the overall highway inspection task information; Extract inspection type information and execution standard information from the overall highway inspection task information. Based on the execution standard information, determine the mandatory data type, collection order requirements, and data format requirements for each inspection item included in the inspection type information. Each test item and its corresponding mandatory data type, collection order requirements and data format requirements are combined into a standardized subtask template, and the standardized subtask templates corresponding to all test items under the test type information are combined into a subtask template set. Extract the detection range information from the overall highway inspection task information. Based on the detection range information, divide the detection range of the highway to be inspected into several connected detection units along the highway travel direction, and assign a unique detection unit identifier to each detection unit. For each detection unit obtained by division, all standardized subtask templates corresponding to the detection type information are retrieved from the subtask template set, instantiated as standardized subtasks belonging to the current detection unit, and a subtask identifier is assigned to each instantiated standardized subtask. The subtask identifier is composed of the detection unit identifier of the detection unit and the type code of the standardized subtask template. The standardized subtasks instantiated from all detection units are aggregated, and each of the aggregated standardized subtasks is marked with a first execution state. The first execution state indicates that the current standardized subtask has not yet been collected and executed, thus obtaining a set of subtasks.

[0016] It should be noted that the following information is obtained from the task release system or inspection management platform: the start and end location information of the road segment, the structure type information, the inspection range information, the inspection type information, the inspection item information, the accuracy requirement information, and the execution standard information. The start and end location information of the road segment is obtained by reading the mileage field in the task instruction. The structure type information is obtained by querying the road segment attributes in the highway asset management system. The inspection range information, the inspection type information, the inspection item information, the accuracy requirement information, and the execution standard information are obtained by parsing the corresponding fields in the task instruction. All the information obtained is combined according to the preset field structure to form the total highway inspection task information.

[0017] Extract inspection type information, periodic inspection information, and execution standard information from the overall highway inspection task information. Based on the execution standard information, determine the mandatory data types, collection order requirements, and data format requirements for each inspection item included in the periodic inspection of inspection type information. Among them, the mandatory data types for road surface smoothness inspection are laser point cloud elevation data and inertial navigation attitude data, and the collection order requirement is to be carried out before crack inspection. The mandatory data types for road surface crack inspection are road surface grayscale images and depth images, and the collection order requirement is to be carried out after smoothness inspection. The mandatory data types for bridge deck pavement condition inspection are high-definition bridge deck images and infrared thermal imaging.

[0018] Each inspection item and its corresponding mandatory data type, collection order requirements, and data format requirements are combined into a standardized subtask template. Specifically, the road surface smoothness inspection item and its corresponding information are combined into a standardized subtask template, the road surface crack inspection item and its corresponding information are combined into a standardized subtask template, the bridge deck pavement condition inspection item and its corresponding information are combined into a standardized subtask template, and the three standardized subtask templates corresponding to all inspection items under the periodic inspection of inspection type information are combined into a subtask template set.

[0019] The detection range information is extracted from the overall highway inspection task information. Based on the detection range information and the preset detection unit division length (set to 100 meters in this embodiment), the detection range of the highway to be inspected is divided into multiple connected detection units along the highway travel direction. Each detection unit is assigned a unique detection unit identifier. K100+000 to K120+000 is divided into 200 detection units. The detection unit identifier format is unit-starting chainage. For example, unit-K100+000 represents a 100-meter road segment starting from K100+000, and unit-K118+000 represents a bridge section detection unit starting from K118+000.

[0020] For each detection unit obtained from the division, all standardized subtask templates corresponding to the detection type information are retrieved from the subtask template set and instantiated as standardized subtasks belonging to the current detection unit. For the road section detection unit between K100+000 and K117+900, two standardized subtask templates for road surface smoothness and road surface cracks are retrieved and instantiated. For the bridge section detection unit between K118+000 and K120+000, three standardized subtask templates for road surface smoothness, road surface cracks, and bridge deck paving condition are retrieved and instantiated. A subtask identifier is assigned to each instantiated standardized subtask. The subtask identifier is composed of the detection unit identifier of the detection unit and the type code of the standardized subtask template. For example, unit-K100+000-PROF-001 represents the road surface smoothness subtask of unit K100+000, and unit-K118+000-BRIDGE-003 represents the bridge deck paving condition subtask of unit K118+000.

[0021] The standardized subtasks instantiated from all detection units are aggregated, and each of the aggregated standardized subtasks is marked with a first execution state. The first execution state indicates that the current standardized subtask has not yet been collected and executed, thus obtaining a set of subtasks.

[0022] Data block generation module: Based on each standardized subtask, collect multimodal detection data according to its required data types, and encapsulate the collected data with subtask identifier, collection personnel, collection time and collection environment information into subtask data blocks; Furthermore, the data block generation module determines the standardized subtask to be executed according to the execution order of the standardized subtasks in the subtask set. The execution order is arranged according to the acquisition order requirements corresponding to the standardized subtasks and the spatial order of the detection units. Extract the required data types and collection requirements from the standardized subtask templates corresponding to the currently pending standardized subtasks; According to the data collection requirements, select the data collection equipment corresponding to the data types that must be collected, and use the selected data collection equipment to collect on-site data of the detection unit to which the standardized subtask to be executed belongs, so as to obtain multimodal detection data; Record the identity information of the personnel performing this data collection, the start and end times of the data collection, as well as the ambient temperature and humidity information during the data collection, as the personnel information, data collection time information, and data collection environment information; The collected multimodal detection data, the subtask identifier of the standardized subtask to be executed, the information of the data collector, the data collection time, and the data collection environment are encapsulated to form a subtask data block; Perform data integrity verification on the subtask data block. The data integrity verification includes verifying whether each field in the subtask data block is completely filled and verifying whether the amount of multimodal detection data meets the preset range. After the verification is passed, the subtask data block is stored in the local data cache.

[0023] It should be noted that the standard subtask to be executed is determined according to the execution order of the standardized subtasks in the subtask set. The execution order is based on the collection order requirements and spatial order of the detection units corresponding to the standardized subtasks. In this embodiment, the road surface smoothness subtask of the K100+000 detection unit is executed first, then the road surface crack subtask of the same unit is executed, and then the various subtasks of the subsequent detection units are executed sequentially along the mileage direction.

[0024] Extract the required data types and collection requirements from the standardized subtask templates corresponding to the currently pending standardized subtasks. Taking the road surface smoothness subtask as an example, the required data types extracted are laser point cloud elevation data and inertial navigation attitude data.

[0025] For the road surface smoothness subtask, a vehicle-mounted laser smoothness meter and an inertial measurement unit were selected as the acquisition devices. The selected acquisition devices were used to collect on-site data of the detection unit to which the standardized subtask to be executed belonged. The detection vehicle traveled at a speed of 70 km / h on the road section from K100+000 to K100+100. The laser smoothness meter collected road surface elevation point cloud data, and the inertial measurement unit simultaneously collected attitude data to obtain multimodal detection data.

[0026] Record the identity information of the personnel performing this data collection (inspector's name and employee number), the start and end times of the data collection, as well as the ambient temperature and humidity information during the data collection, as the personnel information, data collection time information, and data collection environment information.

[0027] The collected multimodal detection data, the subtask identifier of the standardized subtask to be executed, the information of the data collector, the data collection time, and the data collection environment are encapsulated to form a subtask data block.

[0028] Perform data integrity verification on the encapsulated subtask data block. The data integrity verification includes verifying whether each field in the subtask data block is completely filled, and verifying whether the amount of multimodal detection data meets the preset range. After the verification is passed, the subtask data block is stored in the local data cache area of ​​the on-board industrial control computer of the detection vehicle.

[0029] Data integrity verification is divided into two stages: field integrity verification and data volume compliance verification. The next stage can only be entered after the previous stage is passed. If either verification fails, the data block verification of that subtask is deemed to have failed.

[0030] Field integrity verification checks whether all required fields in the subtask data block have been filled and whether the filled values ​​are not empty. The specific checks include: The subtask identifier field, multimodal detection data field, data acquisition personnel information field, data acquisition time information field, and data acquisition environment information field are checked sequentially. Each field is verified to exist and to have a non-empty value. For multimodal detection data fields, further check whether they contain data subfields corresponding to the mandatory data types in the current standardized subtask template, and whether the content of each data subfield can be parsed normally; For the personnel information field, further check whether it contains two subfields, the personnel name and the personnel employee number, and whether both are not empty; for the collection time information field, further check whether it contains two subfields, the collection start time and the collection end time, and whether the format is compliant and the end time is later than the start time. For the environmental information field, further check whether it contains two subfields, environmental temperature and environmental humidity, and whether their values ​​are within a reasonable range.

[0031] If any of the above fields are missing or have an empty value, the validation will terminate immediately and an error message indicating that the field is incomplete will be returned.

[0032] After the first phase of data volume compliance verification is passed, compliance judgment is made for each type of mandatory data type in the multimodal detection data based on its corresponding preset data volume range.

[0033] The preset range refers to: for each type of mandatory data type, based on the length of the detection unit in the current detection task, the technical parameters of the acquisition equipment, and the accuracy requirements specified in the execution standard, the minimum data volume threshold that the data type should acquire should reach is determined in advance. The minimum data volume threshold can be determined through the engineering conversion relationship between the length of the detection unit and the sampling parameters of the acquisition equipment, and is used to characterize the minimum amount of effective data that the data type should generate under normal acquisition conditions.

[0034] Regarding the statement that "the minimum data threshold within the preset range can be determined by the engineering conversion relationship between the length of the detection unit and the sampling parameters of the acquisition device," the following section elaborates on the determination process of this conversion relationship from a logical perspective. It does not involve specific calculation formulas, but only describes the logical derivation relationship between various parameters in words.

[0035] Engineering conversion logic for the minimum data volume threshold within the preset range: The determination of the minimum data volume threshold for the preset range includes three basic pieces of information: the spatial scale information of the detection unit, the technical parameters of the acquisition equipment, and the accuracy requirements specified in the execution standard. This is determined through the following steps: First, the spatial scale information of the detection unit is obtained from the detection unit division length used when dividing the detection unit, including the length of the detection unit along the direction of travel of the highway. This length determines the continuous spatial span that the acquisition equipment needs to cover in this acquisition task.

[0036] Second, the technical parameter information of the acquisition equipment comes from the technical specification document of the acquisition equipment used in this testing task. The technical parameters of different equipment types are different. For continuous sampling equipment, the technical parameters include the number of times the equipment performs sampling within a unit travel distance. This parameter is determined by the sampling trigger frequency set when the equipment leaves the factory and the standard operating speed of the testing vehicle. For imaging equipment, technical parameters include the number of image frames generated per unit travel distance. This parameter is determined by the equipment's frame rate and the standard operating speed of the inspection vehicle. From these technical parameters, we can deduce the number of samples or data points the equipment should generate under normal operating conditions within a unit length of inspection unit.

[0037] Third, the accuracy requirement information is extracted from the overall highway inspection task information, including the spatial sampling interval requirement or minimum resolution requirement for each type of mandatory data type. This accuracy requirement directly defines the minimum number of effective sampling points or effective pixels that should be included per unit length, and is the minimum threshold for judging whether the data volume meets the inspection requirements.

[0038] Based on the above three aspects of information, the specific steps of the engineering conversion are as follows: First, associate the spatial scale information of the detection unit with the accuracy requirement information in the execution standard to determine the minimum number of effective data units that must be achieved to meet the accuracy requirements under the length of the detection unit. The minimum number of valid data units is compared and verified with the standard output data of the acquisition device within a unit length to ensure that the device capability meets the accuracy requirements. The minimum number of valid data units that must be reached under the length of the detection unit is determined as the preset minimum data volume threshold corresponding to the mandatory data type.

[0039] For data stored in file format, the minimum number of valid data units is converted into the minimum file data size threshold. The conversion is based on the standard space occupied by each data unit under the storage format. The standard space occupied is determined by the encoding method and resolution parameters specified in the data format requirements.

[0040] During verification, based on the mandatory data types defined in the current standardized subtask template, the preset minimum data volume threshold corresponding to each mandatory data type is retrieved one by one. The actual data volume of the corresponding type in the collected multimodal detection data is compared with the preset minimum data volume threshold. If the actual data volume of all mandatory data types is not lower than the corresponding preset minimum data volume threshold, the data volume compliance verification passes. If the actual data volume of any mandatory data type is lower than the preset minimum data volume threshold, an error message indicating insufficient data volume is returned, specifying the data type that failed the verification.

[0041] After the above two stages of verification are completed and both are passed, the data integrity verification of the subtask data block is deemed to have passed.

[0042] Collaborative Interaction Module: Establishes a mapping relationship between subtask identifiers and subtask data blocks, sends subtask data blocks to the remote management terminal based on the mapping relationship, receives collaborative opinions returned by the remote management terminal, and updates the execution status of the corresponding subtask. Furthermore, the collaborative interaction module extracts the subtask identifier encapsulated in each subtask data block that has been stored in the local data cache, establishes a one-to-one mapping relationship between the subtask identifier and the storage path of the subtask data block in the local data cache, and obtains a mapping relationship table. Based on the priority order of standardized subtasks, the storage paths of the corresponding subtask data blocks are obtained sequentially from the mapping table. The corresponding subtask data blocks are read according to the storage paths, and the read subtask data blocks are sent to the remote management terminal through the network transmission channel. At the same time, the execution status of the standardized subtask corresponding to the sent subtask data blocks is marked as sent. After sending, listen for and receive the collaborative opinion data returned by the remote management terminal for the sent subtask data block. The collaborative opinion data includes the collaborative opinion content, remote reviewer information, and review timestamp. The received collaborative opinion data is bound to the subtask identifier of the standardized subtask corresponding to the sent subtask data block, and the bound collaborative opinion data is stored in the local database. The execution status of the corresponding standardized subtask is updated from the first execution status to the second execution status. The second execution status indicates that the multimodal detection data corresponding to the standardized subtask has been submitted to the remote management terminal and collaborative opinions have been received.

[0043] It should be noted that for each subtask data block that has been stored in the local data cache, the subtask identifier encapsulated in the subtask data block is extracted, and a mapping relationship between the subtask identifier and the storage path of the subtask data block in the local data cache is established to obtain a mapping relationship table.

[0044] Based on the priority order of standardized subtasks (arranged according to the mileage order of detection units and the order of subtask acquisition), the storage path of the corresponding subtask data block is obtained sequentially from the mapping table. The corresponding subtask data block is read according to the storage path, and the read subtask data block is sent to the remote management terminal through the 4G / 5G network transmission channel. At the same time, the execution status of the standardized subtask corresponding to the sent subtask data block is marked as sent.

[0045] After sending, listen for and receive collaborative feedback data returned by the remote management terminal for the sent subtask data block. The collaborative feedback data includes the content of the collaborative feedback, information of the remote reviewer, and review timestamp.

[0046] The received collaborative opinion data is bound to the subtask identifier of the standardized subtask corresponding to the sent subtask data block, and the bound collaborative opinion data is stored in the collaborative opinion table.

[0047] The execution status of the corresponding standardized subtask is updated from the first execution status to the second execution status. The second execution status indicates that the multimodal detection data corresponding to the standardized subtask has been submitted to the remote management terminal and collaborative opinions have been received.

[0048] Instruction generation module: Based on the returned collaborative opinions, identify the opinion type, combine the detection unit information of the corresponding subtask to generate on-site detection execution instructions and send them to the on-site execution terminal, and update the subtask execution status according to the feedback from the on-site execution terminal.

[0049] Furthermore, the instruction generation module extracts collaborative opinion data bound to the subtask identifier of the standardized subtask from the local database for each standardized subtask in the second execution state. The content of collaborative opinions in the collaborative opinion data is identified by type, and the opinion type of the collaborative opinion content is determined to be either the first opinion type or the second opinion type. The first opinion type indicates that the multimodal detection data corresponding to the standardized sub-task has been approved and does not require supplementary collection, while the second opinion type indicates that the multimodal detection data corresponding to the standardized sub-task has defects and requires on-site supplementary collection. When the opinion type of the collaborative opinion content is the first opinion type, a first on-site detection execution instruction containing task completion confirmation information is generated. When the opinion type of the collaborative opinion content is the second opinion type, based on the supplementary collection requirements specified in the collaborative opinion content, and combined with the detection unit identifier and location information of the detection unit to which the standardized subtask belongs, a second on-site detection execution instruction containing supplementary collection operation content is generated. The generated first or second field detection execution command will be sent to the field execution terminal via wireless communication. Receive instruction execution feedback information returned by the field execution terminal, and update the execution status of the standardized subtask to the third execution status based on the instruction execution feedback information. The third execution status indicates that the collaborative opinions corresponding to the standardized subtask have been processed and the task has been closed.

[0050] It should be noted that for each standardized subtask in the second execution state, collaborative opinion data bound to the subtask identifier of the standardized subtask is extracted from the local database.

[0051] The content of collaborative opinions in the collaborative opinion data is identified by type, and the opinion type is determined to be either the first opinion type or the second opinion type. The first opinion type indicates that the multimodal detection data corresponding to the standardized sub-task has been approved and does not require supplementary collection. The second opinion type indicates that the multimodal detection data corresponding to the standardized sub-task has defects and requires on-site supplementary collection.

[0052] When the collaborative opinion content is of the first opinion type, a first on-site inspection execution instruction containing task completion confirmation information is generated. The instruction content indicates that the task has been approved and no further action is required. When the collaborative opinion content is of the second opinion type, based on the supplementary collection requirements specified in the collaborative opinion content, combined with the detection unit identifier and location information of the detection unit to which the standardized subtask belongs, a second on-site inspection execution instruction containing supplementary collection operation content is generated. The instruction content describes in detail the location, data type, and method of supplementary collection.

[0053] The generated first or second on-site inspection execution command is sent to the on-site execution terminal (the inspection personnel's handheld tablet terminal) via wireless communication. After receiving the second on-site inspection execution command, the on-site execution terminal sends the inspection personnel to the designated location to re-acquire crack images and encapsulates and uploads the newly acquired data according to the aforementioned process; the task is confirmed to be completed upon receiving the first on-site inspection execution command.

[0054] Receive instruction execution feedback information returned by the field execution terminal, and update the execution status of the standardized subtask to the third execution status according to the instruction execution feedback information. The third execution status indicates that the collaborative opinions corresponding to the standardized subtask have been processed and the task is closed. For subtasks of the first opinion type, the feedback is confirmed as completed and the status is directly updated to the third execution status. For subtasks of the second opinion type, the status is updated to the third execution status after the supplemented data is resubmitted and approved and the corresponding feedback is received.

[0055] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0056] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0057] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0058] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0059] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of protection of the described technical solution.

Claims

1. A multimodal feature fusion and remote collaborative management system for highway on-site detection data, characterized in that, include: Data acquisition and task decomposition module: acquires the total highway inspection task information, generates sub-task templates based on the total highway inspection task information, according to the inspection type and execution standards, divides the highway to be inspected into several inspection units, and matches each inspection unit with a corresponding standardized sub-task to obtain a set of sub-tasks; Data block generation module: Based on each standardized subtask, collect multimodal detection data according to its required data types, and encapsulate the collected data with subtask identifier, collection personnel, collection time and collection environment information into subtask data blocks; Collaborative Interaction Module: Establishes a mapping relationship between subtask identifiers and subtask data blocks, sends subtask data blocks to the remote management terminal based on the mapping relationship, receives collaborative opinions returned by the remote management terminal, and updates the execution status of the corresponding subtask. Instruction generation module: Based on the returned collaborative opinions, identify the opinion type, combine the detection unit information of the corresponding subtask to generate on-site detection execution instructions and send them to the on-site execution terminal, and update the subtask execution status according to the feedback from the on-site execution terminal.

2. The highway field detection data multimodal feature fusion and remote collaborative management system according to claim 1, characterized in that, The data acquisition and task decomposition module acquires the start and end location information of the road segment to be inspected, the structural type information of the road to be inspected, the inspection range information of this inspection task, the inspection type information of this inspection task, the inspection item information of this inspection task, the accuracy requirement information of this inspection task, and the execution standard information. The information on the start and end points of the road section, the structural type, the inspection range, the inspection type, the inspection items, the accuracy requirements, and the implementation standards are combined into the overall highway inspection task information.

3. The highway field detection data multimodal feature fusion and remote collaborative management system according to claim 2, characterized in that, Extract inspection type information and execution standard information from the overall highway inspection task information. Based on the execution standard information, determine the mandatory data type, collection order requirements, and data format requirements for each inspection item included in the inspection type information. Each test item and its corresponding mandatory data type, collection order requirements and data format requirements are combined into a standardized subtask template, and the standardized subtask templates corresponding to all test items under the test type information are combined into a subtask template set. Extract the detection range information from the overall highway inspection task information. Based on the detection range information, divide the detection range of the highway to be inspected into several interconnected detection units along the highway travel direction, and assign a unique detection unit identifier to each detection unit.

4. The highway field detection data multimodal feature fusion and remote collaborative management system according to claim 3, characterized in that, For each detection unit obtained by division, all standardized subtask templates corresponding to the detection type information are retrieved from the subtask template set, instantiated as standardized subtasks belonging to the current detection unit, and a subtask identifier is assigned to each instantiated standardized subtask. The subtask identifier is composed of the detection unit identifier of the detection unit and the type code of the standardized subtask template. The standardized subtasks instantiated from all detection units are aggregated, and each of the aggregated standardized subtasks is marked with a first execution state. The first execution state indicates that the current standardized subtask has not yet been collected and executed, thus obtaining a set of subtasks.

5. The highway field detection data multimodal feature fusion and remote collaborative management system according to claim 1, characterized in that, The data block generation module determines the standardized subtask to be executed according to the execution order of the standardized subtasks in the subtask set. The execution order is based on the acquisition order requirements corresponding to the standardized subtasks and the spatial order of the detection units. Extract the required data types and collection requirements from the standardized subtask templates corresponding to the currently pending standardized subtasks; According to the data collection requirements, select the data collection equipment corresponding to the data types that must be collected, and use the selected data collection equipment to collect on-site data of the detection unit to which the standardized subtask to be executed belongs, so as to obtain multimodal detection data; Record the identity information of the personnel performing this data collection, the start and end times of the data collection, as well as the ambient temperature and humidity information during the data collection, as the personnel information, data collection time information, and data collection environment information; The collected multimodal detection data, the subtask identifier of the standardized subtask to be executed, the information of the data collector, the data collection time, and the data collection environment are encapsulated to form a subtask data block; Perform data integrity verification on the subtask data block. The data integrity verification includes verifying whether each field in the subtask data block is completely filled and verifying whether the amount of multimodal detection data meets the preset range. After the verification is passed, the subtask data block is stored in the local data cache.

6. The highway field detection data multimodal feature fusion and remote collaborative management system according to claim 1, characterized in that, The collaborative interaction module extracts the subtask identifier encapsulated in each subtask data block that has been stored in the local data cache, establishes a one-to-one mapping relationship between the subtask identifier and the storage path of the subtask data block in the local data cache, and obtains a mapping relationship table. Based on the priority order of standardized subtasks, the storage paths of the corresponding subtask data blocks are obtained sequentially from the mapping table. The corresponding subtask data blocks are read according to the storage paths, and the read subtask data blocks are sent to the remote management terminal through the network transmission channel. At the same time, the execution status of the standardized subtask corresponding to the sent subtask data blocks is marked as sent. After sending, listen for and receive the collaborative opinion data returned by the remote management terminal for the sent subtask data block. The collaborative opinion data includes the collaborative opinion content, remote reviewer information, and review timestamp. The received collaborative opinion data is bound to the subtask identifier of the standardized subtask corresponding to the sent subtask data block, and the bound collaborative opinion data is stored in the local database. The execution status of the corresponding standardized subtask is updated from the first execution status to the second execution status. The second execution status indicates that the multimodal detection data corresponding to the standardized subtask has been submitted to the remote management terminal and collaborative opinions have been received.

7. The highway field detection data multimodal feature fusion and remote collaborative management system according to claim 1, characterized in that, The instruction generation module extracts collaborative opinion data bound to the subtask identifier of the standardized subtask from the local database for each standardized subtask in the second execution state. The collaborative opinion content in the collaborative opinion data is identified by type, and the opinion type is determined to be either the first opinion type or the second opinion type. The first opinion type indicates that the multimodal detection data corresponding to the standardized sub-task has been approved and does not require supplementary collection, while the second opinion type indicates that the multimodal detection data corresponding to the standardized sub-task has defects and requires on-site supplementary collection.

8. The highway field detection data multimodal feature fusion and remote collaborative management system according to claim 7, characterized in that, When the opinion type of the collaborative opinion content is the first opinion type, a first on-site detection execution instruction containing task completion confirmation information is generated. When the opinion type of the collaborative opinion content is the second opinion type, based on the supplementary collection requirements specified in the collaborative opinion content, and combined with the detection unit identifier and location information of the detection unit to which the standardized subtask belongs, a second on-site detection execution instruction containing supplementary collection operation content is generated. The generated first or second field detection execution command will be sent to the field execution terminal via wireless communication. Receive instruction execution feedback information returned by the field execution terminal, and update the execution status of the standardized subtask to the third execution status based on the instruction execution feedback information. The third execution status indicates that the collaborative opinions corresponding to the standardized subtask have been processed and the task has been closed.

Images