Construction method of AI intelligent software evaluation index system for industrial control software

Abstract

The application discloses an AI intelligent software evaluation index system construction method for industrial control software, relates to the technical field of index system construction, and comprises the following steps: obtaining a semantic period according to log data of an industrial control task; grouping and determining a geometric relationship of a multi-period coordinate semantic record table corresponding to the semantic period to obtain a geometric relationship signature table; obtaining a target task combination based on the geometric relationship signature table, and obtaining an extended task record table based on the target task combination; calculating a success rate index based on the extended task record table, and generating an index table of a semantic evaluation unit based on the success rate index; and the application can provide stable and reliable data support for version selection, upgrade decision and operation optimization of the industrial control software.

Description

Technical Field

[0001] This invention relates to the field of indicator system construction technology, and in particular to a method for constructing an AI intelligent software evaluation indicator system for industrial control software. Background Technology

[0002] In the fields of industrial automation and intelligent manufacturing, industrial control software is widely used in robotic production lines, motion control systems, and complex manufacturing scenarios that collaborate with production execution systems. In actual production processes, industrial control software typically needs to run online for extended periods, continuously performing repetitive tasks such as grasping, placing, assembling, and inspecting. As production cycles change, equipment ages, and process optimization needs evolve, on-site maintenance personnel will modify geometric parameters such as hand-eye calibration, workpiece coordinate definitions, and tool coordinate settings multiple times at different times. However, the vast amount of log data generated by the industrial control system during operation often only records task identifiers, timestamps, and the executed coordinate values ​​within the current coordinate system.

[0003] Existing industrial control software evaluation methods typically rely directly on statistical analysis of long-term accumulated coordinate log data. This assumes that coordinate data recorded across different time periods are under the same spatial reference conditions, thus directly mixing task execution data from different coordinate definition stages. This ignores the frequent coordinate calibration and definition adjustments that occur in industrial settings, easily misinterpreting numerical differences caused by changes in coordinate semantics as software performance fluctuations. This leads to distorted evaluation results and negatively impacts the assessment of the quality of industrial control software versions. Current technologies address this issue by manually filtering log intervals or using simple time segmentation, lacking a method that automatically distinguishes different coordinate semantic conditions from the operational data itself and constructs evaluation indicators under a unified evaluation caliber. Therefore, it is difficult to obtain stable and reliable intelligent software evaluation results in complex, long-term industrial control scenarios. Summary of the Invention

[0004] The purpose of this invention is to address the shortcomings of existing technologies, which make it difficult to obtain stable and reliable intelligent software evaluation results in complex and long-term industrial control scenarios. Therefore, this invention proposes a method for constructing an AI intelligent software evaluation index system for industrial control software.

[0005] To address the problems existing in the prior art, the present invention adopts the following technical solution:

[0006] A methodology for constructing an evaluation index system for AI-powered intelligent software in industrial control software includes:

[0007] S1. Obtain semantic time periods based on log data of industrial control tasks;

[0008] S2. Group and determine the geometric relationships of the multi-time period coordinate semantic record table corresponding to the semantic time period to obtain the geometric relationship signature table;

[0009] S3. Obtain the target task combination based on the geometric relationship signature table, and obtain the extended task record table based on the target task combination;

[0010] S4. Calculate the success rate index based on the extended task record table, and generate an index table for semantic evaluation units based on the success rate index.

[0011] S5. Calculate the comprehensive evaluation index based on the index table, and generate the AI ​​intelligent software evaluation result table based on the comprehensive evaluation index.

[0012] Secondly, this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the AI ​​intelligent software evaluation index system construction method for industrial control software described in any of the above-mentioned claims.

[0013] Compared with the prior art, the beneficial effects of the present invention are:

[0014] 1. This invention introduces a semantic time period segmentation mechanism and segments long-term operation data based on time information in industrial control task logs. This makes task data generated under different calibration states and coordinate definition conditions clearly distinguishable, avoiding the problem of directly mixing and statistically analyzing data from different geometric semantic conditions. Combined with the construction methods of task record sets, task groups, and geometric relationship signatures, it can characterize stable spatial relative relationships between tasks without relying on absolute coordinate values. Thus, in complex industrial sites with frequently adjusted coordinate configurations, it can still accurately identify which tasks correspond to the same actual operation object in different time periods.

[0015] 2. By comparing task groups under different semantic time periods based on geometric relation signatures, target task groups with the same physical semantics are formed, and a unified semantic evaluation unit is assigned to them. This realizes centralized management and unified evaluation of the execution results of the same actual task in different time periods, avoiding the problem that execution deviations caused by changes in coordinate definition are misjudged as changes in software performance. This ensures that evaluation results such as success rate and execution stability can truly reflect the actual operating performance of industrial control software on the same task object.

[0016] 3. Furthermore, this invention transforms the execution results scattered across a large number of operation logs into a well-structured and hierarchical evaluation index system by constructing an index table for semantic evaluation units and a version evaluation set. Combined with an artificial intelligence evaluation model, it analyzes and outputs the comprehensive operation level of different software versions, which can automatically transform complex industrial control operation data into intuitive and quantifiable evaluation results, providing stable and reliable data support for industrial control software version selection, upgrade decisions, and operation optimization. Attached Figure Description

[0017] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, illustrate exemplary embodiments of the invention and, together with their description, serve to explain the invention and do not constitute an undue limitation thereof. In the drawings:

[0018] Figure 1 This is a flowchart illustrating a method for constructing an AI intelligent software evaluation index system for industrial control software, as provided in an embodiment of the present invention. Detailed Implementation

[0019] The technical solutions of the present invention will be clearly and completely described below with reference to the accompanying drawings of the embodiments of the present invention. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments.

[0020] Example: This example provides a method for constructing an AI intelligent software evaluation index system for industrial control software. See [link to example]. Figure 1 Specifically, including:

[0021] S1. Obtain semantic time periods based on log data of industrial control tasks;

[0022] In an embodiment of the present invention, the specific steps for obtaining the semantic time period are as follows:

[0023] Acquire log data for industrial control tasks;

[0024] It should be noted that log data for industrial control tasks refers to the set of data automatically generated and recorded by controllers, robots, or related execution units during the operation of industrial control systems. This data reflects the status and results of industrial control tasks during actual execution.

[0025] Specifically, data is collected through the data interfaces of key execution units and related systems in the industrial control system. The collected data includes control command issuance records, execution response data, and status feedback information generated during the operation of the robot controller; logic program execution data and signal interaction data recorded during the PLC cycle execution; algorithm input parameters, output result data, and execution time information generated when the edge AI module performs visual recognition, target detection, and pose estimation tasks; work order flow records, work step execution details, and production cycle data directly related to industrial control tasks in the production execution system; raw image data and accompanying shooting parameter information collected by industrial cameras; operating status data and position feedback data fed back by servo drives; and task execution result judgment data recorded by quality inspection equipment. All collected data must ensure continuity in time, integrity of data content, and correlation between data from various sources. During the collection process, a reading protocol compatible with the data format of each system is used to ensure the accuracy and security of data transmission. After the collection is completed, the data undergoes preliminary formatting to remove obviously invalid and redundant data, forming an industrial control task log data set that can be directly used for subsequent generation of raw task records.

[0026] The original task record is generated based on the log data; the original task record includes a record identifier, a task identifier, a timestamp, coordinate data, and a result label;

[0027] It should be noted that the original task record is a structured record unit formed based on the log data. The record identifier is used to distinguish different task records, the task identifier is used to indicate the specific industrial control task to which the record belongs, the timestamp is used to characterize the execution time corresponding to the task record, the coordinate data is used to describe the spatial position of the target position or execution position in the control coordinate system during the task execution, and the result label is used to characterize the result status after the task is executed.

[0028] Specifically, the collected industrial control task log data undergoes comprehensive analysis and structured processing. Relevant information uniquely identifying each independent execution action is extracted from the log data. A globally unique record identifier is generated using data coding rules to ensure that each execution action corresponds to a unique record throughout the entire dataset. Next, relevant information representing specific business operations is extracted from the log data and integrated to form task identifiers, enabling each record to accurately correspond to a specific industrial control task. Then, the actual execution time of the task recorded in the log data is extracted and standardized according to a unified time format standard to generate a timestamp that accurately reflects the task execution sequence. Furthermore, spatial location data related to the target position or actual execution position within the control coordinate system during task execution is extracted from the log data and organized into standardized coordinate data according to coordinate data storage specifications. Finally, the result status information after task completion is extracted from the log data and categorized according to unified result judgment rules to generate result labels that clearly reflect the task execution effectiveness. Through this series of data extraction, integration, and standardization processes, various key information scattered throughout the log data is systematically integrated to form a structured original task record containing record identifiers, task identifiers, timestamps, coordinate data, and result labels.

[0029] Build a timeline based on timestamps;

[0030] It should be noted that the timeline is a continuous time series constructed in chronological order based on the timestamps of multiple task records, used to describe the overall distribution of industrial control tasks over time.

[0031] Specifically, all timestamp information is extracted from the original task records. The extracted timestamps are deduplicated to remove duplicate time data. The remaining timestamps are then sorted and organized according to their chronological order. Using the sorted timestamps as the core nodes, a time series that can continuously cover the execution period of all industrial control tasks is constructed. This time series is the time axis, which can fully present the distribution of industrial control tasks in the time dimension, clearly reflect the order and time interval of different tasks, and provide a coherent and orderly time reference framework for subsequent semantic time segmentation based on maintenance record data.

[0032] The timeline is divided into multiple semantic time periods.

[0033] It should be noted that a semantic period refers to a time interval obtained by dividing the time axis according to the time points corresponding to the operation and maintenance record data. Each semantic period corresponds to a range of industrial control task execution time under the condition that the system configuration and coordinate semantics remain relatively consistent.

[0034] Specifically, maintenance record data related to industrial control tasks is collected through the industrial control system's operation and maintenance management platform, equipment maintenance record system, and related operation and maintenance terminals. The collection scope covers key operation records involving geometric semantic configuration changes during industrial control processes, including at least one of the following: hand-eye calibration operation time, tool coordinate adjustment time, and workpiece coordinate definition change time. Auxiliary maintenance information related to geometric semantic adjustments can also be collected. During the collection process, the time accuracy, operational relevance, and data integrity of each maintenance record must be verified to ensure that the collected maintenance record data accurately reflects the time of recalibration of the spatial relationship between the camera and the actuator, the time when the tool reference position is modified, the time when the workpiece reference coordinate system changes, and other related system adjustments and configuration changes. After collection, the maintenance record data is organized according to a unified format, and invalid and duplicate records are removed to form a standardized maintenance record dataset.

[0035] Specifically, all time points related to geometric semantic configuration changes are extracted from the operation and maintenance record data, including at least one of the following: hand-eye calibration operation time, tool coordinate adjustment time, and workpiece coordinate definition change time. These time points are used as dividing nodes on the time axis. The dividing nodes are sorted according to their chronological order. The constructed time axis is then segmented based on the sorted dividing nodes, forming an independent time interval between two adjacent dividing nodes. If there is an industrial control task execution time range between the start of the time axis and the first dividing node, this range is taken as the first semantic period. If there is an industrial control task execution time range between the last dividing node and the end of the time axis, this range is taken as the last semantic period. Multiple semantic periods are obtained through the above division. Within each semantic period, the system configuration and coordinate semantics remain relatively consistent, which can accurately correspond to the execution time range of industrial control tasks under the same geometric semantic conditions.

[0036] S2. Group and determine the geometric relationships of the multi-time period coordinate semantic record table corresponding to the semantic time period to obtain the geometric relationship signature table;

[0037] In an embodiment of the present invention, the specific steps for obtaining the geometric relationship signature table are as follows:

[0038] Generate a multi-time period coordinate semantic record table based on semantic time periods;

[0039] It should be noted that the multi-time period coordinate semantic record table refers to a data table formed by uniformly recording the coordinate data generated during the execution of industrial control tasks in different semantic time periods. It is used to reflect the coordinate performance of the same task in different time intervals.

[0040] Specifically, the process begins by summarizing all the semantic time periods that have been divided, clarifying the time range and corresponding semantic time period identifier for each semantic time period. Then, all field information from the previously generated original task records is extracted, including record identifier, task identifier, timestamp, coordinate data, and result label. Subsequently, the timestamp of each original task record is checked one by one to determine the semantic time period in which the timestamp is located. Then, the corresponding semantic time period identifier is associated with it, and the semantic time period identifier is added as a new field to the original task record. Following a unified data table structure specification, the record identifier, task identifier, semantic time period identifier, timestamp, coordinate data, and result label of each record are sequentially filled into the preset data table framework. At the same time, the integrity of the filled data is checked to ensure that each field of each record is complete, the data format is uniform, and the logic is consistent. Finally, a multi-time period coordinate semantic record table is formed that can systematically present the relevant information of the execution of various industrial control tasks within different semantic time periods.

[0041] Group the multi-time period coordinate semantic record table to obtain the task record set;

[0042] It should be noted that the task record set refers to the data set formed by aggregating the records in the multi-time period coordinate semantic record table according to the task identifier, and is used to describe multiple records of the same industrial control task at multiple execution times.

[0043] Specifically, all records in the multi-time period coordinate semantic record table are extracted, and all records are classified and aggregated based on the task identifier. All records with the same task identifier are integrated into an independent data set. Each such independent data set is a task record set. Each task record set can fully present all relevant records of the same industrial control task at different execution times and different semantic periods, clearly reflecting the multiple executions of the task and the corresponding coordinate data, result labels and other information.

[0044] The task record set is divided into task groups;

[0045] It should be noted that a task group refers to a set of records formed by further dividing multiple task records with spatial or business relationships within the same semantic time period. It is used to reflect the overall execution status of a group of related tasks within that time interval.

[0046] Specifically, for each semantic time period, the spatial and business association attributes of the industrial control tasks corresponding to each task record are sorted out by combining the business knowledge of the industrial control scenario and the actual configuration of the production line. The spatial association attributes include the physical location association and coordinate system association of the task execution, while the business association attributes include the process flow association, production batch association, and functional target association of the task. Based on these association attributes, the task records in the task record set are classified and divided. Multiple task records with spatial or business association are integrated into an independent record set. Each independent record set is a task group. Each task group can centrally reflect the overall execution status of a group of related tasks within the corresponding semantic time period.

[0047] Determine the geometric relationship signature based on the coordinate data of each task record in the task group;

[0048] It should be noted that geometric relationship signature refers to the spatial positional relationship description determined based on the coordinate data of each task record in the task group, which is used to characterize the relative relationship between tasks in spatial distribution.

[0049] Specifically, the coordinate data of all task records in each task group is extracted. First, the coordinate data within the task group is normalized to eliminate interference from coordinate values ​​of different magnitudes. Then, the relative distance, direction angle, and position offset between any two task record coordinate data within the task group are calculated. At the same time, the spatial distribution pattern and topological relationship between multiple task record coordinate data are analyzed. This information, including relative distance, direction angle, position offset, and spatial topological relationship, is quantified and integrated to form a structured descriptive information that can uniquely represent the spatial relative relationship between tasks within the task group. This structured descriptive information is the geometric relationship signature, which can accurately reflect the inherent relative relationship of each task in the spatial distribution within the task group without relying on absolute coordinate semantics.

[0050] A geometric relation signature table is generated based on the semantic time period identifier, the task group identifier, and the geometric relation signature.

[0051] It should be noted that the geometric relationship signature table is a data table formed by using the identifier of the semantic time period, the identifier of the task group, and the corresponding geometric relationship signature as its content. It is used to centrally record the spatial relationship of each task group under different semantic time periods.

[0052] Specifically, the process begins by clearly defining the identifiers for each semantic time period, the identifiers for each task group, and the corresponding geometric relationship signatures for each task group. A standardized data table framework is then constructed, including fields for semantic time period identifiers, task group identifiers, and geometric relationship signatures. Next, the identifier of each task group under each semantic time period is matched with the defined geometric relationship signature for that task group, ensuring that each task group identifier uniquely corresponds to a geometric relationship signature under its respective semantic time period identifier. Subsequently, these matched semantic time period identifiers, task group identifiers, and geometric relationship signatures are sequentially filled into the corresponding fields of the data table framework. At the same time, the filled data is validated to ensure that there are no missing fields, incorrect data associations, or duplicate records. Finally, a geometric relationship signature table is formed that can centrally record the spatial relationships of each task group under different semantic time periods.

[0053] It should be noted that during the long-term operation of an industrial control system, the same production line and the same type of task will undergo multiple calibrations and coordinate adjustments. Although the logs continuously record coordinate data under the same task group, the coordinate system on which this coordinate data depends has changed over time. Simply looking at the coordinate values ​​cannot determine whether they still represent the same spatial positional relationship. By introducing semantic period identifiers, we can distinguish the time range within which coordinate semantics remain consistent. By introducing task group identifiers, we can limit the range of interrelated task records within the same business scenario. And through geometric relationship signatures, we can describe the stable spatial relative positional relationship between tasks within the task group. Combining semantic period identifiers, task group identifiers, and geometric relationship signatures to generate a geometric relationship signature table can transform task records scattered across different time periods and coordinate definitions into a structured description centered on spatial relationships. This provides a direct basis for subsequent judgments on whether tasks in different time periods belong to the same type of task in a spatial sense, avoiding misjudgments caused by analysis based solely on coordinate values.

[0054] S3. Obtain the target task combination based on the geometric relationship signature table, and obtain the extended task record table based on the target task combination;

[0055] In embodiments of the present invention, obtaining the target task combination based on the geometric relation signature table includes:

[0056] The geometric relation signatures corresponding to identifiers with the same task group but different semantic time periods are defined as the target relation signatures;

[0057] It should be noted that the task group identifier is a data marker used to distinguish different task groups. Each task group corresponds to a set of interconnected industrial control task records within the same business scenario and time range. The semantic period identifier is a data marker used to distinguish different time intervals. Each semantic period corresponds to a range of operating time in an industrial control system where coordinate definitions and system configurations remain consistent. The geometric relationship signature is a spatial relationship description formed based on the coordinate data of each task record in the task group, used to reflect the overall structure of the relative positional relationships between tasks within the task group. The target relationship signature refers to the data set formed by aggregating the corresponding geometric relationship signatures as the same comparison object when they have the same task group identifier and belong to different semantic periods.

[0058] Specifically, the core meanings of task group identifiers, semantic time period identifiers, and geometric relationship signatures are first clarified. The task group identifier is used to distinguish different task groups. Each task group corresponds to industrial control task records that are related to each other within the same business scenario and time range. The semantic time period identifier is used to distinguish different time intervals. Each semantic time period corresponds to the operating time range in the industrial control system where the coordinate definition and system configuration are consistent. The geometric relationship signature is a spatial relationship description formed based on the coordinate data of each task record in the task group. It is used to reflect the overall structure of the relative positional relationship between each task within the task group. Then, all records in the geometric relationship signature table are traversed and classified according to the task group identifier. All geometric relationship signatures belonging to different semantic time period identifiers under the same task group identifier are collected. This collected data set is defined as the target relationship signature. It is ensured that each target relationship signature corresponds to a unique task group identifier and contains all geometric relationship signatures of the task group under different semantic time periods. This provides a centralized data object for subsequent identification of task combinations with the same physical semantics through signature comparison.

[0059] By comparing the target relation signatures based on the geometric relation signature table, a combination of target tasks with the same physical semantics is obtained.

[0060] It should be noted that the geometric relationship signature table is a data table used to centrally store the geometric relationship signatures corresponding to each task group under different semantic time periods; the target task group refers to a set of task groups that are determined to be consistent in spatial relationship after comparing the target relationship signatures in the geometric relationship signature table. This set represents the set of task groups that correspond to the same actual task layout in different time intervals.

[0061] Specifically, firstly, all data in the geometric relationship signature table is extracted to clarify the task group identifier, semantic time period identifier, and corresponding geometric relationship signature content for each target relationship signature. Then, a unified comparison rule is established covering relative distance consistency verification, direction angle matching degree determination, and spatial topological relationship logic verification. The relative distance consistency verification is achieved by calculating the deviation value of the relative distance between corresponding tasks in the geometric relationship signatures of different semantic time periods. The direction angle matching degree determination is completed by calculating the difference of direction angle between corresponding tasks. The spatial topological relationship logic verification is carried out by comparing whether the spatial structure logic of the tasks within the task group is consistent. According to the comparison rule, the target relationship signatures of different semantic time periods under the same task group identifier are comprehensively compared one by one. If the relative distance deviation and direction angle difference in the comparison results are within the preset reasonable range, and the spatial topological relationship logic is consistent, then the task groups corresponding to these semantic time periods are determined to be the same in physical semantics. These task groups with the same physical semantics are collected and integrated to form a target task combination with the same physical semantics, ensuring that the task groups in this combination all correspond to the same actual task layout, providing accurate data support for the subsequent allocation of semantic evaluation units.

[0062] It should be noted that within the same production line and the same type of industrial control task, the system may adjust hand-eye calibration, workpiece coordinate definition, or tool coordinate settings multiple times at different time periods. Although the task identifier and coordinate fields recorded in the log remain unchanged, the spatial reference relationships corresponding to these records have changed. It is impossible to confirm whether tasks at different time periods still correspond to the same actual operation object based solely on the task identifier or coordinate values. It is necessary to compare the spatial relationships of tasks with the same task group identifier under different semantic time periods. Task groups that maintain consistency in actual spatial layout and operation objects will be grouped into target task groups with the same physical semantics. This allows subsequent evaluation and analysis to be conducted within the same actual task object, thereby avoiding the inclusion of differences caused by changes in coordinate semantics into task or software performance analysis. This ensures that the evaluation results reflect the changes at the real task level rather than the impact of coordinate definition adjustments.

[0063] In an embodiment of the present invention, an extended task record table is obtained based on the combination of target tasks, including:

[0064] Assign semantic evaluation units to the target task combination;

[0065] It should be noted that a target task combination refers to a set of tasks that correspond to the same actual operation object and the same type of industrial control behavior after spatial relationship comparison in different semantic time periods. This task combination reflects the real task entity that does not change with the coordinate definition in the actual production process. The semantic evaluation unit is a unified evaluation identifier assigned to the target task combination, which is used to represent the smallest evaluation unit for centralized evaluation of the execution of the target task combination in various semantic time periods.

[0066] Specifically, firstly, a comprehensive review of all target task combinations confirmed through spatial relationship comparison is conducted. Then, the different semantic time periods, corresponding task groups, and all associated industrial control tasks covered by each target task combination are clearly defined to ensure that each target task combination accurately corresponds to the same actual operation object and the same type of industrial control behavior. Subsequently, a unified and unique semantic evaluation unit allocation mechanism is established. This mechanism ensures that the semantic evaluation units allocated to different target task combinations are not duplicated. According to this mechanism, a unique evaluation identifier is assigned to each independent target task combination. This evaluation identifier is the semantic evaluation unit. During the allocation process, it is necessary to ensure that each semantic evaluation unit forms a unique association with the corresponding target task combination, so that the semantic evaluation unit becomes the smallest evaluation unit for centralized evaluation of the execution of the target task combination in various semantic time periods.

[0067] Based on the correspondence between the identifiers of semantic time periods, task groups, and semantic evaluation units, a semantic evaluation unit mapping table is generated.

[0068] It should be noted that the semantic evaluation unit mapping table is a data table used to record the correspondence between the identifiers of semantic time periods, task groups, and semantic evaluation units, and is used to clarify which evaluation unit should be assigned to different time intervals and different task groups.

[0069] Specifically, the process begins by comprehensively collecting all identified semantic time period identifiers, task group identifiers, and semantic evaluation unit identifiers assigned to the target task combination. Then, the identifiers of the task groups included under each semantic time period identifier are meticulously analyzed, along with the relationships between each task group identifier and its corresponding semantic evaluation unit identifier. This ensures the accuracy and completeness of the correspondence between the three. Next, a standardized data table framework is constructed, including semantic time period identifier fields, task group identifier fields, and semantic evaluation unit identifier fields. Following the established correspondence, the identifiers of each semantic time period, the corresponding task group identifiers, and the associated semantic evaluation unit identifiers are sequentially filled into the corresponding fields of the data table framework. During the filling process, continuous data validation is performed to check for inconsistencies such as the same semantic time period identifier and task group identifier combination corresponding to multiple semantic evaluation unit identifiers, or the same semantic evaluation unit identifier being associated with unrelated semantic time period identifiers and task group identifiers. Problems discovered during validation are promptly corrected, ultimately forming a semantic evaluation unit mapping table that clearly and accurately records the correspondence between the three.

[0070] The semantic evaluation units are written into the multi-time period coordinate semantic record table according to the semantic evaluation unit mapping table to obtain the extended task record table.

[0071] It should be noted that the extended task record table refers to a data table formed by writing the corresponding semantic evaluation unit identifier into the semantic evaluation unit mapping table on the basis of the multi-time period coordinate semantic record table. This table is used to support subsequent statistical analysis and evaluation calculation of industrial control tasks under a unified evaluation unit.

[0072] Specifically, the generated multi-time period coordinate semantic record table and semantic evaluation unit mapping table are retrieved in their entirety first. The record identifier, task identifier, semantic period identifier, timestamp, coordinate data, and result label fields for each record in the multi-time period coordinate semantic record table are clarified. Simultaneously, the precise correspondence between the semantic period identifier, task group identifier, and semantic evaluation unit identifier in the semantic evaluation unit mapping table is defined. Then, using the semantic period identifier and task group identifier as joint matching keywords, each record in the multi-time period coordinate semantic record table is associated with its corresponding entry in the semantic evaluation unit mapping table to determine the unique semantic evaluation unit identifier for each record. This semantic evaluation unit identifier is then added as a new field to the corresponding record in the multi-time period coordinate semantic record table. After addition, a comprehensive data validation is performed on the expanded data table to verify whether each record has been successfully associated with a semantic evaluation unit identifier, whether there are any mismatched or missing identifiers, whether the data format of each field is consistent, and whether the logic is coherent, ensuring no data anomalies. Finally, an expanded task record table containing record identifier, task identifier, semantic period identifier, semantic evaluation unit identifier, timestamp, coordinate data, and result label is formed.

[0073] S4. Calculate the success rate index based on the extended task record table, and generate an index table for semantic evaluation units based on the success rate index.

[0074] In an embodiment of the present invention, the success rate metric is calculated based on the extended task record table, including:

[0075] The extended task record table is grouped to obtain multiple semantic evaluation record sets;

[0076] It should be noted that the semantic evaluation record set refers to the data set formed by grouping the task records in the extended task record table according to the semantic evaluation unit identifier. Each semantic evaluation record set corresponds to the multiple executions of the same actual task at different times.

[0077] Specifically, firstly, all task records in the extended task record table are extracted completely, and all field information of each record, including record identifier, task identifier, semantic time period identifier, semantic evaluation unit identifier, timestamp, coordinate data, and result label, is identified. Then, using the semantic evaluation unit identifier as the unique and core grouping criterion, all task records are classified and aggregated. All task records with the same semantic evaluation unit identifier are integrated into an independent and complete data set. Each such independent data set is a semantic evaluation unit record set. Each semantic evaluation unit record set can systematically present the multiple executions of the same actual task under different semantic time periods and execution times, clearly reflecting the various related information corresponding to the actual task. This provides a structured data source organized according to a unified actual task dimension for subsequent calculation of success rate indicators, dispersion indicators, and generation of semantic evaluation unit indicator tables.

[0078] Determine the number of successful records and the total number of records in the semantic evaluation record set based on the result labels;

[0079] It should be noted that the result label refers to the data used to characterize the execution result of the industrial control task corresponding to each task record, and is used to reflect whether the task has achieved the expected execution state; the number of successful records refers to the number of task records whose result labels indicate successful execution in a certain semantic evaluation record set, and the total number of records refers to the total number of all task records contained in the semantic evaluation record set.

[0080] Specifically, first, the result labels corresponding to all task records in each semantic evaluation record set are identified. These result labels are used to characterize the execution result of the industrial control task corresponding to each task record, reflecting whether the task has achieved the expected execution state. Then, all task records in each semantic evaluation record set are traversed one by one, and the task records whose result labels indicate successful execution are counted one by one. The total number of successful records in the semantic evaluation record set is obtained. At the same time, the total number of all task records in the semantic evaluation record set is counted to obtain the total number of records. During the counting process, it is necessary to ensure that all records are traversed without omission or duplicate counting, and the judgment criteria for result labels are always kept consistent, so as to provide accurate and reliable basic data support for subsequent calculation of success rate indicators.

[0081] The success rate is calculated by dividing the number of successful records by the total number of records.

[0082] It should be noted that the success rate index is data obtained by calculating the ratio of the number of successful records to the total number of records, and is used to reflect the overall success level of industrial control tasks under the corresponding semantic evaluation unit in the actual execution process.

[0083] In an embodiment of the present invention, the specific steps for generating the index table are as follows:

[0084] Calculate the average coordinates based on the coordinate data of each task record in the semantic evaluation record set;

[0085] It should be noted that each task record in the semantic evaluation record set corresponds to the execution process of an actual industrial control task; coordinate data refers to the spatial position data of the target position or execution position in the control coordinate system during the execution of the industrial control task; average coordinates refers to the representative spatial position obtained by summarizing the coordinate data of multiple task records within the same semantic evaluation record set, which is used to reflect the concentration of task execution positions under this semantic evaluation unit.

[0086] Specifically, first, we identify all the task records contained in each semantic evaluation record set. Each task record corresponds to the execution process of an actual industrial control task, and its coordinate data is used to describe the spatial position of the target position or execution position in the control coordinate system during the execution of the task. Then, we extract the coordinate data of all task records in each semantic evaluation record set, determine the dimensions of the coordinate data and the numerical range of each dimension, and use the arithmetic mean method to calculate all coordinate values ​​under the same dimension. That is, we first sum the coordinate values ​​of all task records under the dimension, and then divide the sum by the total number of task records in the semantic evaluation record set to obtain the average value of the dimension. The average value is combined with the average values ​​of each dimension to form the average coordinate corresponding to the semantic evaluation record set. The average coordinate can reflect the concentration of task execution positions under the semantic evaluation unit.

[0087] Calculate the average distance based on the average coordinates;

[0088] It should be noted that the mean distance refers to the average distance between the coordinate data of each task record and the average coordinate in the same semantic evaluation record set. It is used to reflect the overall deviation of the task execution position from the representative position.

[0089] Specifically, first, the average coordinates corresponding to each semantic evaluation record set are determined. These average coordinates are representative spatial positions obtained by summing the coordinate data of all task records within the set. Then, the coordinate data of each task record in the semantic evaluation record set is extracted. Using a unified distance calculation method, such as Euclidean distance or the average of the absolute values ​​of the differences in each dimension, the distance between the coordinate data of each task record and the average coordinates is calculated one by one to obtain the individual distance value corresponding to each record. Then, all individual distance values ​​are summed, and the summation result is divided by the total number of task records in the semantic evaluation record set to obtain the average distance. This average distance can reflect the overall deviation of the task execution position from the representative position.

[0090] The mean distance is used as the dispersion index of the semantic evaluation unit;

[0091] It should be noted that the dispersion index uses the mean distance as the evaluation result to describe the stability of multiple task executions in spatial location under the same semantic evaluation unit.

[0092] It should be noted that, under the same semantic evaluation unit, each task record corresponds to the spatial performance of the same actual task in multiple executions. Ideally, these tasks should repeat actions around a stable execution position. Therefore, the average coordinates can reflect the representative execution position under the semantic evaluation unit. The distance between the coordinates of each task record and the average coordinates reflects the degree of deviation of a single execution from the overall execution center. The average distance obtained by summarizing these deviations can comprehensively characterize the spatial dispersion of multiple task executions under the semantic evaluation unit. Therefore, using the average distance as the dispersion index of the semantic evaluation unit can intuitively reflect the spatial stability level of the task in the repeated execution process.

[0093] Based on the success rate index, dispersion index, and semantic evaluation unit identifier, a table of indexes for semantic evaluation units is generated.

[0094] It should be noted that the semantic evaluation unit index table refers to a data table formed by summarizing the corresponding success rate index and dispersion index using the semantic evaluation unit identifier as an index, and is used to characterize the comprehensive execution performance of each semantic evaluation unit.

[0095] Specifically, the process begins by comprehensively collecting all semantic evaluation unit identifiers, along with the corresponding success rate and dispersion metrics for each identifier. Then, a standardized data table framework is constructed, including semantic evaluation unit identifier fields, success rate metric fields, and dispersion metric fields. Each semantic evaluation unit identifier is precisely matched with its corresponding success rate and dispersion metrics to ensure that each identifier uniquely corresponds to a complete and accurate set of metric data, with no duplicate, missing, or incorrect matches. All matched data is then sequentially filled into the corresponding fields of the data table framework according to a unified format. During the filling process, continuous data validation is performed to check the completeness, logical consistency, and format standardization of the field data, promptly correcting any issues found. Finally, a metric table is formed, indexed by the semantic evaluation unit identifier, summarizing the corresponding success rate and dispersion metrics for each semantic evaluation unit.

[0096] It should be noted that in industrial control systems, the same type of task may be executed repeatedly at different times. However, due to multiple calibrations and coordinate adjustments, these execution records cannot be directly mixed together for statistical analysis; otherwise, the success rate and stability results obtained will not reflect the true situation. In the aforementioned processing, records from different time periods but corresponding to the same actual task have been uniformly grouped into the same semantic evaluation unit. At this point, it is still necessary to centrally organize the execution results and spatial performance of these scattered task records into a stable evaluation result. Therefore, by statistically analyzing the success rate of each semantic evaluation unit and calculating the dispersion of execution positions, these results are combined with the corresponding semantic evaluation unit identifiers to form an index table for the semantic evaluation unit. This transforms complex and scattered operation logs into a set of clear and directly comparable evaluation results, thereby providing a clear and reliable data foundation for subsequent software version evaluation, production line status analysis, and decision-making.

[0097] S5. Calculate the comprehensive evaluation index based on the index table, and generate the AI ​​intelligent software evaluation result table based on the comprehensive evaluation index.

[0098] In an embodiment of the present invention, the calculation of a comprehensive evaluation index based on an index table includes:

[0099] Obtain the version identifier of the industrial control software;

[0100] It should be noted that the version identifier of industrial control software is a data mark used to distinguish the software versions formed by the same industrial control software at different development stages or in different deployment states. Each version identifier corresponds to a specific software program and control logic.

[0101] Specifically, the version identifier of industrial control software is first defined to distinguish versions of the same software at different development stages or deployment states. Each identifier corresponds to a specific software program and control logic. Then, the possible sources of the version identifier are determined, including deployment configuration documents of the industrial control system, software installation and update logs, filing data in the version management system, upgrade records retained by the operation and maintenance team, and configuration files of operating parameters of AI modules. Version information related to the target industrial control software is comprehensively collected from these channels, including release version number, iteration update sequence number, deployment timestamp, and other content that can uniquely identify the software version. This ensures that each version identifier can clearly correspond to a specific software program and control logic, and finally, a standardized and accurate version identifier for the industrial control software is extracted.

[0102] Based on the version identifier and the identifier of the semantic evaluation unit, the indicator table is grouped to obtain multiple version evaluation sets;

[0103] It should be noted that a version evaluation set refers to a data set formed by aggregating the indicator records corresponding to multiple semantic evaluation units under the same software version identifier, which is used to describe the overall performance of the software version on multiple actual tasks.

[0104] Specifically, the process begins by fully retrieving the generated semantic evaluation unit indicator table and the acquired industrial control software version identifier. This process clarifies the core data fields in the indicator table, such as the semantic evaluation unit identifier, success rate indicator, and dispersion indicator. Simultaneously, it confirms the software version information corresponding to the version identifier and the time coverage of each version. Then, using the version identifier and semantic evaluation unit identifier as the joint grouping basis, a systematic classification and aggregation process is performed on all indicator records in the indicator table. Indicator records corresponding to different semantic evaluation units under the same version identifier are integrated into an independent and complete data set. Each such independent data set is a version evaluation set. Each version evaluation set can comprehensively present the overall performance of the corresponding software version in multiple actual tasks, clearly reflecting the success rate and stability of the version under different semantic evaluation units. During the aggregation process, it is necessary to ensure no record omissions, no cross-version mismatches, and no duplicate groupings, ultimately forming multiple version evaluation sets with a well-structured and accurate data.

[0105] The comprehensive evaluation index is calculated based on the success rate index and dispersion index of each semantic evaluation unit in the version evaluation set.

[0106] It should be noted that the comprehensive evaluation index is the data obtained by summarizing and calculating the success rate index and dispersion index of multiple semantic evaluation units in the same version evaluation set. It is used to characterize the overall operating level of the industrial control software version in the actual production environment.

[0107] Specifically, the success rate index is normalized to provide a unified scale for direct comparison among different semantic evaluation units. Simultaneously, the dispersion index is reverse-normalized to ensure that lower dispersion corresponds to higher evaluation results, thus guaranteeing consistency in numerical values ​​across different types of indicators. The normalized success rate index and the normalized dispersion index are combined one by one to form a unit evaluation value for each semantic evaluation unit. This unit evaluation value comprehensively reflects the reliability and spatial stability of the task execution results under that semantic evaluation unit. The unit evaluation values ​​of all semantic evaluation units in the version evaluation set are aggregated and calculated, and the aggregated results are averaged according to the number of semantic evaluation units to obtain a comprehensive evaluation index characterizing the overall operational level of the version evaluation set. This comprehensive evaluation index is then associated with the corresponding software version identifier to reflect the overall execution performance of the industrial control software version in a real-world production environment based on multiple actual tasks.

[0108] Specifically, the process of combining the normalized success rate index and the normalized dispersion index one by one to form the unit evaluation value corresponding to each semantic evaluation unit is as follows: For any semantic evaluation unit, let its normalized success rate index be Rs, with a value ranging from zero to one, used to represent the reliability of the task execution result under this semantic evaluation unit; let its normalized dispersion index be Rd, with a value ranging from zero to one, used to represent the dispersion of the task execution position under this semantic evaluation unit, where a larger value of Rd indicates a higher degree of dispersion; firstly, the dispersion index is processed in reverse to obtain the stability index. Ss, calculated as one minus Rd, represents the stability of the task execution location. The normalized success rate index Rs and the stability index Ss are multiplied to obtain the unit evaluation value U corresponding to the semantic evaluation unit, where U equals Rs multiplied by Ss. Through this calculation method, when the semantic evaluation unit has both a high success rate and high spatial stability, its unit evaluation value is correspondingly high. Conversely, when either index is low, the unit evaluation value decreases accordingly. This allows the unit evaluation value to comprehensively reflect the execution reliability and spatial stability of the semantic evaluation unit on the same numerical scale.

[0109] In an embodiment of the present invention, the specific steps for generating the AI ​​intelligent software evaluation result table are as follows:

[0110] The evaluation input data is constructed based on version identifiers and comprehensive evaluation indicators;

[0111] It should be noted that the evaluation input data refers to the data set formed by combining the version identifier with the corresponding comprehensive evaluation index, which is used as the input content for subsequent intelligent evaluation processing.

[0112] Specifically, the process begins by comprehensively collecting all version identifiers of the acquired industrial control software, along with the comprehensive evaluation indicators corresponding to each version identifier. Then, a unified data structure specification for the evaluation input data is defined to ensure that the data format meets the input requirements of the subsequent AI evaluation model. Next, each version identifier is precisely associated with its corresponding comprehensive evaluation indicator, ensuring that each version identifier uniquely corresponds to a complete and accurate set of comprehensive evaluation indicator data. During the combination process, the established data structure specifications are strictly followed to maintain the consistency and standardization of all data formats, ultimately forming evaluation input data with a clear structure, accurate data, and standardized format.

[0113] The evaluation data is input into the trained AI evaluation model to obtain the evaluation results.

[0114] It should be noted that the AI ​​evaluation model refers to a computational model formed by training on historical industrial control software operation data. This model can output corresponding software evaluation results based on the input evaluation data. The evaluation result refers to the result data obtained by the AI ​​evaluation model after processing the evaluation input data, which is used to characterize the performance level of the corresponding software version.

[0115] Specifically, the process begins by comprehensively collecting historical data from different software versions and runtime scenarios within industrial control environments. This data includes success rate metrics, dispersion metrics, corresponding version identifiers, and actual performance labels for each semantic evaluation unit. The actual performance labels must accurately reflect the software's performance in a real production environment. The collected historical data is then preprocessed, including removing missing data, correcting outliers, and standardizing the data format. The preprocessed data is then divided into training, validation, and test sets according to a reasonable ratio, ensuring that these three datasets comprehensively cover different runtime scenarios and software versions. Next, the core architecture of the AI ​​evaluation model is determined to be a neural network. The model's input is defined as the comprehensive evaluation index data corresponding to the version identifier, and the output is the evaluation result representing the software's performance level. Simultaneously, the model's hyperparameters are set, including the learning rate, loss function, optimizer, and iteration count. The training process involves iteratively training a neural network model. The learning rate is dynamically adjusted based on the model's convergence. After each training iteration, validation set data is used to evaluate the model's output against the actual performance labels. Based on the evaluation results, hyperparameters such as the learning rate and optimizer parameters are adjusted to continuously optimize the model and reduce overfitting or underfitting. Once the model has reached the preset number of iterations, the generalization ability is comprehensively validated using test set data. If the model's accuracy, error, and other core evaluation metrics meet the practical application requirements of industrial control scenarios, the model training is complete. If not, the process returns to the data preprocessing stage or model architecture adjustment stage for re-optimization. Finally, a trained AI evaluation model with satisfactory performance is obtained. This model can accurately receive evaluation input data and output reliable software evaluation results.

[0116] Specifically, the trained AI evaluation model is formed using historical industrial control software operation data. It possesses the ability to output corresponding software evaluation results based on input evaluation data. It is necessary to confirm the input data format, data transmission protocol, and interface call specifications supported by the model to ensure complete compatibility with the constructed evaluation input data. Subsequently, preprocessing operations are performed on the evaluation input data, including data format conversion, outlier removal, and data integrity verification, ensuring the input data meets the model's acceptance standards. Then, according to the preset interface specifications and data transmission process, the preprocessed evaluation input data is transmitted completely and accurately to the AI ​​evaluation model. During transmission, the data transmission status is monitored in real time to avoid data loss, transmission interruptions, or format errors. After the model successfully receives the data, its computational processing flow is initiated. The model will perform in-depth analysis and calculation of the comprehensive evaluation indicators corresponding to the version identifier in the evaluation input data based on its built-in algorithm logic, generating evaluation result data that characterizes the performance level of the corresponding software version. Finally, the complete evaluation results are extracted from the model output, and the completeness, rationality, and format specifications of the result data are verified to ensure that the result data meets the expected evaluation dimensions and output requirements, providing direct and reliable core data support for the subsequent generation of the AI ​​intelligent software evaluation result table.

[0117] An AI intelligent software evaluation result table is generated based on the evaluation results.

[0118] It should be noted that the AI ​​intelligent software evaluation result table is a data table that summarizes the evaluation results of various software versions. It is used to centrally display the evaluation conclusions of different industrial control software versions in actual application scenarios.

[0119] Specifically, the core purpose of the AI-powered software evaluation results table is to centrally display the evaluation results of different industrial control software versions in practical application scenarios, providing a basis for comparing old and new versions, evaluating supplier performance, and making production management decisions. Then, it comprehensively collects the version identifiers of all industrial control software, along with the evaluation results corresponding to each version identifier. These evaluation results are obtained by processing the evaluation input data using a trained AI evaluation model, objectively representing the success rate, stability, and overall operational level of the corresponding software version at the semantic evaluation unit level. Next, a standardized data table framework is constructed, including version identifier fields and evaluation result fields. Furthermore, semantic evaluation unit identifiers and success rates can be added according to the business needs of industrial control scenarios. Auxiliary fields such as summary index value and dispersion index summary value accurately match each version identifier with its corresponding evaluation results and related auxiliary data, ensuring that each version identifier uniquely corresponds to a complete and accurate set of evaluation data without mismatches, omissions, or duplications. Then, all the matched data are sequentially filled into the corresponding fields of the data table framework according to a unified format. During the filling process, the standardization of data format and the consistency of logic are strictly verified, and abnormal issues such as missing data and disordered format are corrected in a timely manner. Finally, an AI intelligent software evaluation result table with a clear structure, accurate data, comprehensive dimensions, and easy access and analysis is formed, providing intuitive and reliable data support for version selection, performance optimization, and production management decisions of industrial control software.

[0120] Secondly, this application provides a computer-readable storage medium storing instructions that, when executed on a computer, cause the computer to perform the AI ​​intelligent software evaluation index system construction method for industrial control software described in any of the above-mentioned claims.

[0121] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. A method for constructing an AI intelligent software evaluation index system for industrial control software, characterized in that, Includes the following steps: S1. Obtain semantic time periods based on log data of industrial control tasks; The specific steps to obtain the semantic time period are as follows: Acquire log data for industrial control tasks; The original task record is generated based on the log data; the original task record includes a record identifier, a task identifier, a timestamp, coordinate data, and a result label; Build a timeline based on timestamps; The timeline is divided into multiple semantic time periods; Among them, semantic time period refers to the time interval obtained by dividing the time axis according to the time point corresponding to the operation and maintenance record data; S2. Group and determine the geometric relationships of the multi-time period coordinate semantic record table corresponding to the semantic time period to obtain the geometric relationship signature table; The specific steps to obtain the geometric relation signature table are as follows: Generate a multi-time period coordinate semantic record table based on semantic time periods; Group the multi-time period coordinate semantic record table to obtain the task record set; The task record set is divided into task groups; Determine the geometric relationship signature based on the coordinate data of each task record in the task group; A geometric relationship signature table is generated based on the semantic time period identifier, the task group identifier, and the geometric relationship signature. Among them, the geometric relationship signature table refers to a data table formed by the identifier of the semantic time period, the identifier of the task group and the corresponding geometric relationship signature, which is used to centrally record the spatial relationship of each task group under different semantic time periods. S3. Obtain the target task combination based on the geometric relationship signature table, and obtain the extended task record table based on the target task combination; The specific steps to obtain the target task combination are as follows: The geometric relation signatures corresponding to identifiers with the same task group but different semantic time periods are defined as the target relation signatures; By comparing the target relation signatures based on the geometric relation signature table, a combination of target tasks with the same physical semantics is obtained; Among them, the target task combination refers to a set of task groups that are determined to be consistent in spatial relationship after comparing the target relationship signatures in the geometric relationship signature table. It represents a set of task groups that correspond to the same actual task layout in different time intervals. S4. Calculate the success rate index based on the extended task record table, and generate an index table for semantic evaluation units based on the success rate index. S5. Calculate the comprehensive evaluation index based on the index table, and generate the AI ​​intelligent software evaluation result table based on the comprehensive evaluation index.

2. The method for constructing an AI intelligent software evaluation index system for industrial control software according to claim 1, characterized in that, An extended task record table is obtained based on the combination of target tasks, including: Assign semantic evaluation units to the target task combination; Based on the correspondence between the identifiers of semantic time periods, task groups, and semantic evaluation units, a semantic evaluation unit mapping table is generated. The semantic evaluation units are written into the multi-time period coordinate semantic record table according to the semantic evaluation unit mapping table to obtain the extended task record table.

3. The method for constructing an AI intelligent software evaluation index system for industrial control software according to claim 2, characterized in that, Success rate metrics are calculated based on the extended task record table, including: The extended task record table is grouped to obtain multiple semantic evaluation record sets; Determine the number of successful records and the total number of records in the semantic evaluation record set based on the result labels; The success rate is calculated by dividing the number of successful records by the total number of records.

4. The method for constructing an AI intelligent software evaluation index system for industrial control software according to claim 3, characterized in that, The specific steps for generating the indicator table are as follows: Calculate the average coordinates based on the coordinate data of each task record in the semantic evaluation record set; Calculate the average distance based on the average coordinates; The mean distance is used as the dispersion index of the semantic evaluation unit; Based on the success rate index, dispersion index, and semantic evaluation unit identifier, a table of indexes for semantic evaluation units is generated.

5. The method for constructing an AI intelligent software evaluation index system for industrial control software according to claim 4, characterized in that, The comprehensive evaluation indicators are calculated based on the indicator table, including: Obtain the version identifier of the industrial control software; Based on the version identifier and the identifier of the semantic evaluation unit, the indicator table is grouped to obtain multiple version evaluation sets; The comprehensive evaluation index is calculated based on the success rate index and dispersion index of each semantic evaluation unit in the version evaluation set.

6. The method for constructing an AI intelligent software evaluation index system for industrial control software according to claim 5, characterized in that, The specific steps for generating the AI ​​intelligent software evaluation result table are as follows: The evaluation input data is constructed based on version identifiers and comprehensive evaluation indicators; The evaluation data is input into the trained AI evaluation model to obtain the evaluation results. An AI intelligent software evaluation result table is generated based on the evaluation results.

7. A computer-readable storage medium, characterized in that: The system stores instructions that, when executed on a computer, cause the computer to perform the method for constructing an AI intelligent software evaluation index system for industrial control software as described in any one of claims 1 to 6.

Images