An ai digital human steel smelting working condition data collection and intelligent configuration method

Abstract

The application discloses an AI digital person steel smelting working condition data collection and intelligent configuration method, which comprises the following steps: S1, receiving the voice or text instruction of the operator through the digital person interaction module, performing semantic understanding and intention analysis, and generating a working condition data collection request; S2, responding to the request through the video management module, and calling the corresponding target video stream from the steel plant monitoring system; S3, performing space-time alignment on the target video stream and real-time smelting data through the data fusion module, and performing feature labeling on the key events based on a message pushing mechanism to generate fusion data and complete the collection of target working condition data; S4, controlling the digital person to present the target video stream on the large screen through the large screen fusion display module, and superimposedly displaying the smelting data according to the fusion data; and S5, controlling the digital person to feed back the video calling result and / or real-time alarm commentary in the form of voice and / or subtitles through the digital person interaction module.

Description

Technical Field

[0001] This application relates to the field of intelligent steel smelting technology, and in particular to an AI digital human method for collecting and intelligently configuring steel smelting working condition data. Background Technology

[0002] The steel smelting process involves numerous video monitoring points (such as the scrap steel inspection area, converter furnace mouth, and continuous casting crystallizer) and various real-time process data (such as temperature, composition, alarm information, and operation records). Operators need to use the monitoring system to view the production site in real time and correlate relevant data for fault diagnosis, quality analysis, and production scheduling.

[0003] Currently, the inventors recognize that video surveillance and data management systems in steel enterprises typically employ the following technical solutions:

[0004] Video retrieval method: Operators need to manually enter the camera number or select the camera from the list and set the time range in order to retrieve video recordings from the specified location. When it is necessary to find specific events (such as abnormal scrap steel quality inspection or equipment failure), operators often need to try different locations one by one and repeatedly adjust the time interval, which is cumbersome and time-consuming.

[0005] Data and video correlation: Process data (such as alarm records, quality judgment results, and material information) and video streams are usually stored in different systems, lacking automatic correlation. Operators need to switch between the video playback interface and the data report interface separately, manually comparing timestamps to find the corresponding relationships, resulting in low efficiency in fault location and analysis.

[0006] Large-screen interaction: The factory's centralized control center is usually equipped with a large display screen to centrally display multiple camera feeds and production data. The existing interaction method is mainly based on a fixed layout, and the position, size, and number of video windows are not easy to adjust. Operators need to perform multiple operations using a mouse or console to switch screens or adjust parameters, making the interaction neither intuitive nor flexible.

[0007] Adaptability to smelting scenarios: Existing monitoring systems are mostly general-purpose video management platforms that are not customized for the professional terminology and business processes of steel smelting. For example, operators cannot directly use industry-standard terms such as "retrieve video of the No. 3 scrap steel quality inspection point" for retrieval, and the system also has difficulty understanding professional search conditions such as "scrap steel type," "quality inspection status," and "fault type."

[0008] Report generation method: Most production reports are in a fixed format, requiring manual data filtering and generation. It is difficult to dynamically adjust the content and format of the report according to the user's specific needs, resulting in low flexibility and efficiency.

[0009] Furthermore, the operational data generated during the steel smelting process (including equipment operating parameters, process data, quality inspection data, etc.) are scattered across different systems such as MES, ERP, and L2, lacking a unified collection and correlation mechanism. When operators analyze an anomaly in a particular operational condition, they need to log into multiple systems to query data and then manually make a comprehensive judgment based on video footage, which is inefficient. At the same time, the display screen in the central control center usually requires manual configuration—that is, manually selecting display content, layout windows, and configuring data binding relationships. This process is highly specialized and complex, and when monitoring needs change, technicians need to reconfigure it, resulting in poor flexibility and long response times. Summary of the Invention

[0010] This application provides an AI digital human method for collecting and intelligently configuring steel smelting working condition data, aiming to solve the problems of existing steel smelting monitoring systems, such as difficulty in collecting scattered working condition data, cumbersome video search, separation of data and video, poor flexibility in large screen configuration, lack of industry terminology adaptation, and rigid report generation.

[0011] Firstly, a method for collecting and intelligently configuring AI digital human data on steel smelting conditions is provided. This method is applied to a computer device equipped with a digital human interaction module, a video management module, a data fusion module, and a large-screen fusion display module. The computer device is communicatively connected to at least one microphone array, at least one large display screen, and multiple surveillance cameras within the steel plant. The method includes:

[0012] S1: Receive the operator's voice or text commands through the digital human interaction module, perform semantic understanding and intent parsing on the commands, and generate a work condition data collection request containing search conditions;

[0013] S2: Through the video management module, based on the search conditions, respond to the working condition data collection request and retrieve the target video stream corresponding to the search conditions from the steel plant monitoring system;

[0014] S3: The target video stream and real-time smelting data are spatiotemporally aligned through the data fusion module, and key events in the video are labeled with features based on the message push mechanism to generate fused data and complete the collection of target working condition data.

[0015] S4: The large-screen fusion display module intelligently configures the presentation of the fused data and the target video stream on the large display screen, controls the digital human to present the target video stream on the large display screen, and overlays the corresponding smelting data on the video screen according to the fused data;

[0016] S5: Control the digital human through the digital human interaction module to provide feedback to the operator in the form of voice and / or subtitles on the video retrieval results and / or real-time alarm explanations.

[0017] Optionally, in the above scheme, the intent parsing in step S1 specifically includes:

[0018] The core elements of the instruction are extracted using a metallurgical professional intent recognition model. These core elements include the data type of the working condition, the location of the scrap steel inspection point, the supplier name, the type of scrap steel, the inspection status, the fault type, the time interval, and the camera location.

[0019] When the intent to "calculate benefits" is identified, a corresponding data query request is generated. The data query request is used to trigger the data fusion module to call a preset report service.

[0020] In the above scheme, optionally, step S2, retrieving the target video stream specifically includes:

[0021] The corresponding camera is located based on a preset 3D mapping table of "purpose-location-camera position";

[0022] The “Purpose” field identifies the business scenario served by the camera, including at least one of scrap steel inspection, converter monitoring, continuous casting monitoring, and electric furnace monitoring.

[0023] The "location" field identifies the physical location number, which corresponds one-to-one with the actual geographical location within the steel smelting plant area, and serves as the index key for non-video metadata in the associated business database.

[0024] The "Camera Location" field identifies the device number and installation angle information of at least one camera installed at this location;

[0025] The non-video metadata in the search criteria is logically associated with the video stream through "location + time" to achieve multi-condition combined search; wherein, the non-video metadata includes supplier, scrap steel type, status, and fault type.

[0026] Optionally, in the above scheme, step S2, the method of retrieving the target video stream further includes:

[0027] An intelligent caching mechanism is adopted to pre-store the data of the connected video interfaces. When the target video stream corresponding to the working condition data collection request exists in the cache, it is retrieved directly from the cache; when it does not exist in the cache, it is retrieved from the monitoring camera or the monitoring database.

[0028] Optionally, in the above scheme, step S3, spatiotemporal alignment and feature annotation specifically includes:

[0029] Based on the timestamp of the video stream and the fixed physical location information of the camera, the video frames are aligned and bound with the real-time smelting data at the same moment.

[0030] The system listens for and receives early warning event messages through a message push mechanism. These early warning events include danger alarms, quality issues, and malfunctions.

[0031] The occurrence time and location of the warning event are automatically marked in the video stream, and fused data containing event tags, timestamps, location information and corresponding smelting data is generated;

[0032] The fused data is packaged into a report-formatted file, which contains verification data and benefit calculation data under the search conditions.

[0033] Optionally, in the above scheme, the intelligent configuration in step S4 specifically includes:

[0034] Based on the resolution of the large display screen and the current display mode, the number of windows, window size, window position, and window transparency of the target video stream are adaptively adjusted. The display modes include single-point mode, multi-point mode, and full-flow mode, with different window layouts in different modes.

[0035] Optionally, in the above scheme, step S5, the step of providing the operator with feedback on the video retrieval results and / or real-time alarm explanations, specifically includes:

[0036] If a benefit calculation request is received, the reporting service is invoked to generate a benefit calculation report containing ERP synchronized data and then displayed.

[0037] If a real-time alarm is detected, the digital human will be controlled to broadcast the alarm content via voice. The alarm content includes the location name and the event type.

[0038] If a scheduled broadcast instruction is received, historical statistical data will be automatically collected and broadcast.

[0039] Optionally, in the above scheme, the real-time alarm explanation in step S5 may include:

[0040] Event-triggered: After the data fusion module receives a new early warning event message through the message push mechanism, the digital human interaction module sends an explanation trigger signal. The digital human interaction module retrieves the explanation script from the pre-configured event-script mapping table according to the event type and location, and controls the digital human to perform voice broadcast.

[0041] Scheduled broadcast: According to a preset timing strategy, data reports from the previous time window are collected from the database, core indicators are extracted, and the digital human is controlled to broadcast voice messages and display subtitles.

[0042] Secondly, an AI digital human steel smelting working condition data collection and intelligent configuration system is provided to implement the methods described in any of the above, including:

[0043] The digital human interaction module is used to receive voice or text commands from the operator, perform semantic understanding and intent parsing, generate a request for collecting work condition data containing search conditions, and generate feedback content and control the digital human to provide feedback to the operator in the form of voice and / or subtitles on video retrieval results and / or real-time alarm explanations.

[0044] The video management module is used to respond to the working condition data collection request and retrieve the corresponding target video stream from the steel plant monitoring system;

[0045] The data fusion module is used to align the target video stream with real-time smelting data in time and space, and to annotate key events in the video based on a message push mechanism to generate fused data and complete the collection of target working condition data.

[0046] The large-screen fusion display module is used to intelligently configure the presentation of the fused data and the target video stream on the large display screen, control the digital human to present the target video stream on the large display screen, and overlay the corresponding smelting data on the video screen according to the fused data.

[0047] Thirdly, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method as described in any of the preceding claims.

[0048] Compared with the prior art, this application has at least the following beneficial effects:

[0049] Based on further analysis and research of existing technical problems, this application recognizes that existing steel smelting monitoring systems suffer from issues such as cumbersome video retrieval, separation of data and video, unfriendly interaction, lack of industry terminology adaptation, and rigid report generation. Through the coordinated operation of a digital human interaction module, video management module, data fusion module, and large-screen fusion display module, operators can directly retrieve target videos via voice or text commands, eliminating the need to manually search for camera numbers and time intervals, significantly improving the efficiency and accuracy of video retrieval. By aligning video streams with real-time smelting data in time and space and automatically marking key events based on a message push mechanism, automatic data and video fusion is achieved, avoiding the tedious process of manually comparing timestamps and effectively shortening fault location time. Furthermore, by aggregating video streams associated with target operating condition data, automatic collection of operating condition data is achieved, avoiding the need for operators to manually search across multiple systems. This application eliminates the cumbersome process of querying; by automatically and intelligently configuring the system based on the large screen resolution and display mode, it eliminates the need for manual window layout configuration and data binding, significantly lowering the professional threshold for large screen operation and improving the response speed of monitoring screen adjustments. Using a digital human as the visual interaction hub, the system allows free movement, scaling, and overlay of videos and data on the large screen, supporting adaptive window layouts and multi-video splicing, further reducing the operational threshold and making the interaction more intuitive and natural, adapting to the usage habits of metallurgical operators. Through a speech recognition and intent parsing model optimized for metallurgical terminology, it can accurately understand industry terms such as scrap steel inspection points, scrap steel types, inspection status, and fault types, improving the accuracy of command recognition. Furthermore, this application supports users requesting report generation in natural language and enables event-triggered real-time explanations and scheduled broadcasts via the digital human, achieving flexible report generation and proactive information push. Therefore, this application effectively solves the technical defects of existing steel smelting monitoring systems, significantly improving the operational efficiency, fault diagnosis accuracy, and human-computer interaction experience of the smelting process. Attached Figure Description

[0050] Figure 1 This is a flowchart illustrating an AI digital human method for collecting and intelligently configuring steel smelting process data, provided as an embodiment of this application. Detailed Implementation

[0051] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0052] In the description of this application: unless otherwise stated, "a plurality of" means two or more. The terms "first," "second," "third," etc., in this application are intended to distinguish the objects referred to and do not have any special meaning in terms of technical connotation (e.g., they should not be construed as an emphasis on importance or order). Expressions such as "including," "comprising," and "having" also mean "not limited to" (certain units, components, materials, steps, etc.).

[0053] In one embodiment, an AI digital human steel smelting working condition data collection and intelligent configuration method is provided. The method is applied to a computer device configured with a digital human interaction module, a video management module, a data fusion module and a large screen fusion display module. The computer device is communicatively connected to at least one microphone array, at least one large display screen and multiple monitoring cameras in the steel plant. The method includes steps S1-S5.

[0054] S1: Receive voice or text commands from the operator through the digital human interaction module, perform semantic understanding and intent parsing on the commands, and generate a work condition data collection request containing search conditions.

[0055] In one embodiment, this application provides a method for collecting and intelligently configuring AI digital human data on steel smelting conditions. This method is applied to a computer device equipped with a digital human interaction module, a video management module, a data fusion module, and a large-screen fusion display module. This device is communicatively connected to a microphone array, a large display screen, and multiple surveillance cameras in the steel plant.

[0056] In step S1, the digital human interaction module receives voice or text commands from the operator. First, it uses a metallurgical intent recognition model (which employs a BERT+CRF structure and is trained using instruction data from scenarios including scrap steel inspection and converter control) to extract core elements from the commands, including scrap steel inspection location, supplier name, scrap steel type, inspection status, fault type, time interval, and camera location. When the intent to "calculate benefits" is recognized, a data query request is generated. This request is used to subsequently trigger the data fusion module to call a preset report service (the data required for the report service comes from automatic synchronization from the ERP system).

[0057] In one embodiment, step S1, intent parsing specifically includes:

[0058] The core elements of the instruction are extracted using a metallurgical professional intent recognition model. These core elements include the data type of the working condition, the location of the scrap steel inspection point, the supplier name, the type of scrap steel, the inspection status, the fault type, the time interval, and the camera location.

[0059] When the intent to "calculate benefits" is identified, a corresponding data query request is generated. The data query request is used to trigger the data fusion module to call a preset report service.

[0060] In this embodiment, after receiving the operator's natural language instructions, the digital human interaction module first calls a speech recognition model optimized for metallurgical terminology to transcribe the instructions. This model has been specifically trained for the dialects and industry terms used in steel smelting, achieving an accuracy rate of over 98%. Subsequently, a metallurgical intent recognition model built based on the BERT+CRF algorithm parses the instructions, extracting core elements closely related to scrap steel inspection, including the scrap steel inspection location, supplier name, scrap steel type (e.g., heavy, light, small, etc., without restrictions on specific grade, thickness, or shape), inspection status (queuing, inspection, waiting, inspection completed, or settlement), fault type (oil contamination, high sulfur content, sealed components, hazardous materials, etc.), time interval, and camera location. The extraction of these elements allows the system to understand the operator's actual needs without requiring manual input of conditions.

[0061] When the intent recognition model identifies the intent to "calculate benefits" from an instruction (e.g., an operator asking "calculate the economic benefits of the current scrap steel ratio"), the model does not generate a request for collecting operational data; instead, it generates a data query request. This request is sent to the data fusion module, which triggers the module to call a pre-defined report service. It should be noted that the basic data required for this report service—such as scrap steel prices and molten iron costs—has been automatically synchronized to the local database by the ERP system. The logic and calculation model for benefit calculation have also been pre-configured in the report service. The system only needs to dynamically generate reports based on search criteria (such as time intervals and scrap steel types) without requiring the digital human model to perform complex numerical calculations itself.

[0062] S2: The video management module responds to the working condition data collection request based on the search conditions and retrieves the target video stream corresponding to the search conditions from the steel plant monitoring system.

[0063] The essence of "aggregation" is: within the system, using "location" as an index, to establish logical relationships between various existing data (video, business data, alarm data) through "location + time" to form a unified and searchable data set.

[0064] In step S2, the video management module responds to the request to retrieve the target video stream. First, it locates the corresponding camera based on a pre-established 3D mapping table of "purpose-location-camera position": where "purpose" identifies the business scenario (e.g., scrap steel quality inspection, converter monitoring, etc.), "location" is the physical location number and serves as the index key for associating non-video metadata, and "camera position" includes the device number and installation angle. Then, it logically associates the non-video metadata (supplier, scrap steel type, status, fault type) in the search conditions with the video stream using "location + time," enabling multi-condition combined retrieval. Furthermore, an intelligent caching mechanism is employed: pre-stored data from connected video interfaces; if the requested video stream exists in the cache, it is retrieved directly; otherwise, it is retrieved from the camera or the database.

[0065] In one embodiment, step S2, retrieving the target video stream specifically includes:

[0066] The corresponding camera is located based on a preset 3D mapping table of "purpose-location-camera position";

[0067] The “Purpose” field identifies the business scenario served by the camera, including at least one of scrap steel inspection, converter monitoring, continuous casting monitoring, and electric furnace monitoring.

[0068] The "location" field identifies the physical location number, which corresponds one-to-one with the actual geographical location within the steel smelting plant area, and serves as the index key for non-video metadata in the associated business database.

[0069] The "Camera Location" field identifies the device number and installation angle information of at least one camera installed at this location;

[0070] The non-video metadata in the search criteria is logically associated with the video stream through "location + time" to achieve multi-condition combined search; wherein, the non-video metadata includes supplier, scrap steel type, status, and fault type.

[0071] In step S2 of this embodiment, the video management module adopts a fast positioning mechanism based on a three-dimensional mapping table of "purpose-location-camera position".

[0072] In traditional steel plant monitoring systems, operators need to remember camera numbers or rely on paper ledgers to find the corresponding images. In this embodiment, however, a pre-built three-dimensional mapping table is used to associate each camera with its business scenario (such as scrap steel quality inspection, converter monitoring, continuous casting monitoring or electric furnace monitoring), physical location number, and specific equipment number and installation angle.

[0073] The "location" field is particularly important. It not only corresponds to the actual geographical location within the factory area, but also serves as an index key for non-video metadata in the associated business database. This means that when an operator issues an instruction containing conditions such as "supplier name" or "scrap steel type," the system does not directly identify this information from the video stream. Instead, it uses the dimensions of "location" and "time" to logically associate the video stream with existing scrap steel quality inspection data, alarm records, and other data within the system.

[0074] Data such as supplier information, scrap steel type, inspection status, and fault type already exist in the system database, eliminating the need for temporary integration with external MES or ERP systems. Therefore, the video management module supports multi-condition combined searches, such as simultaneously specifying location, supplier, scrap steel type, and time range. The system first matches matching records and their corresponding time periods in the business data table, then locates the cameras based on the location mapping table, and finally extracts video clips for the corresponding time period. This approach avoids the latency of cross-system calls and achieves deep integration of surveillance video and business data, significantly improving the efficiency and accuracy of video retrieval under complex conditions.

[0075] In one embodiment, step S2 further includes retrieving the target video stream by:

[0076] An intelligent caching mechanism is adopted to pre-store the data of the connected video interfaces. When the target video stream corresponding to the working condition data collection request exists in the cache, it is retrieved directly from the cache; when it does not exist in the cache, it is retrieved from the monitoring camera or the monitoring database.

[0077] In one embodiment, the video retrieval in step S2 further incorporates an intelligent caching mechanism to improve response efficiency in frequent access scenarios.

[0078] Given that there are multiple high-profile fixed locations at the steel smelting site (such as scrap unloading points, converter furnace openings, and tapping outlets), operators often repeatedly retrieve video footage or recent historical clips from the same location within a short period. To avoid repeatedly establishing camera connections or querying the monitoring database for each request, the video management module maintains a cache pool to pre-store video interface data from established connections.

[0079] When the digital robot generates a work condition data collection request based on the operator's instructions, the module first checks whether the corresponding video stream (usually determined by the camera identifier and time interval) already exists in the cache. If a match is found, the video stream is read directly from the cache and returned, significantly reducing response time. If a match is not found, the original video stream is retrieved from the real-time monitoring camera or historical monitoring database in the usual manner and stored in the cache for use by subsequent identical or similar requests.

[0080] In some embodiments, the system also pre-stores the data of the video interface being connected, ensuring that the image can be quickly displayed when the software is opened. Through this caching mechanism, this embodiment significantly reduces the pressure on network bandwidth and backend storage from repeated retrievals while ensuring data consistency. It is especially suitable for real-world scenarios where multiple operators in shift work sequentially view the same monitoring screen.

[0081] S3: The target video stream and real-time smelting data are spatiotemporally aligned through the data fusion module, and key events in the video are labeled with features based on the message push mechanism to generate fused data, thus completing the collection of target working condition data.

[0082] In step S3, the data fusion module performs spatiotemporal alignment and feature annotation. Specifically: based on the video stream timestamp and the fixed physical location information of the camera, the video frames are aligned and bound with real-time smelting data at the same moment; a message push mechanism is used to listen for and receive early warning event messages (including danger alarms, quality objections, and faults); the occurrence time and location of events are automatically annotated in the video stream, generating fused data containing event tags, timestamps, location information, and corresponding smelting data. Finally, the fused data is packaged into a report format file, which contains quality verification data and benefit calculation data under the retrieval conditions.

[0083] In one embodiment, step S3, spatiotemporal alignment and feature annotation specifically includes:

[0084] Based on the timestamp of the video stream and the fixed physical location information of the camera, the video frames are aligned and bound with the real-time smelting data at the same moment.

[0085] The system listens for and receives early warning event messages through a message push mechanism. These early warning events include danger alarms, quality issues, and malfunctions.

[0086] The occurrence time and location of the warning event are automatically marked in the video stream, and fused data containing event tags, timestamps, location information and corresponding smelting data is generated;

[0087] The fused data is packaged into a report-formatted file, which contains verification data and benefit calculation data under the search conditions.

[0088] In one embodiment, the spatiotemporal alignment and feature annotation in step S3 are specifically implemented as follows.

[0089] After the video stream is retrieved, the data fusion module first uses the timestamps embedded in the video frames and the fixed physical location information of the cameras to bind each frame to real-time smelting data at the same time and location. For example, the video frame of "Point 03" at a certain moment will automatically match the corresponding quality inspection record, temperature data, or alarm status. Since all cameras and business systems are synchronized in time, this alignment can achieve millisecond-level accuracy.

[0090] Building upon this, the module uses a push notification mechanism to monitor real-time alert events (including hazard alarms, quality objections, equipment malfunctions, etc.) from the detection model or business system. When an alert message is received, the system automatically locates the corresponding time and position in the video stream, adds an event tag (such as "sealed component alarm"), and merges the smelting data at that moment to form structured fused data. This process can be understood as "information pushed by a message service," based on which the system automatically labels the video.

[0091] Finally, the fused data is packaged into a report-formatted file. This file not only contains quality inspection data that meets the search criteria (such as supplier, scrap steel type, and inspection status), but also integrates benefit calculation results to form a comprehensive report that combines video information, business data, and cost analysis. This design allows operators to directly obtain a structured document that can be used for traceability and decision-making while retrieving video data, eliminating the need to manually piece together information from multiple systems.

[0092] S4: The large-screen fusion display module intelligently configures the presentation of the fused data and the target video stream on the large display screen, controls the digital human to present the target video stream on the large display screen, and overlays the corresponding smelting data on the video screen according to the fused data.

[0093] In step S4, the large-screen fusion display module controls the digital human to present the target video stream on the large screen, and overlays the corresponding smelting data on the video screen according to the fusion data (such as displaying temperature curves or quality indicators on one side of the video).

[0094] In this application, "configuration" refers to the process of automatically configuring and adjusting the number, layout, size, transparency, and data overlay method of video windows according to the display scenario and user needs; "intelligent configuration" is different from the traditional configuration method that requires manual dragging and dropping of controls and manual configuration of data binding. It refers to the process in which the system automatically completes the above configuration according to the large screen resolution, display mode, and integrated data characteristics.

[0095] In one embodiment, when presenting the target video stream, the large-screen fusion display module overlays the temperature curves of corresponding points at the bottom or side of the video screen. Specifically, the data fusion module aligns real-time collected data such as converter furnace temperature, molten steel temperature, or crystallizer cooling water temperature with the video frames according to timestamps, generating a continuous curve. The large-screen fusion display module overlays this curve in a semi-transparent or independent window at the bottom of the video screen. The curve automatically scrolls to the right over time, marking the current temperature value, alarm threshold, and historical comparison reference line. When the temperature exceeds the preset threshold, the curve changes color and flashes, while a digital human voice prompts "Temperature exceeded, please pay attention." Operators can simultaneously observe the video screen and temperature change trend without switching to other interfaces, improving fault analysis efficiency.

[0096] In one embodiment, the intelligent configuration in step S4 specifically includes:

[0097] Based on the resolution of the large display screen and the current display mode, the number of windows, window size, window position, and window transparency of the target video stream are adaptively adjusted. The display modes include single-point mode, multi-point mode, and full-flow mode, with different window layouts in different modes.

[0098] In one embodiment, the way the digital human presents the video stream in step S4 fully considers the actual needs of the large-screen display at the steel smelting site. Since the resolution of the large screens in different steel plants varies (e.g., 1080p or 4K), and the number of points that operators need to pay attention to varies in different scenarios, this embodiment introduces an adaptive display optimization strategy.

[0099] Specifically, the large-screen fusion display module first detects the resolution of the current display screen and automatically adjusts the layout of the video windows according to the user-selected display mode (single-point mode, multi-point mode, or full-process mode). In single-point mode, the system typically highlights the target point in a main window, while also displaying related data in a few auxiliary windows. In multi-point mode, the system divides the large screen into multiple areas (e.g., a 2×2 or 3×3 grid) and displays video streams from different points simultaneously. In full-process mode, the system stitches together video feeds from multiple perspectives according to the smelting process sequence (e.g., scrap steel inspection, converter smelting, continuous casting, etc.) to help operators gain a comprehensive understanding of the production status.

[0100] In addition to the number and size of windows, the module dynamically adjusts the transparency of the windows based on the importance of the data. For example, for alarm information or key process parameters, the system reduces the transparency of the overlaid data to ensure clear visibility, while auxiliary information is displayed semi-transparently to avoid obscuring the main video content. This adaptive mechanism allows the digital human to move and zoom freely on the large screen, while ensuring that the integrated display of video and data does not interfere with the operator's line of sight, thus improving the interactive experience and information acquisition efficiency.

[0101] S5: Control the digital human through the digital human interaction module to provide feedback to the operator in the form of voice and / or subtitles on the video retrieval results and / or real-time alarm explanations.

[0102] In step S5, the digital human interaction module controls the digital human to provide feedback on the results in the form of voice and / or subtitles. The feedback of video retrieval results and / or real-time alarm explanations to the operator specifically includes three scenarios:

[0103] If a benefit calculation request is received, the reporting service is invoked to generate and display a benefit calculation report containing ERP synchronized data. If a real-time alarm is detected, the digital human is controlled to announce the alarm content (location name and event type) via voice. If a timed announcement instruction is received, historical statistical data is automatically collected and announced. Furthermore, real-time alarm explanations are divided into event-triggered and timed announcement types.

[0104] Event-triggered: After the data fusion module receives a new warning event through the message push mechanism, the digital human interaction module sends an explanation trigger signal, retrieves the explanation script from the pre-configured event-script mapping table according to the event type and location, and controls the digital human to broadcast the message.

[0105] Scheduled broadcast: According to a preset time strategy (e.g., 8:00 AM every day), data reports from the previous time window are collected from the database, and the core indicators are extracted and then broadcast with voice and subtitles.

[0106] In one embodiment, the feedback in step S5 specifically includes:

[0107] If a benefit calculation request is received, the reporting service is invoked to generate a benefit calculation report containing ERP synchronized data and then displayed.

[0108] If a real-time alarm is detected, the digital human will be controlled to broadcast the alarm content via voice. The alarm content includes the location name and the event type.

[0109] If a scheduled broadcast instruction is received, historical statistical data will be automatically collected and broadcast.

[0110] In one embodiment, the feedback mechanism in step S5 covers three typical scenarios: operator-initiated inquiries, real-time system alerts, and periodic reporting, and is implemented as follows.

[0111] When the system receives a benefit calculation request (e.g., an operator asks "calculate the economic benefits of the current scrap steel ratio"), the digital human interaction module does not perform the calculation itself. Instead, it triggers the data fusion module to call a pre-defined report service. The raw data required for the benefit calculation (such as scrap steel prices and molten iron costs) has been automatically synchronized to the local database by the ERP system, and the calculation logic and results have also been pre-processed. The report service dynamically generates reports containing ratio suggestions, cost comparisons, etc., based on the search criteria in the request (such as time range, scrap steel type, etc.), and displays them on a large screen using a digital human, supplemented by voice announcements of key conclusions.

[0112] When a real-time alarm is detected, the system receives the warning event message via a push notification mechanism (the message must include at least the location name and event type, such as "Alarm for a sealed component at location 3"). The digital human then controls the voice broadcast of the alarm content, allowing operators to be aware of the anomaly without actively checking. The triggering conditions for this real-time commentary (such as the occurrence of a warning message) can be freely configured, with commentary content such as "A suspected sealed component has been found at location 2 in the Xinxing site." Simultaneously, the alarm content is usually displayed in subtitles at the bottom of the large screen, facilitating information confirmation in noisy environments.

[0113] When a scheduled broadcast instruction is received, the system automatically executes data collection and broadcasting tasks according to a preset timing strategy (e.g., 8:00 AM every day). This mechanism can "automatically collect yesterday's data reports and broadcast them daily." The digital robot extracts historical data from the previous statistical window from the database, summarizes key indicators (such as total inspection volume, number of alarms, pass rate, etc.), and then broadcasts them in voice format, while simultaneously displaying statistical charts on a large screen. This method is suitable for shift handover morning meetings or daily data reviews, reducing the workload of manually compiling reports and improving the timeliness and standardization of information transmission.

[0114] In one embodiment, the real-time alarm explanation in step S5 specifically includes:

[0115] Event-triggered: After the data fusion module receives a new early warning event message through the message push mechanism, the digital human interaction module sends an explanation trigger signal. The digital human interaction module retrieves the explanation script from the pre-configured event-script mapping table according to the event type and location, and controls the digital human to perform voice broadcast.

[0116] Scheduled broadcast: According to a preset timing strategy, data reports from the previous time window are collected from the database, core indicators are extracted, and the digital human is controlled to broadcast voice messages and display subtitles.

[0117] In one embodiment, the real-time alarm explanation in step S5 is further divided into two specific implementation methods: event-triggered and timed broadcast.

[0118] Event-triggered commentary is primarily used in emergency warning scenarios. When the data fusion module receives a new warning event message (such as a hazard alarm, quality issue, or equipment malfunction) via a push notification mechanism, the digital human interaction module is immediately triggered, sending a commentary trigger signal. The system has a pre-configured mapping relationship between events and commentary scripts (e.g., the script template corresponding to an oil spill alarm at a specific location). Based on the received alarm event type and location name, the digital human interaction module retrieves the appropriate commentary script from this mapping table and then controls the digital human to deliver the voice announcement. This allows operators to immediately know "what happened at which location" without actively checking the screen, enabling a rapid response.

[0119] Scheduled broadcast commentary is used for periodic data reporting scenarios. The system automatically triggers the digital human to collect data reports from the database within the previous time window, according to a pre-set schedule by the administrator or operator (e.g., 8 AM daily or at shift handover time). This mechanism can "automatically collect yesterday's data reports and broadcast them daily." The data fusion module extracts core indicators (such as total scrap steel inspection volume, alarm frequency, and pass rate) from the synchronized business database. The digital human control module then broadcasts these indicators via voice, with subtitles displayed simultaneously at the bottom of the large screen. This method eliminates the tedious process of manually compiling and reading reports, making it particularly suitable for information transmission during shift handovers and improving the efficiency and standardization of data reporting.

[0120] In one embodiment, the digital human serves as a central visual interaction hub, freely moving and scaling on a large display screen, overlaying video feeds and real-time data. Specifically, under the control of the large-screen integrated display module, the digital human can respond to operator voice commands (such as "move the digital human to the right" or "zoom in on the digital human") or preset interaction logic, changing its position and size on the screen. When the digital human moves, its overlaid video window and data display area adjust accordingly, avoiding obscuring key process data. After scaling, the digital human maintains consistent facial expressions and body movements, ensuring natural interaction. Through this method, operators can flexibly adjust the digital human's position based on the layout of the content they are currently focusing on, improving the efficiency of large-screen space utilization.

[0121] In one embodiment, the large-screen fusion display module supports intelligent cropping of video content. Specifically, for video streams from fixed cameras, the system automatically identifies and crops the core smelting area in the video frame based on the business scenario served by the camera (such as scrap steel inspection, converter monitoring, continuous casting monitoring, and electric furnace monitoring) and preset region of interest parameters. The core area includes, but is not limited to, the unloading hopper in the scrap steel inspection area, the furnace mouth and slag line of the converter, and the liquid level in the continuous casting crystallizer. The cropped video retains only this core area, removing surrounding invalid or redundant image information, and presents it in a magnified or centered manner in the large-screen window. When the operator voice-requests a video of a specific location, the large-screen fusion display module directly displays the cropped, focused image, reducing irrelevant visual interference and facilitating the operator's quick observation of key smelting details.

[0122] In one embodiment, the system's built-in engine analyzes data metrics in real time and dynamically generates visual charts. Specifically, after aligning the video stream with real-time smelting data, the data fusion module continuously monitors key process parameters at each location (such as quality inspection pass rate, fault type distribution, and scrap steel price fluctuations). When an operator issues commands such as "display benefit trends" or "display quality inspection data charts," the engine automatically extracts corresponding metrics from the fused data based on current search criteria or timing strategies, dynamically generating visual charts such as bar charts, line charts, or pie charts. The generated charts are overlaid and displayed next to the video screen or in a designated area on a large screen, and support real-time narration of the chart content by a digital human. Operators do not need to manually select chart types and ranges; the engine automatically adapts the most suitable chart format based on data characteristics and user intent to assist in operational decision-making.

[0123] In one embodiment, the digital human interaction module controls the digital human to synchronize lip movements and body gestures during voice broadcasts. Specifically, when the digital human provides real-time alarm explanations or feedback on video retrieval results, the system drives the digital human's facial model to produce lip movements that match the pronunciation based on preset voice text and emotion tags, while simultaneously coordinating with natural body movements (such as nodding, gesturing, facing the video window on the large screen, etc.). The lip-sync is driven by the voice duration and phoneme sequence, and the body movements select corresponding postures from the action library based on keywords in the explanation content (such as "attention," "zoom in," "compare"). Through the coordination of lip-sync, body movements, and subtitle display, the digital human provides natural and intuitive operation guidance, enhancing the friendliness and comprehensibility of human-computer interaction.

[0124] In one embodiment, the large-screen fusion display module allows operators to zoom in on video details using gestures. Specifically, when facing the large screen, operators can zoom in and out on the currently focused video window using specific gestures (such as spreading their hands outwards). The large-screen fusion display module recognizes the operator's hand movements through a connected depth camera or gesture sensor. When a zoom-in gesture is detected, the display ratio of the specified video window is gradually increased, and the window position is automatically adjusted to avoid obscuring other information; when a zoom-out gesture is detected, the window returns to its original size. Gesture control can be used in parallel with voice control: for example, the operator first selects "video at point 3" via voice, and then uses gestures to zoom in on the window to observe the local details of scrap steel unloading. The digital human will provide voice feedback when the gesture occurs: "The video has been zoomed in for you." This interaction method lowers the operating threshold and is adapted to the habits of metallurgical operators.

[0125] In one embodiment, the data fusion module automatically adjusts the priority and overlay transparency of the displayed data based on the importance weights of the data. Specifically, the system presets importance levels for different types of smelting data: for example, real-time alarm events have the highest priority (weight 10), followed by quality inspection status (weight 7), supplier information has a general priority (weight 5), and ambient temperature has a low priority (weight 2). When multiple data points need to be overlaid on the video screen simultaneously, the large-screen fusion display module allocates display areas according to their weights: high-priority data is displayed prominently at the top or center of the screen with 0% transparency (completely opaque); medium-priority data is displayed on the side with 30% transparency; and low-priority data is displayed in the corner or hidden by default with 70% transparency. Operators can also temporarily adjust the priority of a data type using voice commands (e.g., "bring supplier information to the front"). This dynamic weighting mechanism ensures that key information is presented preferentially within the limited large-screen space, avoiding information overload.

[0126] In one embodiment, an example of a complete response flow is provided.

[0127] The operator issues a voice command: "Open the video feed at the No. 1 converter opening." The digital human interaction module then executes the following sub-steps:

[0128] Upon hearing the instruction: Collect audio via the microphone array.

[0129] Analysis intent: The intent recognition model judged it as "video retrieval" and extracted the entity "No. 1 converter furnace opening".

[0130] Action Execution: The video management module locates the camera number corresponding to converter No. 1's furnace opening based on the "Purpose-Location-Camera" mapping table and activates its video stream; the data fusion module automatically retrieves real-time smelting data (furnace temperature, oxygen blowing volume, furnace age) from the database; the large-screen fusion display module projects the video stream onto the center of the large screen and overlays data such as temperature and furnace age onto the right side of the video screen in dashboard format; the digital human interaction module controls the digital human to provide voice feedback: "Video of converter No. 1's furnace opening has been activated, current furnace temperature is 1600 degrees Celsius." The operator can then say "Adjust to 15-degree viewing angle," and the video management module sends a control command to the PTZ camera to adjust the angle, and the digital human again provides feedback: "Viewpoint adjusted."

[0131] In one embodiment, an AI digital human steel smelting working condition data collection and intelligent configuration system is provided to implement the method described in any of the above embodiments, including:

[0132] The digital human interaction module is used to receive voice or text commands from the operator, perform semantic understanding and intent parsing, generate a request for collecting work condition data containing search conditions, and generate feedback content and control the digital human to provide feedback to the operator in the form of voice and / or subtitles on video retrieval results and / or real-time alarm explanations.

[0133] The video management module is used to respond to the working condition data collection request and retrieve the corresponding target video stream from the steel plant monitoring system;

[0134] The data fusion module is used to align the target video stream with real-time smelting data in time and space, and to annotate key events in the video based on a message push mechanism to generate fused data and complete the collection of target working condition data.

[0135] The large-screen fusion display module is used to intelligently configure the presentation of the fused data and the target video stream on the large display screen, control the digital human to present the target video stream on the large display screen, and overlay the corresponding smelting data on the video screen according to the fused data.

[0136] In one embodiment, a computer-readable storage medium is provided having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the above embodiments.

[0137] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

Claims

1. A method for collecting and intelligently configuring AI digital human steel smelting working condition data, characterized in that, The method is applied to a computer device equipped with a digital human interaction module, a video management module, a data fusion module, and a large-screen fusion display module. The computer device is communicatively connected to at least one microphone array, at least one large display screen, and multiple surveillance cameras within the steel plant. The method includes: S1: Receive the operator's voice or text commands through the digital human interaction module, perform semantic understanding and intent parsing on the commands, and generate a work condition data collection request containing search conditions; S2: Through the video management module, based on the search conditions, respond to the working condition data collection request and retrieve the target video stream corresponding to the search conditions from the steel plant monitoring system; S3: The target video stream and real-time smelting data are spatiotemporally aligned through the data fusion module, and key events in the video are labeled with features based on the message push mechanism to generate fused data and complete the collection of target working condition data. S4: The large-screen fusion display module intelligently configures the presentation of the fused data and the target video stream on the large display screen, controls the digital human to present the target video stream on the large display screen, and overlays the corresponding smelting data on the video screen according to the fused data; S5: Control the digital human through the digital human interaction module to provide feedback to the operator in the form of voice and / or subtitles on the video retrieval results and / or real-time alarm explanations.

2. The method according to claim 1, characterized in that, In step S1, intent parsing specifically includes: The core elements of the instruction are extracted using a metallurgical professional intent recognition model. These core elements include the data type of the working condition, the location of the scrap steel inspection point, the supplier name, the type of scrap steel, the inspection status, the fault type, the time interval, and the camera location. When the intent to "calculate benefits" is identified, a corresponding data query request is generated. The data query request is used to trigger the data fusion module to call a preset report service.

3. The method according to claim 1, characterized in that, In step S2, retrieving the target video stream specifically includes: The corresponding camera is located based on a preset 3D mapping table of "purpose-location-camera position"; The "Purpose" field identifies the business scenario served by the camera, including at least one of scrap steel inspection, converter monitoring, continuous casting monitoring, and electric furnace monitoring. The "location" field identifies the physical location number, which corresponds one-to-one with the actual geographical location within the steel smelting plant area, and serves as the index key for non-video metadata in the associated business database; The "Camera Location" field identifies the device number and installation angle information of at least one camera installed at this location; The non-video metadata in the search criteria is logically associated with the video stream through "location + time" to achieve multi-condition combined search; wherein, the non-video metadata includes supplier, scrap steel type, status, and fault type.

4. The method according to claim 1, characterized in that, In step S2, the method of retrieving the target video stream further includes: An intelligent caching mechanism is adopted to pre-store the data of the connected video interfaces. When the target video stream corresponding to the working condition data collection request exists in the cache, it is retrieved directly from the cache; when it does not exist in the cache, it is retrieved from the monitoring camera or the monitoring database.

5. The method according to claim 1, characterized in that, In step S3, spatiotemporal alignment and feature annotation specifically include: Based on the timestamp of the video stream and the fixed physical location information of the camera, the video frames are aligned and bound with the real-time smelting data at the same moment. The system listens for and receives early warning event messages through a message push mechanism. These early warning events include danger alarms, quality issues, and malfunctions. The occurrence time and location of the warning event are automatically marked in the video stream, and fused data containing event tags, timestamps, location information and corresponding smelting data is generated; The fused data is packaged into a report-formatted file, which contains verification data and benefit calculation data under the search conditions.

6. The method according to claim 1, characterized in that, The intelligent configuration described in step S4 specifically includes: Based on the resolution of the large display screen and the current display mode, the number of windows, window size, window position, and window transparency of the target video stream are adaptively adjusted. The display modes include single-point mode, multi-point mode, and full-flow mode, with different window layouts in different modes.

7. The method according to claim 1, characterized in that, In step S5, the step of providing the operator with feedback on the video retrieval results and / or real-time alarm explanations specifically includes: If a benefit calculation request is received, the reporting service is invoked to generate a benefit calculation report containing ERP synchronized data and then displayed. If a real-time alarm is detected, the digital human will be controlled to broadcast the alarm content via voice. The alarm content includes the location name and the event type. If a scheduled broadcast instruction is received, historical statistical data will be automatically collected and broadcast.

8. The method according to claim 1, characterized in that, The real-time alarm explanation in step S5 specifically includes: Event-triggered: After the data fusion module receives a new early warning event message through the message push mechanism, the digital human interaction module sends an explanation trigger signal. The digital human interaction module retrieves the explanation script from the pre-configured event-script mapping table according to the event type and location, and controls the digital human to perform voice broadcast. Scheduled broadcast: According to a preset timing strategy, data reports from the previous time window are collected from the database, core indicators are extracted, and the digital human is controlled to broadcast voice messages and display subtitles.

9. An AI digital human system for collecting and intelligently configuring steel smelting process data, used to implement the method described in any one of claims 1-8, characterized in that, include: The digital human interaction module is used to receive voice or text commands from operators, perform semantic understanding and intent parsing, and generate a request for collecting work condition data containing search conditions. And for generating feedback content and controlling the digital human to provide feedback to the operator in the form of voice and / or subtitles on video retrieval results and / or real-time alarm explanations; The video management module is used to respond to the working condition data collection request and retrieve the corresponding target video stream from the steel plant monitoring system; The data fusion module is used to align the target video stream with real-time smelting data in time and space, and to annotate key events in the video based on a message push mechanism to generate fused data and complete the collection of target working condition data. The large-screen fusion display module is used to intelligently configure the presentation of the fused data and the target video stream on the large display screen, control the digital human to present the target video stream on the large display screen, and overlay the corresponding smelting data on the video screen according to the fused data.

10. A computer-readable storage medium, characterized in that, It stores a computer program thereon, which, when executed by a processor, implements the method as described in any one of claims 1 to 8.

Images