A data recognition method, device and equipment based on a video signal and a medium

Abstract

The application discloses a data recognition method and device based on a video signal, equipment and a medium, and relates to the technical field of display. The method comprises the following steps: recognizing a picture of a video signal source in advance, storing region information of a determined target region in association with a prompt word as target region configuration, determining a target region range from a current video frame according to the pre-stored region information during data recognition, and performing fixed-point recognition according to the pre-stored prompt word. Compared with full-picture recognition of the whole picture each time, single recognition only processes the content in the target region image, so that the resource consumption and time delay of recognition are reduced. Meanwhile, the structured data is output to a visual system, so that the standard format data which can be directly used is provided for visual display.

Description

Technical Field

[0001] This invention relates to the field of display technology, and in particular to a data recognition method, apparatus, device, and medium based on video signals. Background Technology

[0002] Large-screen visualization systems are widely used in various scenarios such as smart parks, industrial monitoring, and data centers. They are used to centrally display operational data from various business systems in the form of charts and numbers on a large screen, enabling managers to conduct centralized monitoring and decision-making. Large-screen visualization systems typically connect to data in two ways: one is by directly connecting to the customer's database or data source, retrieving data, cleaning and filtering it, and then visualizing it; the other is by connecting to the customer's business system through interface protocols to read data, storing it in the database, and then visualizing it. However, both of these methods require the customer to provide database connection information or business system interfaces, requiring customer authorization and corresponding technical support, which poses data security and system security risks. Furthermore, in real-world scenarios, there are often many legacy systems that are no longer maintainable. The original vendors of these systems have ceased technical support, or the systems themselves lack source code and maintenance documentation, making it impossible to provide any data interfaces or database access permissions.

[0003] For scenarios where software interface integration is not possible, existing technologies extract screen text data by frame extraction from display channels such as HDMI / VGA, combined with Optical Character Recognition (OCR) technology. However, this frame extraction combined with OCR solution suffers from low recognition accuracy when dealing with complex layouts, non-standard interfaces, tables, and heterogeneous data. Existing technologies also employ large-scale models for recognition, but this requires significant computing resources and suffers from high latency.

[0004] Therefore, how to achieve highly reliable data identification with low computing power consumption without accessing customer databases and business interfaces is a technical problem that urgently needs to be solved. Summary of the Invention

[0005] In view of this, the purpose of this invention is to provide a method, apparatus, device, and medium for data recognition based on video signals, which can achieve video signal data recognition at low cost and high efficiency without accessing customer databases and business interfaces. The specific solution is as follows: In a first aspect, this application discloses a data recognition method based on video signals, including: Obtain the target area configuration corresponding to the current video signal source; the target area configuration is obtained by pre-identifying the target area by recognizing the image of the video signal source, and storing the area information of the target area and the prompt words used to extract data from the target area together; According to the target region configuration, the target region image is determined from the video frames of the current video signal source; Based on the prompt words in the target region configuration, the target region image is identified using a multimodal recognition model to obtain structured data and output it to the visualization system.

[0006] Optionally, before obtaining the target area configuration corresponding to the current video signal source, the method further includes: Assign a unique signal source identifier to each video signal source; The video signal source is subjected to image recognition to determine the target area in the image of the video signal source; The target area information is associated with the prompt words used to extract data from the target area to obtain the target area configuration corresponding to the video signal source; Construct a mapping relationship between the signal source identifier and the target area configuration.

[0007] Optionally, the step of performing image recognition on the video signal source to determine the target area in the image of the video signal source includes: The video signal source is segmented using a segmentation model to obtain multiple candidate boxes, and a serial number is added to each candidate box. The image containing the serial number is input into the multimodal recognition model. The multimodal recognition model performs semantic recognition on the image content corresponding to each serial number, filters out the candidate boxes containing data from the multiple candidate boxes, and adds semantic labels, bounding box information and confidence scores to the filtered candidate boxes to obtain candidate regions. The candidate regions are filtered and / or fine-tuned to obtain the target region.

[0008] Optionally, the prompt word includes at least one of the following: coordinate information of the target area, semantic description, numerical range constraints, unit constraints, and output format constraints.

[0009] Optionally, after obtaining the structured data and outputting it to the visualization system, the process further includes: The structured data is validated based on the numerical range constraints, and / or unit constraints, and / or output format constraints in the prompt words; If the verification fails, the target region image is enlarged to obtain a new region image, and the new region image is re-identified and verified according to the prompt words; If the re-verification fails, the video frames corresponding to the target area image will be re-identified and verified according to the prompt words.

[0010] Optionally, before recognizing the target region image using a multimodal recognition model based on the prompt words in the target region configuration, the method further includes: Calculate the degree of change between the target region image in the current video frame and the target region image of the same region in the previous video frame; If the degree of change exceeds a preset degree of change threshold, then the operation of recognizing the target region image using a multimodal recognition model based on the prompt words configured in the target region is executed.

[0011] Optionally, determining the target region image from the video frames of the current video signal source includes: Statistical analysis of the frequency of change of each target region in each video signal source; The recognition frequency of the video signal source is adjusted according to the changing frequency; wherein the changing frequency and the recognition frequency are positively correlated. The target video frame is extracted from the current video signal source according to the identification frequency, and the target region image is determined from the target video frame.

[0012] Secondly, this application discloses a data recognition device based on video signals, comprising: The target area configuration acquisition module is used to acquire the target area configuration corresponding to the current video signal source; the target area configuration is obtained by pre-identifying the target area by recognizing the image of the video signal source, and storing the area information of the target area and the prompt words used to extract data from the target area together; The target region image determination module is used to determine the target region image from the video frames of the current video signal source according to the target region configuration; The data recognition module is used to recognize the target region image based on the prompt words in the target region configuration using a multimodal recognition model, obtain structured data, and output it to the visualization system.

[0013] Thirdly, this application discloses an electronic device, comprising: Memory, used to store computer programs; A processor is used to execute the computer program to implement the aforementioned data recognition method based on video signals.

[0014] Fourthly, this application discloses a computer-readable storage medium for storing a computer program; wherein the computer program, when executed by a processor, implements the aforementioned data recognition method based on video signals.

[0015] In this application, a target region configuration corresponding to the current video signal source is obtained. The target region configuration is obtained by pre-identifying the target region by recognizing the image of the video signal source, and storing the region information of the target region and the prompt words used to extract data from the target region. According to the target region configuration, the target region image is determined from the video frame of the current video signal source. Based on the prompt words in the target region configuration, the target region image is recognized using a multimodal recognition model to obtain structured data and output to a visualization system.

[0016] Therefore, by pre-identifying the video signal source, the identified target area's regional information is associated with prompts and stored as a target area configuration. During data recognition, the target area range is determined from the current video frame according to the pre-stored regional information, and point recognition is performed based on the pre-stored prompts. Compared to performing full-frame recognition on each frame, each recognition only processes the content within the target area image, reducing resource consumption and latency. Simultaneously, by outputting structured data to the visualization system, standard format data that can be directly used for visualization is provided. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0018] Figure 1 A flowchart of a data recognition method based on video signals provided in this application; Figure 2 A flowchart of a specific data identification preparation stage is provided for this application; Figure 3 A flowchart of a specific data verification method provided in this application; Figure 4 A flowchart of a specific data identification method based on variability provided in this application; Figure 5 A schematic diagram of a data recognition device based on video signals is provided in this application; Figure 6 This application provides a structural diagram of an electronic device. Detailed Implementation

[0019] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] In existing technologies, frame extraction is performed through display channels such as HDMI / VGA, combined with Optical Character Recognition (OCR) technology to extract screen text data. However, the frame extraction combined with OCR solution suffers from low recognition accuracy when dealing with complex layouts, non-standard interfaces, tables, and heterogeneous data. Existing technologies also use large models for recognition, but this requires significant computing resources and has high latency. To overcome the above technical problems, this application proposes a data recognition method based on video signals, which can achieve video signal data recognition at low cost and high efficiency without accessing customer databases and business interfaces.

[0021] This application discloses a data recognition method based on video signals. See [link to relevant documentation]. Figure 1 As shown, the method may include the following steps: Step S11: Obtain the target area configuration corresponding to the current video signal source; the target area configuration is obtained by pre-identifying the target area by recognizing the image of the video signal source, and storing the area information of the target area and the prompt words used to extract data from the target area.

[0022] Target area configuration is performed during the preparation phase. The target area is pre-identified by identifying the video signal source's frame. Then, the target area's information is associated with prompts used to extract data from it, resulting in the target area configuration. The target area is the region within the video signal source's frame that carries service data, such as a digital area displaying "Real-time Power: 1284.6kW" or a table area containing multiple rows of data. The area information can be the target area's location information, used to locate the target area within the video frame. Specifically, it can be bounding box coordinates (x, y, w, h), i.e., the x-coordinate of the top-left corner, y-coordinate, width, and height of the area. It should be noted that the video signal source mentioned above is a video signal input through interfaces such as HDMI (High-Definition Multimedia Interface), VGA (Video Graphic Array), DP (DisplayPort), DVI (Digital Visual Interface), or network streaming media.

[0023] Understandably, identifying the target area from the entire frame requires significant computational resources. Therefore, pre-determining it during the preparation phase allows the focus to be placed solely on the target area during actual data recognition, thereby improving data recognition efficiency and reducing computational resources. Specifically, the target area configuration is pre-established and stored before data acquisition. This process is infrequent, occurring once during system initialization or updated when the layout of the video signal source changes. The target area configuration is persistently stored for repeated use later.

[0024] In some implementations, the prompt words include at least one of the following: coordinate information of the target region, semantic description, numerical range constraints, unit constraints, and output format constraints. Coordinate information describes the position of the target region within a video frame, and can be bounding box coordinates (x, y, w, h). Semantic description is a textual explanation of the meaning of the target region's data; for example, the target region displays the real-time total power. Numerical range constraints limit the numerical range of the recognition result; for example, the value should be between 0 and 10000, used for subsequent validity verification of the recognition result. Unit constraints limit the physical unit of the recognition result; for example, the unit is kW. Output format constraints specify the output format of the recognition result; for example, outputting in JSON format, or returning tabular data as a JSON array. Prompt words can improve the recognition accuracy and output format standardization of multimodal recognition models. By embedding constraint information into the prompt words, the model can be guided to output results that meet expectations. This embodiment does not limit the prompt words; they can be flexibly combined and used according to the specific data type of the target region and business needs.

[0025] Step S12: Determine the target region image from the video frames of the current video signal source according to the target region configuration.

[0026] The current video frame is a single frame image obtained by truncating a specific video signal. Video frames can be periodically extracted according to a preset frame rate, or triggered when a change in the screen content is detected. Then, according to the region information in the target region configuration, such as bounding box coordinates, the corresponding target region image is cropped or located from the video frame. Subsequent identification only focuses on this target region image, whose pixel count is much smaller than the entire frame; for example, the entire frame may be 1920×1080 pixels, while the target region image may only be 360×80 pixels.

[0027] Step S13: Based on the prompt words in the target region configuration, use a multimodal recognition model to recognize the target region image, obtain structured data, and output it to the visualization system.

[0028] Multimodal recognition models are deep learning models capable of understanding images and text. They can perform visual understanding of input target region images and extract structured data based on prompts. Multimodal recognition models can be lightweight models deployed locally on edge devices or can utilize cloud-based model services.

[0029] Specifically, after obtaining the target region image, the cropped region sub-image and corresponding prompt words are input into the multimodal recognition model. Based on the semantics, data type, units, and output format constraints described in the prompt words, the model only recognizes the target region image and outputs structured data. Structured data refers to data conforming to a preset format, such as numerical or text data in JSON format, which has clearly defined field names and data types, facilitating direct parsing and use by downstream visualization systems. Finally, the recognized structured data is output to the visualization system via wired or wireless network transmission for large-screen display or business data aggregation.

[0030] By pre-establishing and storing target area configurations, the acquisition phase only requires point-to-point recognition of the identified target areas according to this configuration. This eliminates the need for full-frame analysis of the entire image and re-identifies and locates the target areas, significantly reducing computational power consumption and latency per recognition cycle. Furthermore, due to the use of a multimodal recognition model for direct understanding of the target area image, it offers stronger layout adaptability and semantic understanding capabilities compared to traditional OCR solutions, resulting in higher recognition reliability in scenarios involving tables, heterogeneous data, and non-standard interfaces. Finally, structured data is output to the visualization system, enabling it to directly use standard format data for chart rendering and business processing without additional parsing or adaptation, achieving highly reliable video signal visualization.

[0031] As can be seen from the above, this embodiment pre-identifies the video signal source's image, associates the identified target area's regional information with prompts, and stores it as a target area configuration. During data recognition, the target area range is determined from the current video frame according to the pre-stored regional information, and fixed-point recognition is performed based on the pre-stored prompts. Compared to performing full-frame recognition on each frame, a single recognition process only processes the content within the target area image, reducing resource consumption and latency. Simultaneously, by outputting structured data to the visualization system, it provides directly usable standard format data for visualization.

[0032] This application discloses a specific data identification preparation stage process in its embodiments. See [link to relevant documentation]. Figure 2 As shown, before obtaining the target area configuration corresponding to the current video signal source, the following steps may be included: Step S21: Assign a unique signal source identifier to each video signal source.

[0033] The Signal Source ID is used to distinguish different video signal sources; when a device has multiple video signal acquisition interfaces (such as multiple HDMI inputs), each video signal source connected to a physical interface is assigned a unique ID.

[0034] Step S22: Perform image recognition on the video signal source to determine the target area in the image of the video signal source.

[0035] After assigning a unique source identifier to the video signal source, one or more frames of the signal source are captured, and the content of the frame is identified to determine the data-carrying areas as target regions. Image recognition can employ a multimodal recognition model to perform full-frame semantic understanding of the entire frame, identifying all data-carrying target regions, such as number display areas, table areas, and status text areas. Each identified target region corresponds to a unique region identifier, and its bounding box coordinates within the frame are recorded. The accuracy of the target region is crucial to the accuracy of data recognition; therefore, adjustments can be made manually after multimodal recognition model analysis.

[0036] Step S23: Associate the region information of the target region with the prompt words used to extract data from the target region to obtain the target region configuration corresponding to the video signal source.

[0037] After identifying the target regions, corresponding prompts are generated for each region. The prompts can be generated based on the semantic tags, data types, units, and output format requirements of the target region. Subsequently, the region information of each target region is associated with its corresponding prompt to form the target region configuration for the video signal source. This configuration is then manually reviewed and corrected before being stored.

[0038] Step S24: Construct a mapping relationship between the signal source identifier and the target area configuration.

[0039] A one-to-one mapping relationship is established between the assigned signal source identifiers and the generated target area configurations, and this mapping is persistently stored. This allows for quick querying and loading of the corresponding target area configuration based on the currently processed signal source identifier during data identification. When the interface layout of a signal source changes, steps S22 to S24 above only need to be re-executed for that signal source; the configurations of other signal sources remain unaffected.

[0040] In this embodiment, the allocation of signal source identifiers, image recognition, configuration generation, and mapping relationship construction are all completed before the acquisition phase, and are one-off or low-frequency operations. The mapping relationship between signal source identifiers and target area configurations established through the above process facilitates the rapid acquisition of target area configurations and enables the separate maintenance of multiple signal sources.

[0041] In a specific implementation, the step of performing image recognition on the video signal source to determine the target region in the image of the video signal source may include: segmenting the image of the video signal source using a segmentation model to obtain multiple candidate boxes, and adding a serial number label to each candidate box; inputting the image containing the serial number label into the multimodal recognition model, using the multimodal recognition model to perform semantic recognition on the image content corresponding to each serial number label, filtering out candidate boxes containing data from the multiple candidate boxes, and adding semantic labels, bounding box information, and confidence scores to the filtered candidate boxes to obtain candidate regions; and filtering and / or fine-tuning the candidate regions to obtain the target region.

[0042] Understandably, a segmentation model can be a deep learning model used for instance segmentation or object detection in an image, such as SAM (Segment Anything Model). A segmentation model can separate visual elements (such as text boxes, buttons, icons, tables, etc.) in an image and generate a bounding box, or candidate box, for each segmented visual element. After obtaining multiple candidate boxes, a numbered label, such as ①, ②, ③, etc., is superimposed on each candidate box on the image, giving each candidate box a unique visual identifier. This label is used by subsequent multimodal recognition models for visual localization and reference.

[0043] The image, overlaid with sequence marks, is input into a multimodal recognition model. This model possesses visual localization and semantic understanding capabilities. Based on the image content corresponding to each sequence mark, the model understands the semantic information carried by each region. The multimodal recognition model performs semantic recognition on the image content corresponding to each sequence mark, determining whether the region contains extractable business data, such as numbers, text, or tables. For candidate boxes containing data content, the model filters them out, assigns them semantic labels, and outputs the bounding box information and confidence score of the region. The candidate boxes, filtered and semantically labeled by the multimodal model, become candidate regions. Each candidate region contains three elements: semantic label, bounding box, and confidence score.

[0044] After the multimodal recognition model outputs candidate regions, these regions can be filtered and / or fine-tuned through automatic screening or manual confirmation and fine-tuning to obtain the final target region. Automatic screening methods include filtering by confidence threshold, eliminating candidate regions with confidence levels below a preset threshold. Manual fine-tuning involves the user manually adjusting the position and size of the candidate region's bounding box on a visual interface, as well as confirming or modifying the semantic labels to ensure the accuracy of the target region's location and semantic annotation. The target region obtained after screening and / or fine-tuning serves as the final recognition target for the video signal source.

[0045] In this embodiment, the combination of a segmentation model and a multimodal recognition model enables automatic identification of data regions from a purely visual image. The segmentation model is responsible for visual element segmentation, cutting the image into independent objects; the multimodal recognition model is responsible for semantic understanding and filtering, identifying the region that truly carries business data from numerous visual objects. Simultaneously, through filtering and fine-tuning, the accuracy of the target region is further ensured.

[0046] This application discloses a specific data verification method. See [link to relevant documentation] Figure 3 As shown, after obtaining structured data and outputting it to the visualization system, the following steps are also included: S31: Validate the structured data according to the numerical range constraints, and / or unit constraints, and / or output format constraints in the prompt words.

[0047] While multimodal recognition models offer high accuracy in identifying target region images, they may still exhibit recognition bias or format errors in certain cases. To ensure the reliability of the output data, the structured data obtained through recognition needs to be validated against pre-defined constraints in the prompts. For example, whether the numerical values ​​of the recognition results are within the preset range constraints; whether the units of the recognition results meet expectations; and whether the output format of the recognition results satisfies the format requirements specified in the prompts. If all the above checks pass, the recognition result is deemed valid and directly output to the visualization system; otherwise, it is not output temporarily, and a prompt for manual review is generated.

[0048] S32: If the verification fails, the target area image is enlarged to obtain a new area image, and the new area image is re-identified and verified according to the prompt word.

[0049] If the structured data fails validation, for example, if the recognition result value exceeds the preset range or the format does not meet expectations, the first-level rollback process is triggered. In this step, the target region image is enlarged by a certain proportion based on the original boundary, such as by 1.2 times, to obtain a new region image. Then, this new region image is re-identified and validated based on the same prompt words. Since multimodal recognition models may be highly sensitive to boundaries when understanding image content, appropriately enlarging the region range can sometimes provide more contextual information, helping the model to extract target data more accurately.

[0050] S33: If the re-verification fails, the video frame corresponding to the target area image is re-identified and verified according to the prompt word.

[0051] If the first-level backtracking still fails the verification, the second-level backtracking process is triggered. In this step, instead of recognizing only the target area image, the entire frame containing the target area image is input into the multimodal recognition model. The model then searches and recognizes the target data across the entire frame based on the prompt words. Full-frame backtracking is suitable for situations where the target area's location has shifted or the area's boundaries have changed significantly. If the full-frame backtracking recognition and verification pass, the re-recognized structured data is output. It is evident that by using tiered backtracking, when conventional fixed-point recognition fails, the process expands from area recognition to full-frame recognition, improving recognition stability and reliability, and preventing data acquisition interruptions due to a single recognition failure.

[0052] This application also discloses a specific data identification method based on variability, see [link to relevant documentation]. Figure 4 As shown, before recognizing the target region image using a multimodal recognition model based on the prompt words in the target region configuration, the following steps are also included: Step S41: Calculate the degree of change between the target region image in the current video frame and the target region image of the same region in the previous video frame.

[0053] The degree of change represents the extent to which the image content of a target region changes between two adjacent frames. It should be noted that these two adjacent frames refer to adjacent frames after frame extraction, not two temporally adjacent frames in the video. During runtime, the device extracts frames from the video signal source at a certain frame rate. For each target region, at least the image of the target region from the previous frame is saved as a reference. When a new video frame is acquired, an image of the same target region location is cropped from the new frame according to the region information in the target region configuration, and then compared with the reference image saved from the previous frame. The degree of change can be calculated using various image similarity measurement methods, such as pixel difference ratio (i.e., the proportion of corresponding pixel differences between two frames exceeding a given tolerance), Structural Similarity Index Measure (SSIM), or Hamming distance (the number of different bits between two hash values) from Perceptual Hash (pHash algorithm). The specific algorithm is not limited in this embodiment.

[0054] Step S42: If the degree of change exceeds a preset degree of change threshold, then perform the operation of recognizing the target region image using a multimodal recognition model based on the prompt words configured in the target region.

[0055] After calculating the degree of change, it is compared with a preset degree of change threshold. If the degree of change does not exceed the preset degree of change threshold, it is determined that the image content of the target area has not changed substantially, and the numerical data of this area is the same as the previous recognition result, so there is no need to repeat the recognition; the previous recognition result can be used. If the degree of change exceeds the preset degree of change threshold, it is determined that the image content of the target area has changed substantially, that is, the data may have been updated, and re-recognition is required.

[0056] The preset variability threshold can be set according to the data characteristics of the target area; that is, the threshold can be different for different target areas. For example, a lower threshold can be used for real-time power areas that change frequently to increase sensitivity; a higher threshold can be used for relatively stable text labels to avoid invalid triggering.

[0057] It is evident that by triggering data recognition based on variability, recognition is avoided for every frame of the image. Recognition is only performed when the content of the target area changes, thereby further reducing the consumption of ineffective computing power.

[0058] Furthermore, determining the target region image from the video frames of the current video signal source may include: statistically analyzing the change frequency of each target region in each video signal source; adjusting the recognition frequency of the video signal source according to the change frequency; wherein the change frequency and the recognition frequency are positively correlated; extracting the target video frame from the current video signal source according to the recognition frequency, and determining the target region image from the target video frame.

[0059] It should be noted that when a device simultaneously receives multiple video signal sources, the data change frequencies of each signal source often differ: some signal sources experience frequent data changes, thus requiring higher acquisition frequencies to maintain real-time data transmission; while other signal sources exhibit low data change frequencies, eliminating the need for high-frequency acquisition. Therefore, by dynamically adjusting the recognition frequency of each signal source based on its actual change frequency and that of each target area, timely identification of multiple signal sources can be achieved with limited end-side computing power.

[0060] Specifically, the number of times each target area is triggered for recognition and the number of times it is not triggered are recorded in real time. The number of changes in each target area per unit time is counted to obtain the change frequency. For example, if a target area is recognized 12 times in the past minute, then the change frequency of that target area is 0.2 times / second. Based on the statistically obtained change frequency, the recognition frequency corresponding to the video signal source is dynamically adjusted. For signal sources with a high change frequency, the frame skipping interval is shortened to increase the recognition frequency; for signal sources with a low change frequency, the frame skipping interval is lengthened to decrease the recognition frequency, so that computing power is allocated to signal sources with more frequent changes. There is a positive correlation between change frequency and recognition frequency, that is, the higher the change frequency, the higher the recognition frequency, and the lower the change frequency, the lower the recognition frequency.

[0061] After determining the recognition frequency of each signal source, the acquisition interfaces of multiple video signal sources are polled and frame-triggered according to the recognition frequency of each signal source. Video frames are extracted from the corresponding signal source, and then the target area image is determined from the extracted video frames according to the target area configuration of the signal source, and subsequent recognition operations are performed.

[0062] The aforementioned video signal-based data recognition method can be applied to devices with signal acquisition hardware, such as embedded devices or industrial control computers. The device includes: a signal acquisition module, which has one or more video signal acquisition interfaces, each assigned a unique signal source ID for pairing the data generated by that signal source and for taking screenshots / frame extraction; a storage module, used to save the target area configuration corresponding to each signal source ID; and a multimodal recognition module, which runs a large multimodal model, supporting local models on the device side or calling cloud models. The device outputs the results to an external visualization / data collection system via a network and an MCP / tool ​​call interface.

[0063] During the data preparation phase, full-screen recognition is performed to identify target areas and construct target area configurations.

[0064] Before formal data acquisition begins, it's crucial to determine the locations, data types, and output formats for a given video signal. Specifically, the signal acquisition module first assigns a unique source identifier (sid) to each incoming video signal source and captures the entire screen of that source. The multimodal recognition module then identifies the entire screen, outputting candidate regions that could potentially carry data. To improve positioning accuracy, auxiliary annotation methods such as Set-of-Mark can be used to first mark candidate boxes, which are then named by the model. Candidate regions are then selected based on confidence thresholds and confirmed and fine-tuned using automatic labeling or manual specification to obtain the final target regions. Each target region includes a semantic label, bounding box coordinates, and confidence level. For each target region, corresponding prompts are generated based on its semantics, unit, numerical range, and output format, and associated with the source identifier. These prompts are persistently saved by the storage module, forming a unique target region configuration for that source. One source may correspond to multiple target region configurations. Once established, the target region configuration can be reused long-term, only being updated when the screen layout changes.

[0065] During the data acquisition phase, full-screen recognition is no longer performed; instead, targeted recognition of the target area image is performed based on prompts. Specifically, upon device startup, the target area configurations of each signal source are first read into memory by the storage module for high-speed matching. A continuous loop then begins: the scheduler polls multiple acquisition ports based on the frame extraction interval (Δt) and priority of each area, thus balancing multiple signal sources with limited edge computing power. In each round, the signal acquisition module extracts a frame from the currently selected signal source and, based on the coordinates in the signal source configuration, crops sub-images corresponding to each target area. For each sub-image, the system first calls the change degree subroutine to compare it with the previous reference sub-image; if the content has not changed substantially, the current recognition is skipped, and the previous result is used directly to avoid unnecessary recalculation. Only when the content changes does the system send the target area image along with its prompts to the multimodal recognition module for targeted recognition; since only a local sub-image is sent to the model instead of the entire image, the computational cost and latency of a single recognition are significantly reduced. The identified results are verified. Data that passes verification is output by the output module in a preset format, either via MCP or a tool call interface, to an external visualization or data collection system. If verification fails, a tiered rollback is triggered, such as enlarging the cropped area for re-identification, rolling back the entire frame, or prompting a restart of the preparation phase if necessary. Simultaneously, the system dynamically adjusts the frame sampling interval based on the recent actual change frequency of each region, providing a higher refresh rate for frequently changing areas and a lower sampling frequency for areas with gradual changes.

[0066] This allows the high-consumption full-screen understanding and recognition to be completed in one go during the preparation stage, while the acquisition stage only requires low-computation fixed-point recognition. This significantly reduces the computing power consumption on the device side and ensures controllable latency. It can be deployed on the device side and supports polling recognition from multiple signal sources.

[0067] Take a smart industrial park integrated management and control screen as an example. A command center in an industrial park needs to build a comprehensive visualization screen that aggregates two sets of data: ① an early energy consumption monitoring system (this system has a C / S architecture, is no longer maintained by the original vendor, and has no external API); ② a security video management platform (only has a web interface, with no open API). Traditionally, this would require the vendor to develop the interface, which is practically impossible.

[0068] The method employs an embedded recognition device to perform data recognition based on video signals. This device has four HDMI input acquisition channels. Fixed-point recognition during the acquisition phase is completed by a local multimodal model on the device side. Full-screen recognition during the preparation phase can call a cloud-based multimodal model. It connects to a large-screen visualization system via a gigabit Ethernet port and an MCP interface.

[0069] Signal input: HDMI-1, connect to the display output of the energy consumption monitoring system workstation (1920×1080), signal source ID=SRC-ENERGY; HDMI-2, connect to the display output of the security platform (1920×1080), signal source ID=SRC-SECURITY.

[0070] Taking SRC-ENERGY as an example, during the data preparation stage: the device assigns SRC-ENERGY to HDMI-1; a full-screen frame is captured and sent to the cloud multimodal model for full-screen recognition, and the maintenance personnel confirm and fine-tune the required target area in the configuration interface. Table 1 below shows four example target areas: Table 1: Target Area Diagram

[0071] Each region is converted into a prompt word and associated with SRC-ENERGY for storage. For example: R1: "Identify the numbers within the coordinate (1520, 120, 360, 80) region, which represent the real-time total power in kW, with the output value rounded to one decimal place"; R3: "Identify this table region, output {building name, current power (kW)} line by line, and return it as a JSON array." This forms the target region configuration of SRC-ENERGY, which is then stored in the storage module.

[0072] During the data acquisition phase, after the device starts up, it reads all SRC-ENERGY and SRC-SECURITY configurations from the storage module and loads them into memory; it extracts one frame every 2 seconds for HDMI-1 and matches the R1–R4 prompt words according to SRC-ENERGY; it sends the target area image, signal source ID, and prompt words to the end-side model for fixed-point recognition to obtain structured data, such as: { "SRC-ENERGY": { Real-time total power (kW): 1284.6 Today's electricity consumption (kWh): 21560.0 Current electricity price: 0.82 Building power consumption: [{"Building A": 312.4}, {"Building B": 289.1}, {"Building C": 203.7}] } Finally, the data is output to the visualization dashboard system via the MCP interface according to the preset schema. The dashboard updates components such as power curves and electricity consumption rankings in real time. SRC-SECURITY also outputs the number of online devices / alarms and displays them on the screen. When the energy consumption system interface is redesigned and the frame position changes, only the preparation stage of SRC-ENERGY needs to be redone, and the SRC-SECURITY configuration remains unaffected. Since the device has two sets of configurations saved locally, daily multi-source identification and switching does not require reconfiguration.

[0073] Full-screen recognition is performed only once per channel during the preparation phase; during the acquisition phase, only fixed-point recognition is performed. The single-frame inference time on the edge side is significantly lower than that of full-screen recognition. Four channels of 1080p can run continuously on this embedded device, ensuring that the update latency for each signal change is controlled within 500ms-1500ms. No business interfaces / databases are connected throughout the process, enabling data from two legacy systems to be displayed on the screen; interface changes trigger push notifications, and the real-time data meets the requirements of the command and control dashboard.

[0074] Accordingly, this application also discloses a data recognition device based on video signals, see [link to relevant documentation]. Figure 5 As shown, the device includes: The target area configuration acquisition module 11 is used to acquire the target area configuration corresponding to the current video signal source; the target area configuration is obtained by pre-identifying the target area by recognizing the image of the video signal source, and storing the area information of the target area and the prompt words used to extract data from the target area together; The target region image determination module 12 is used to determine the target region image from the video frames of the current video signal source according to the target region configuration; The data recognition module 13 is used to recognize the target region image using a multimodal recognition model based on the prompt words in the target region configuration, obtain structured data, and output it to the visualization system.

[0075] In some specific embodiments, the device further includes: The target region configuration generation module is used to assign a unique signal source identifier to each video signal source before obtaining the target region configuration corresponding to the current video signal source; perform image recognition on the video signal source to determine the target region in the image of the video signal source; associate the region information of the target region with the prompt words used to extract data from the target region to obtain the target region configuration corresponding to the video signal source; and construct a mapping relationship between the signal source identifier and the target region configuration.

[0076] In some specific embodiments, the target region configuration generation module is used to segment the video signal source using a segmentation model to obtain multiple candidate boxes and add a sequence number to each candidate box; input the image containing the sequence number to the multimodal recognition model, and use the multimodal recognition model to perform semantic recognition on the image content corresponding to each sequence number, filter out candidate boxes with data from the multiple candidate boxes, and add semantic labels, bounding box information and confidence scores to the filtered candidate boxes to obtain candidate regions; filter and / or fine-tune the candidate regions to obtain the target region.

[0077] In some specific embodiments, the prompt word includes at least one of the following: coordinate information of the target area, semantic description, numerical range constraints, unit constraints, and output format constraints.

[0078] In some specific embodiments, the device further includes: The verification module is used to verify the structured data after it has been obtained and output to the visualization system, based on the numerical range constraints, and / or unit constraints, and / or output format constraints in the prompt words. If the verification fails, the target area image is enlarged to obtain a new area image, and the new area image is re-identified and verified based on the prompt words. If the re-verification fails, the video frames corresponding to the target area image are re-identified and verified based on the prompt words.

[0079] In some specific embodiments, the device further includes: The variability calculation module is used to calculate the variability between the target region image in the current video frame and the target region image in the same region in the previous video frame before recognizing the target region image using a multimodal recognition model based on the prompt words configured in the target region; if the variability exceeds a preset variability threshold, then the operation of recognizing the target region image using a multimodal recognition model based on the prompt words configured in the target region is executed.

[0080] In some specific embodiments, the target region image determination module 12 can be used to: count the change frequency of each target region in each video signal source; adjust the recognition frequency of the video signal source according to the change frequency; wherein the change frequency and the recognition frequency are positively correlated; extract the target video frame from the current video signal source according to the recognition frequency, and determine the target region image from the target video frame.

[0081] Furthermore, this application also discloses an electronic device, see [link to relevant documentation]. Figure 6 As shown, the content in the figure should not be considered as any limitation on the scope of use of this application.

[0082] Figure 6 This is a schematic diagram of the structure of an electronic device 20 provided in an embodiment of this application. The electronic device 20 may specifically include: at least one processor 21, at least one memory 22, a power supply 23, a communication interface 24, an input / output interface 25, and a communication bus 26. The memory 22 stores a computer program, which is loaded and executed by the processor 21 to implement the relevant steps in the video signal-based data recognition method disclosed in any of the foregoing embodiments.

[0083] In this embodiment, the power supply 23 is used to provide operating voltage for each hardware device on the electronic device 20; the communication interface 24 can create a data transmission channel between the electronic device 20 and external devices, and the communication protocol it follows can be any communication protocol applicable to the technical solution of this application, and is not specifically limited here; the input / output interface 25 is used to acquire external input data or output data to the outside world, and its specific interface type can be selected according to specific application needs, and is not specifically limited here.

[0084] In addition, the memory 22, as a carrier for resource storage, can be a read-only memory, random access memory, disk or optical disk, etc. The resources stored thereon include operating system 221, computer program 222 and data 223 including target area configuration, etc. The storage method can be temporary storage or permanent storage.

[0085] The operating system 221 manages and controls the various hardware devices on the electronic device 20 and the computer program 222 to enable the processor 21 to perform calculations and processing on the massive amounts of data 223 in the memory 22. The operating system 221 can be Windows Server, Netware, Unix, Linux, etc. In addition to including a computer program capable of performing the video signal-based data recognition method executed by the electronic device 20 as disclosed in any of the foregoing embodiments, the computer program 222 may further include computer programs capable of performing other specific tasks.

[0086] Furthermore, this application also discloses a computer storage medium storing computer-executable instructions. When the computer-executable instructions are loaded and executed by a processor, they implement the data recognition method steps based on video signals disclosed in any of the foregoing embodiments.

[0087] Furthermore, this application also discloses a computer program product, including a computer program that, when executed by a processor, implements the data recognition method steps based on video signals disclosed in any of the foregoing embodiments.

[0088] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatus disclosed in the embodiments, since it corresponds to the method disclosed in the embodiments, the description is relatively simple; relevant parts can be referred to in the method section.

[0089] The steps of the methods or algorithms described in conjunction with the embodiments disclosed herein can be implemented directly by hardware, a software module executed by a processor, or a combination of both. The software module can be located in random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, hard disk, removable disk, CD-ROM, or any other form of storage medium known in the art.

[0090] Finally, it should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0091] The foregoing has provided a detailed description of a data recognition method, apparatus, device, and medium based on video signals provided by the present invention. Specific examples have been used to illustrate the principles and implementation methods of the present invention. The descriptions of the above embodiments are only for the purpose of helping to understand the method and core ideas of the present invention. At the same time, for those skilled in the art, there will be changes in specific implementation methods and application scope based on the ideas of the present invention. Therefore, the content of this specification should not be construed as a limitation of the present invention.

Claims

1. A data recognition method based on video signals, characterized in that, include: Get the target area configuration corresponding to the current video signal source; The target area is configured to be determined in advance by identifying the image of the video signal source, and the area information of the target area and the prompt words used to extract data from the target area are associated and stored. According to the target region configuration, the target region image is determined from the video frames of the current video signal source; Based on the prompt words in the target region configuration, the target region image is identified using a multimodal recognition model to obtain structured data and output it to the visualization system.

2. The data recognition method based on video signals according to claim 1, characterized in that, Before obtaining the target region configuration corresponding to the current video signal source, the method further includes: Assign a unique signal source identifier to each video signal source; The video signal source is subjected to image recognition to determine the target area in the image of the video signal source; The target area information is associated with the prompt words used to extract data from the target area to obtain the target area configuration corresponding to the video signal source; Construct a mapping relationship between the signal source identifier and the target area configuration.

3. The data recognition method based on video signals according to claim 2, characterized in that, The step of performing image recognition on the video signal source to determine the target area in the image of the video signal source includes: The video signal source is segmented using a segmentation model to obtain multiple candidate boxes, and a serial number is added to each candidate box. The image containing the serial number is input into the multimodal recognition model. The multimodal recognition model performs semantic recognition on the image content corresponding to each serial number, filters out the candidate boxes containing data from the multiple candidate boxes, and adds semantic labels, bounding box information and confidence scores to the filtered candidate boxes to obtain candidate regions. The candidate regions are filtered and / or fine-tuned to obtain the target region.

4. The data recognition method based on video signals according to claim 1, characterized in that, The prompt word includes at least one of the following: coordinate information of the target area, semantic description, numerical range constraints, unit constraints, and output format constraints.

5. The data recognition method based on video signals according to claim 4, characterized in that, After obtaining the structured data and outputting it to the visualization system, the process also includes: The structured data is validated based on the numerical range constraints, and / or unit constraints, and / or output format constraints in the prompt words; If the verification fails, the target region image is enlarged to obtain a new region image, and the new region image is re-identified and verified according to the prompt words; If the re-verification fails, the video frames corresponding to the target area image will be re-identified and verified according to the prompt words.

6. The data recognition method based on video signals according to any one of claims 1 to 5, characterized in that, Before recognizing the target region image using a multimodal recognition model based on the prompt words configured in the target region, the method further includes: Calculate the degree of change between the target region image in the current video frame and the target region image of the same region in the previous video frame; If the degree of change exceeds a preset degree of change threshold, then the operation of recognizing the target region image using a multimodal recognition model based on the prompt words configured in the target region is executed.

7. The data recognition method based on video signals according to claim 6, characterized in that, Determining the target region image from the video frames of the current video signal source includes: Statistical analysis of the frequency of change of each target region in each video signal source; The recognition frequency of the video signal source is adjusted according to the changing frequency; wherein the changing frequency and the recognition frequency are positively correlated. The target video frame is extracted from the current video signal source according to the identification frequency, and the target region image is determined from the target video frame.

8. A data recognition device based on video signals, characterized in that, include: The target area configuration acquisition module is used to acquire the target area configuration corresponding to the current video signal source; The target area is configured to be determined in advance by identifying the image of the video signal source, and the area information of the target area and the prompt words used to extract data from the target area are associated and stored. The target region image determination module is used to determine the target region image from the video frames of the current video signal source according to the target region configuration; The data recognition module is used to recognize the target region image based on the prompt words in the target region configuration using a multimodal recognition model, obtain structured data, and output it to the visualization system.

9. An electronic device, characterized in that, include: Memory, used to store computer programs; A processor for executing the computer program to implement the data recognition method based on video signals as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, Used to store computer programs; wherein the computer programs, when executed by a processor, implement the data recognition method based on video signals as described in any one of claims 1 to 7.

Images