Data information collection method based on visual recognition and related device

Abstract

The application belongs to the technical field of big data, and relates to a data information collection method based on visual recognition and related equipment, which receives a target read-only screenshot instruction; in response to the target read-only screenshot instruction, performs read-only screenshot processing on a to-be-screenshot information page to obtain a read-only picture after screenshot; calibrates all target recognition regions contained in all read-only pictures; extracts data information in all calibrated target recognition regions to complete data information collection. The method can be applied to a medical institution or a hospital digital service process, for example, an intelligent digital case generation scene, to replace manual automation to perform read-only screenshot processing on a detection report, automatically collect data information by using an artificial intelligence model processing mode on text data and picture data of the obtained read-only screenshot detection report picture, and quickly extract page display content.

Description

Technical Field

[0001] This application relates to the field of big data technology and is applied to scenarios involving the collection of page display content. It involves data information collection methods and related equipment based on visual recognition. Background Technology

[0002] Currently, with the development of artificial intelligence, machine learning technology is being successfully applied in more and more medical fields, such as telemedicine, intelligent case generation, and robot consultation.

[0003] In the field of digital healthcare, intelligent case generation is a common business scenario. For example, patent application CN202411267641.3 discloses a visual information extraction method based on a multimodal large model. This method involves fine-tuning the multimodal large model within a specific domain, allowing the user to upload a target image and input text prompts. The multimodal large model then generates preliminary extraction results, which are combined with optical recognition results to generate the final extracted information. However, this method merely collects and processes data based on multimodal and optical extraction, falling far short of the required higher data extraction efficiency. Therefore, improving the extraction efficiency of page display content has become a pressing issue. Summary of the Invention

[0004] The purpose of this application is to propose a data information acquisition method and related equipment based on visual recognition, so as to improve the efficiency of extracting page display content.

[0005] Firstly, embodiments of this application provide a data information acquisition method based on visual recognition, which adopts the following technical solution: The data acquisition method based on visual recognition includes the following steps: Receive a target read-only screenshot instruction, wherein the target read-only screenshot instruction provides the page address information and page rendering status information of the page to be screenshotted; In response to the target read-only screenshot instruction, the page to be screenshotted is processed for read-only screenshotting to obtain a read-only image after screenshotting; All read-only images are input into a pre-trained target recognition region extraction model to identify all target recognition regions contained in all read-only images; The multimodal data acquisition component in the target recognition region calibration model is used to extract data information from all calibrated target recognition regions, thus completing the data information acquisition.

[0006] Secondly, embodiments of this application also provide a data acquisition device based on visual recognition, which adopts the following technical solution: The data acquisition device based on visual recognition includes: A screenshot command receiving module is used to receive a target read-only screenshot command, wherein the target read-only screenshot command provides the page address information and page rendering status information of the page to be screenshotted; The read-only screenshot processing module is used to respond to the target read-only screenshot instruction, perform read-only screenshot processing on the information page to be screenshotted, and obtain a read-only image after the screenshot is taken. The recognition region labeling module is used to input all read-only images into the pre-trained target recognition region extraction model and label all target recognition regions contained in all read-only images; The multimodal data acquisition module is used to extract data information from all calibrated target recognition regions using the multimodal data acquisition component in the target recognition region calibration model, thereby completing the data information acquisition.

[0007] Thirdly, embodiments of this application also provide a computer device that adopts the technical solution described below: A computer device includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the data information acquisition method based on visual recognition described above.

[0008] Fourthly, embodiments of this application also provide a computer-readable storage medium, which adopts the technical solutions described below: A computer-readable storage medium storing computer-readable instructions, which, when executed by a processor, implement the steps of the visual recognition-based data acquisition method described above.

[0009] Compared with the prior art, the embodiments of this application have the following main advantages: The visual recognition-based data acquisition method described in this application involves receiving a target read-only screenshot instruction; responding to the target read-only screenshot instruction, performing read-only screenshot processing on the page to be screenshotted to obtain a read-only image after screenshotting; inputting all read-only images into a pre-trained target recognition region extraction model to label all target recognition regions contained in all read-only images; and using the multimodal data acquisition component in the target recognition region labeling model to extract data information from all labeled target recognition regions, thus completing the data acquisition. Applying this method to the digital service processes of medical institutions or hospitals, such as in the intelligent digital medical record generation scenario, can replace manual processing of read-only screenshots of test reports. For the text and image data in the obtained read-only screenshot test report images, an automated artificial intelligence model is used to collect data information, quickly extract the content to be displayed on the page, and then automatically fill it into the medical record generation template to generate the final medical record sent to the doctor for review. After the doctor's review, it is intelligently forwarded to the patient's mobile phone. This effectively avoids typos that occur when manually generating medical records and improves the efficiency of sending medical records to patients in large-scale scenarios with many patients and a large volume of medical records. Attached Figure Description

[0010] To more clearly illustrate the solutions in this application, the accompanying drawings used in the description of the embodiments of this application will be briefly introduced below. Obviously, the accompanying drawings described below are some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0011] Figure 1 This is an exemplary system architecture diagram to which this application can be applied; Figure 2 This is a flowchart of an embodiment of the visual recognition-based data acquisition method according to this application; Figure 3 yes Figure 2 The flowchart shown is a specific embodiment of a visual recognition-based data acquisition method for optimizing read-only screenshot knowledge for different pages of information to be screenshotted. Figure 4 yes Figure 3 A flowchart of a specific embodiment of step 304 shown; Figure 5 yes Figure 4 A flowchart of a specific embodiment of step 402 shown; Figure 6 yes Figure 4 A flowchart of a specific embodiment of step 403 shown; Figure 7 yes Figure 2 A flowchart of a specific embodiment of step 202 shown; Figure 8 yes Figure 7 A flowchart of a specific embodiment of step 705 shown; Figure 9 This is a schematic diagram of the structure of an embodiment of the visual recognition-based data acquisition device according to this application; Figure 10 This is a schematic diagram of the structure of one embodiment of the computer device according to this application. Detailed Implementation

[0012] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application pertains; the terminology used herein in the specification of the application is for the purpose of describing particular embodiments only and is not intended to be limiting of the application; the terms "comprising" and "having," and any variations thereof, in the specification, claims, and foregoing drawings of this application, are intended to cover non-exclusive inclusion. The terms "first," "second," etc., in the specification, claims, or foregoing drawings of this application are used to distinguish different objects, not to describe a particular order.

[0013] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a separate or alternative embodiment mutually exclusive with other embodiments. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0014] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings.

[0015] like Figure 1 As shown, system architecture 100 may include terminal device 101, network 102, and server 103. Terminal device 101 may be a laptop 1011, tablet 1012, or mobile phone 1013. Network 102 is used as a medium to provide a communication link between terminal device 101 and server 103. Network 102 may include various connection types, such as wired, wireless communication links, or fiber optic cables.

[0016] Users can use terminal device 101 to interact with server 103 via network 102 to receive or send messages, etc. Various communication client applications can be installed on terminal device 101, such as web browser applications, shopping applications, search applications, instant messaging tools, email clients, social media platform software, etc.

[0017] Terminal device 101 can be various electronic devices with a display screen and support web browsing. In addition to laptops 1011, tablets 1012, or mobile phones 1013, terminal device 101 can also be an e-book reader, an MP3 player (Moving Picture Experts Group Audio Layer III), an MP4 player (Moving Picture Experts Group Audio Layer IV), a laptop computer, and a desktop computer, etc.

[0018] Server 103 can be a server that provides various services, such as a backend server that provides support for the pages displayed on terminal device 101.

[0019] It should be noted that the visual recognition-based data acquisition method provided in this application is generally executed by a server, and correspondingly, the visual recognition-based data acquisition device is generally located in the server.

[0020] It should be understood that Figure 1 The number of terminal devices, networks, and servers shown is merely illustrative. Depending on implementation needs, any number of terminal devices, networks, and servers can be included.

[0021] Continue to refer to Figure 2 The diagram illustrates a flowchart of an embodiment of a visual recognition-based data acquisition method according to this application. The visual recognition-based data acquisition method includes the following steps: Step 201: Receive the target read-only screenshot command.

[0022] The target read-only screenshot instruction provides the page address information and page rendering status information of the page to be screenshotted.

[0023] Specifically, the target read-only screenshot instruction means that only a read-only screenshot of the page to be captured is allowed, and no other modification operations can be performed on the page to be captured. Other modification operations include, for example, human-computer interaction operations that modify the page data in the page to be captured.

[0024] By receiving a target read-only screenshot command, and providing the page address and rendering status information of the page to be screenshotted within the target read-only screenshot command, it is ensured that subsequent read-only screenshot processing of the page to be screenshotted can be accurately performed based on the page address information of the page to be screenshotted. This avoids introducing other operations during the screenshot process and prohibits the introduction of dirty data into the screenshot.

[0025] In this embodiment, the read-only screenshot instruction means that when the instruction is executed, only read-only and screenshot processing of the target page is allowed. The read-only image obtained after the screenshot processing can only be used for content recognition and extraction operations. Data modification operations on the content contained in the read-only image itself are not allowed. After obtaining the read-only image from the screenshot, a copy of the read-only image can be created to facilitate subsequent image recognition. The copy image can then be processed using a binary method or grayscale processing to obtain the pre-processed image corresponding to the read-only image.

[0026] In this embodiment, the target read-only screenshot command is generally sent by the display terminal where the information page to be screenshotted is located. For example, after the hospital patient information management terminal (PMS) sends the patient information to the target display terminal, the target display terminal sends the target read-only screenshot command to the preset screenshot processing component. At this time, the target display terminal includes the doctor's computer display terminal, the self-service interface terminal in the medical hall, etc.

[0027] In this embodiment, the visual recognition-based data collection method is applied to the digital service process of medical institutions or hospitals. It can not only replace manual data collection, but also avoid the errors that are prone to occur when manually inputting data, thereby improving the efficiency and accuracy of data collection.

[0028] Step 202: In response to the target read-only screenshot instruction, perform read-only screenshot processing on the page to be screenshotted to obtain a read-only image after screenshotting.

[0029] In this embodiment, responding to the target read-only screenshot instruction includes performing a read-only screenshot operation on the page to be screenshotted according to the target read-only screenshot instruction.

[0030] Specifically, for example: first, parse the target read-only screenshot instruction, extract the page address information and page rendering state information of the page to be screenshotted, and use the page address information of the page to be screenshotted as the target address information for the screenshot operation.

[0031] The page rendering status information here can be used as a prerequisite for taking a screenshot, ensuring that a read-only screenshot is performed when the page rendering status is normal, and obtaining a read-only image after the screenshot. This avoids performing a read-only screenshot when the page rendering status is abnormal, such as when there is a black screen or a white screen.

[0032] Step 203: Input all read-only images into the pre-trained target recognition region extraction model and label all target recognition regions contained in all read-only images.

[0033] In this embodiment, all read-only images obtained by screenshotting are input into a pre-trained target recognition region extraction model. The model uses artificial intelligence to calibrate all target recognition regions in all read-only images, which facilitates efficient subsequent data collection.

[0034] Specifically, the target recognition region can be understood as the region for which data information needs to be collected, corresponding to different screenshot information pages, learned during the target recognition region extraction model training process. The target recognition region includes two main types: text content regions and image content regions.

[0035] In this embodiment, the target recognition region extraction model, which has been pre-trained, is used to mark the target recognition region of all read-only images according to the regions for which data information needs to be collected for different screenshot information pages that it has learned. This makes it easier to collect data information directly based on the marked range and improves the efficiency of data information collection.

[0036] Step 204: Using the multimodal data acquisition component in the target recognition region calibration model, extract data information from all calibrated target recognition regions to complete data information acquisition.

[0037] In this embodiment, the multimodal data acquisition component in the target recognition region calibration model includes at least a multimodal data acquisition component capable of acquiring text data and image data, which can acquire text data and image data in the screenshot image to complete data information acquisition.

[0038] In this embodiment, when the multi-modal data acquisition component is used to extract text data and image data from the screenshot-captured image, the context semantic understanding and cross-modal semantic analysis and fusion capabilities of the multi-modal data acquisition component itself can be utilized to perform error correction or completion processing on the extracted text data and image data. Specifically, when the multi-modal data acquisition component is subjected to region learning for data information acquisition corresponding to different information pages to be screenshot-captured, recognition and correction learning for common misspelled fields corresponding to different information pages to be screenshot-captured may also be introduced. The common misspelled fields include omitted characters caused by writing habits in the medical field, for example, "xi pao", "xi bao", abbreviated as "C pao", "C bao", etc.

[0039] Specifically, when the data information acquisition method based on visual recognition is applied to the digital service process of a medical institution or a hospital, for example, in an intelligent digital medical record generation scenario, it can replace manual work to automatically perform read-only screenshot processing on test reports. For the text data and image data in the obtained read-only screenshot test report image, after data information acquisition is performed, the data is automatically filled into a medical record generation template to generate a final medical record to be sent to a doctor for review. After the doctor completes the review, the medical record is intelligently forwarded to the patient's mobile terminal, which can not only effectively avoid the occurrence of misspelled characters when generating medical records by manual typing, but also improve the efficiency of distributing medical record reports to patients in large-data-scale scenarios with a large number of patients and a large medical record processing volume.

[0040] The data information acquisition method based on visual recognition provided in this embodiment includes: receiving a target read-only screenshot instruction; in response to the target read-only screenshot instruction, performing read-only screenshot processing on the information page to be screenshot-captured to obtain a read-only image after screenshot capture; inputting all the read-only images into a pre-trained target recognition region extraction model, and marking all target recognition regions contained in all the read-only images; and using the multi-modal data acquisition component in the target recognition region marking model to extract data information in all the marked target recognition regions, so as to complete data information acquisition. When the method is applied to the digital service process of a medical institution or a hospital, for example, in an intelligent digital medical record generation scenario, it can replace manual work to automatically perform read-only screenshot processing on test reports. For the text data and image data in the obtained read-only screenshot test report image, data information acquisition is automatically performed by means of artificial intelligence model processing, and then the data is automatically filled into a medical record generation template in a subsequent step to generate a final medical record to be sent to a doctor for review. After the doctor completes the review, the medical record is intelligently forwarded to the patient's mobile terminal, which can not only effectively avoid the occurrence of misspelled characters when generating medical records by manual typing, but also improve the efficiency of distributing medical record reports to patients in large-data-scale scenarios with a large number of patients and a large medical record processing volume.

[0041] Specifically, for example, on digital healthcare platforms with a large user base, conventional data collection methods are prone to causing server crashes and slow processing. The visual recognition-based data collection method proposed in this embodiment only requires performing read-only screenshot processing on the page to be captured to obtain read-only images. Then, all read-only images are input into a pre-trained target recognition region extraction model to label all target recognition regions contained in all read-only images. Using the multimodal data acquisition component in the target recognition region labeling model, data information is extracted from all labeled target recognition regions to complete data collection. This fully utilizes the inherent correlation between different target labeling regions and different page address information learned by the pre-trained target recognition region extraction model, as well as the multi-level screenshot strategies corresponding to different page address information. This enables rapid screenshot strategy invocation and labeling region determination based on page address information, allowing for fast and efficient subsequent data collection. This improves the recognition and acquisition efficiency of related data acquisition processors and avoids server crashes due to processing pressure or processing delays.

[0042] In this embodiment, after obtaining the read-only image after taking a screenshot, the read-only image can be cached in a preset queryable image library to facilitate subsequent retrieval of the read-only image. The preset queryable image library is connected to the target image query and retrieval end in an end-to-end bridging mode. For example, the preset queryable image library includes an image caching service library based on the standard DICOM Modality Worklist. The image caching service library based on the standard DICOM Modality Worklist can be connected to the target image query and retrieval end in an end-to-end bridging mode, which can quickly realize the storage and retrieval of images in the library.

[0043] Continue to refer to Figure 3 Before step 202, Figure 3 It provides steps for optimizing read-only screenshot knowledge for different pages containing different information. Figure 3 yes Figure 2 The flowchart shown is a specific embodiment of a visual recognition-based data acquisition method for optimizing read-only screenshot knowledge for different pages to be screenshotted, including: Step 301: Pre-obtain the multi-level initial screenshot strategy set for different pages of information to be screenshotted; Specifically, when initially taking screenshots of different pages of information to be screenshotted, the multi-level initial screenshot strategy can be understood as a pre-set strategy based on human experience, i.e., obtaining the multi-level initial screenshot strategy pre-set based on human experience.

[0044] Step 302: According to the multi-level initial screenshot strategy corresponding to different information pages to be screenshotted, perform read-only screenshot processing respectively to obtain all read-only images after screenshotting corresponding to all information pages to be screenshotted. Specifically, based on a multi-level initial screenshot strategy pre-set by human experience, screenshot processing is performed on the corresponding information pages to be screenshotted, resulting in read-only images of all the screenshots for each of the information pages to be screenshotted.

[0045] Step 303: Based on the page address information corresponding to each of the pages to be screenshotted, classify and organize all the read-only images after screenshotting corresponding to each of the pages to be screenshotted, and obtain a read-only image set with the page address information as the classification identifier and the read-only images after screenshotting as the classification element. Specifically, all the read-only images after screenshots are categorized and organized according to different page address information to obtain a read-only image set with page address information as the category identifier and the read-only images after screenshots as the category elements.

[0046] Step 304: The multi-level initial screenshot strategy corresponding to different pages of information to be screenshotted is reverse-optimized through the read-only image set, and the multi-level screenshot optimization strategy corresponding to different pages of information to be screenshotted is obtained as read-only screenshot knowledge feedback to the preset read-only screenshot processing terminal.

[0047] In this embodiment, the step of reverse-optimizing the multi-level initial screenshot strategy corresponding to different pages to be screenshotted using the read-only image set can be understood as follows: Since the read-only image set compiled in steps 301 to 303 is based on a pre-set multi-level initial screenshot strategy using human experience, it is inevitable that text or image data will be segmented during screenshotting, resulting in an inappropriate screenshot range. Here, the multi-level initial screenshot strategy corresponding to different pages to be screenshotted is reverse-optimized using the read-only image set. This ensures that the subsequent read-only screenshot processing has an appropriate screenshot position range, and the final multi-level screenshot optimization strategy corresponding to different pages to be screenshotted is fed back as read-only screenshot knowledge to the preset read-only screenshot processing end. This guides the preset read-only screenshot processing end to perform subsequent screenshot processing for different pages to be screenshotted.

[0048] Continue to refer to Figure 4 , Figure 4 yes Figure 3 A flowchart of a specific embodiment of step 304 shown includes: Step 401: Input the read-only image set corresponding to all page address information into the pre-built target recognition region extraction model. The pre-built target recognition region calibration model includes a region calibration component, which includes a region calibration component based on OCR optical recognition technology or a region calibration component based on CNN convolutional neural network. Step 402: Using the region labeling component in the target recognition region extraction model, label the text data region and image data region in each read-only image; Step 403: Based on the region calibration results of the region calibration component, identify whether there are missing screenshot content in the text data region and image data region of each read-only image; Step 404: If there is missing screenshot content in the current read-only image, mark the current read-only image, and backtrack the corresponding page address information and the multi-level screenshot initial strategy adopted based on the current read-only image, and optimize and adjust the multi-level screenshot initial strategy adopted. Step 405: Until there is no missing screenshot content in the text data and image data of each read-only image, the multi-level screenshot optimization strategy corresponding to different page address information is fed back to the preset read-only screenshot processing terminal as read-only screenshot knowledge. The multi-level screenshot initial strategy and the multi-level screenshot optimization strategy respectively specify the corresponding screenshot window position and size specifications.

[0049] Specifically, steps 401 to 405 essentially involve using the region labeling component in the target recognition region extraction model to continuously iterate and optimize the initial multi-level screenshot strategy corresponding to different page address information. This ensures that no text or image data is missing from the screenshot in each read-only image. The multi-level screenshot optimization strategy corresponding to different page address information is then fed back as read-only screenshot knowledge to the preset read-only screenshot processing terminal. This ensures that subsequent screenshots using the multi-level screenshot optimization strategy corresponding to different page address information meet the text and image data labeling requirements, preventing inappropriate cropping errors.

[0050] In this embodiment, before executing the step of calibrating the text data region and image data region in each read-only image using the region calibration component in the target recognition region extraction model, i.e., before step 402, the method further includes: blanking out the non-textual and non-image data in each read-only image, retaining only the text data and image data in each read-only image. Specifically, symbols, marks, lines, watermarks, etc., in the text data or image data are blanked out to ensure the cleanliness of each read-only image before performing region calibration processing.

[0051] Continue to refer to Figure 5 , Figure 5 yes Figure 4 A flowchart of a specific embodiment of step 402 shown includes: Step 501: Using the region calibration component, identify the boundary information corresponding to all text data and image data in each read-only image, wherein the boundary information includes vertex coordinate information, center point coordinate information, and coordinate information of all points constituting the boundary region box; Step 502: Based on the boundary information corresponding to all text data and image data, mark the text data region and image data region in each read-only image.

[0052] Continue to refer to Figure 6 , Figure 6 yes Figure 4 A flowchart of a specific embodiment of step 403 shown includes: Step 601: Based on the region calibration results, determine whether both the text data region and the image data region in the current read-only image have complete calibration region boxes; Specifically, the complete calibration region box can be understood as a closed calibration region box, such as a closed rectangular calibration region box, or for example, a closed circular calibration region box.

[0053] Step 602: If both the text data region and the image data region in the current read-only image have complete bounding boxes, then there will be no missing screenshot content. Step 603: If any text data region or image data region in the current read-only image does not have a complete bounding box, then there will be missing screenshot content.

[0054] Specifically, the statement that the calibration region box is not complete, meaning that the calibration region box is not closed, can be understood as text data or image data being cut off during the screenshot processing. Therefore, it is determined that there is missing screenshot content.

[0055] In this embodiment, the pre-built target recognition region extraction model further includes a multimodal data acquisition component. After executing the step of feeding back the multi-level screenshot optimization strategies corresponding to different page address information as read-only screenshot knowledge to the preset read-only screenshot processing terminal, the method further includes: feeding back the screenshot window position and size specifications corresponding to the multi-level screenshot optimization strategies as information acquisition region knowledge to the multimodal data acquisition component, wherein the multimodal data acquisition component includes an LLM-based multimodal data acquisition component; and obtaining the pre-trained target recognition region extraction model.

[0056] Specifically, the pre-built target recognition region extraction model includes not only a region calibration component but also a multimodal data acquisition component. Because the region calibration component continuously optimizes and adjusts the multi-level screenshot initial strategies corresponding to different page address information, the resulting screenshots from the optimized multi-level screenshot optimization strategies for different page address information meet the requirements of both text and image data calibration, avoiding inappropriate cutting errors. Furthermore, since the multi-level screenshot optimization strategies record the corresponding screenshot window position and size information, this information is fed back to the multimodal data acquisition component as knowledge of the information acquisition region. This ensures that the multimodal data acquisition component can efficiently acquire target data information subsequently, improving data acquisition efficiency. The LLM-based multimodal data acquisition component can be understood as a multimodal data acquisition component built around LLM (Large Language Model) with a visual encoder added to the input and output processing, rather than a data acquisition component specifically for pure text data processing.

[0057] In this embodiment, the region calibration component in the pre-built target recognition region extraction model is first used to optimize and adjust the multi-level screenshot initial strategy corresponding to different page address information. This allows the corresponding multi-level screenshot optimization strategy to be directly obtained from the read-only screenshot knowledge learned by the preset read-only screenshot processing terminal when the page address information is actually identified, thus avoiding segmentation defects in the screenshot image. Furthermore, the region calibration component in the pre-built target recognition region extraction model is first used to train the multi-modal data acquisition component in the pre-built target recognition region extraction model to learn information acquisition region knowledge based on the multi-level screenshot optimization strategy corresponding to different page address information. This enables the subsequent rapid and efficient data acquisition using the multi-modal data acquisition component combined with information acquisition region knowledge.

[0058] Continue to refer to Figure 7 , Figure 7 yes Figure 2 A flowchart of a specific embodiment of step 202 shown includes: Step 701: Parse the target read-only screenshot command to obtain the page address information and page rendering status information of the page to be screenshotted; Step 702: Redirect to the page containing the information to be screenshotted based on the page address information; Step 703: Determine whether the page to be screenshotted is in a screenshot-capturing state based on the page rendering state information, wherein the page rendering state information includes whether the page is in a black screen or blank rendering state. Specifically, if the page is in a black screen or blank rendering state, the captured image will have no meaning or value, and the page to be captured will be set to an uncaptureable state. Conversely, if the page rendering state is a normal page display state, the page to be captured will be set to a captureable state.

[0059] Step 704: If the page to be screenshotted is in an uncaptureable state, send an instruction to update and load the page content to the page to be screenshotted. Step 705: Until the page to be screenshotted is in a screenshot-capturing state, obtain the multi-level screenshot optimization strategy corresponding to the page address information from the preset read-only screenshot processing terminal, and perform screenshot processing on the page to be screenshotted in combination with the multi-level screenshot optimization strategy to obtain a read-only image after screenshotting.

[0060] In this embodiment, the multi-level screenshot optimization strategy includes a full-page screenshot strategy, a multi-window non-nested screenshot strategy, and a multi-level nested window screenshot strategy. Specifically, the full-page screenshot strategy refers to performing a read-only screenshot of the entire page, the multi-window non-nested screenshot strategy refers to performing read-only screenshots of the entire page using multiple windows that are not nested, and the multi-level nested window screenshot strategy refers to performing multi-window nested screenshot processing on the entire page. It can be understood that the initial multi-level screenshot strategy also correspondingly includes the full-page screenshot strategy, the multi-window non-nested screenshot strategy, and the multi-level nested window screenshot strategy.

[0061] Furthermore, the full-page screenshot strategy, multi-window non-nested screenshot strategy, and multi-layered window nested screenshot strategy in the multi-level screenshot optimization strategy have different invocation priorities based on the complexity of the page layout corresponding to the page address information. For example, when the page layout corresponding to the current page address information is relatively simple, the invocation priority of the multi-level screenshot optimization strategy is that the full-page screenshot strategy takes precedence over the multi-window non-nested screenshot strategy, which in turn takes precedence over the multi-layered window nested screenshot strategy. Conversely, when the page layout corresponding to the current page address information is relatively complex, the invocation priority of the multi-level screenshot optimization strategy is that the multi-layered window nested screenshot strategy takes precedence over the multi-window non-nested screenshot strategy, which in turn takes precedence over the full-page screenshot strategy. In another embodiment of this application, the invocation priority of the multi-level screenshot optimization strategy corresponding to different page address information can also be the target knowledge to be learned during training. By learning the call priority levels of multi-level screenshot optimization strategies corresponding to different page address information, when acquiring the page address information corresponding to a read-only image during actual data collection, a more suitable screenshot strategy can be quickly and efficiently selected from the corresponding multi-level screenshot optimization strategies according to the call priority levels.

[0062] Continue to refer to Figure 8 , Figure 8 yes Figure 7 A flowchart of a specific embodiment of step 705 shown includes: Step 801: The full-page screenshot strategy, the multi-window non-nested screenshot strategy, and the multi-window nested screenshot strategy are used sequentially to perform screenshot processing on the information page to be screenshotted; Step 802: Obtain the full-page screenshot image captured under the full-page screenshot strategy; Step 803: Obtain multiple combined and stitched page screenshots taken under the multi-window non-nested screenshot strategy; Step 804: Obtain the screenshot image of the multi-layered nested page captured under the multi-layered window nesting screenshot strategy; Step 805: The entire page screenshot, the multiple combined and stitched page screenshots, and the multi-layered nested page screenshots are collectively used as the read-only image after the screenshot is taken.

[0063] Specifically, by learning and optimizing each page to be screenshotted, a corresponding multi-level screenshot optimization strategy is developed, which avoids the problem of too few read-only screenshot images corresponding to a single page to be screenshotted, thus ensuring the high accuracy of subsequent data collection.

[0064] In this embodiment, the step of inputting all read-only images into a pre-trained target recognition region extraction model and labeling all target recognition regions contained in all read-only images, i.e., step 203, includes: using the region labeling component in the pre-trained target recognition region extraction model to perform region bounding labeling processing on all target recognition regions contained in all read-only images, wherein the region labeling component includes a region labeling component based on OCR (Optical Character Recognition) optical recognition technology or a region labeling component based on CNN (Convolutional Neural Network) convolutional neural network; obtaining all target recognition regions contained in each of the read-only images; In this embodiment, the step of extracting data information from all calibrated target recognition regions using the multimodal data acquisition component in the target recognition region calibration model, i.e., step 204, includes: extracting data information from all calibrated target recognition regions using the multimodal data acquisition component in the target recognition region calibration model, wherein the multimodal data acquisition component includes an LLM-based multimodal data acquisition component; and categorizing and organizing the extracted data information according to its type to complete the data information acquisition. Specifically, the extracted text data and image data are categorized and organized.

[0065] In this embodiment, before executing the step of using the multimodal data acquisition component in the target recognition region calibration model to extract data information from all calibrated target recognition regions and complete data information acquisition, the method further includes: determining whether the size of the current target recognition region conforms to a preset recognition region size range; if the size of the current target recognition region does not conform to the preset recognition region size range, then using a proportional scaling method to process the current target recognition region to meet the preset recognition region size range. Specifically, when the target recognition region calibrated by the region calibration component is too large or too small, which is not conducive to the acquisition by the multimodal data acquisition component, a scaling process can be used to process the current target recognition region to meet the preset recognition region size range, facilitating the acquisition by the multimodal data acquisition component and improving data information acquisition efficiency.

[0066] The visual recognition-based data acquisition method provided in this embodiment receives a target read-only screenshot command; responds to the target read-only screenshot command, performs read-only screenshot processing on the page to be screenshotted, and obtains a read-only image after screenshotting; inputs all read-only images into a pre-trained target recognition region extraction model to label all target recognition regions contained in all read-only images; and uses the multimodal data acquisition component in the target recognition region labeling model to extract data information from all labeled target recognition regions, thus completing the data acquisition. This method also provides a method to first utilize the region calibration component in the pre-built target recognition region extraction model to optimize and adjust the multi-level screenshot initial strategies corresponding to different page address information. This allows the corresponding multi-level screenshot optimization strategies to be directly obtained from the read-only screenshot knowledge learned by the preset read-only screenshot processing terminal when the page address information is actually identified, thus avoiding segmentation defects in the screenshot image. Furthermore, it enables the use of the region calibration component in the pre-built target recognition region extraction model to train the multi-modal data acquisition component in the pre-built target recognition region extraction model to learn information acquisition region knowledge based on the multi-level screenshot optimization strategies corresponding to different page address information. This allows for rapid and efficient data acquisition using the multi-modal data acquisition component in conjunction with information acquisition region knowledge. Applying this method to the digital service processes of medical institutions or hospitals, such as in the intelligent digital medical record generation scenario, can replace manual, automated, and flawless processing of read-only screenshots of test reports quickly and efficiently. For the text and image data in the obtained read-only screenshot test report images, an automated artificial intelligence model is used to collect data information, which is then automatically filled into the medical record generation template to generate the final medical record sent to the doctor for review. After the doctor's review, the record is intelligently forwarded to the patient's mobile phone. This effectively avoids typos that occur when manually generating medical records and improves the efficiency of sending medical records to patients in large-scale scenarios with many patients and a large volume of medical records.

[0067] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) refers to the theories, methods, technologies, and application systems that use digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0068] The fundamental technologies of artificial intelligence generally include sensors, dedicated image processing chips, cloud computing, distributed storage, big data processing technology, operating / interactive systems, and mechatronics. Artificial intelligence software technologies mainly include computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0069] The visual recognition-based data acquisition method provided in this embodiment receives a target read-only screenshot command; responds to the target read-only screenshot command, performs read-only screenshot processing on the page to be screenshotted, and obtains a read-only image after screenshotting; inputs all read-only images into a pre-trained target recognition region extraction model to label all target recognition regions contained in all read-only images; and uses the multimodal data acquisition component in the target recognition region labeling model to extract data information from all labeled target recognition regions, thus completing the data acquisition. Applying this method to the digital service processes of medical institutions or hospitals, such as in the intelligent digital medical record generation scenario, can replace manual automated processing of read-only screenshots of test reports. For the text and image data in the obtained read-only screenshot test report images, an artificial intelligence model is automatically used to collect data information, which is then automatically filled into the medical record generation template to generate the final medical record sent to the doctor for review. After the doctor's review, the record is intelligently forwarded to the patient's mobile phone. This effectively avoids typos that occur when manually generating medical records and improves the efficiency of sending medical records to patients in large-scale scenarios with many patients and a large volume of medical records.

[0070] Further reference Figure 9 As a response to the above Figure 2 To implement the method shown, this application provides an embodiment of a data acquisition device based on visual recognition, which is similar to... Figure 2 Corresponding to the method embodiments shown, this device can be specifically applied to various electronic devices.

[0071] like Figure 9 As shown, the visual recognition-based data acquisition device 900 described in this embodiment includes: a screenshot instruction receiving module 901, a read-only screenshot processing module 902, a recognition area calibration module 903, and a multimodal data acquisition module 904. Wherein: The screenshot instruction receiving module 901 is used to receive a target read-only screenshot instruction, wherein the target read-only screenshot instruction provides the page address information and page rendering status information of the page to be screenshotted; The read-only screenshot processing module 902 is used to respond to the target read-only screenshot instruction, perform read-only screenshot processing on the information page to be screenshotted, and obtain a read-only image after the screenshot is taken. The recognition region labeling module 903 is used to input all read-only images into the pre-trained target recognition region extraction model and label all target recognition regions contained in all read-only images; The multimodal data acquisition module 904 is used to extract data information from all calibrated target recognition regions using the multimodal data acquisition component in the target recognition region calibration model, thereby completing the data information acquisition.

[0072] This application involves receiving a target read-only screenshot instruction; responding to the target read-only screenshot instruction, performing read-only screenshot processing on the page to be screenshotted to obtain a read-only image; inputting all read-only images into a pre-trained target recognition region extraction model to calibrate all target recognition regions contained in all read-only images; and using the multimodal data acquisition component in the target recognition region calibration model to extract data information from all calibrated target recognition regions, thus completing data information acquisition. Applying this method to the digital service process of medical institutions or hospitals, such as in intelligent digital medical record generation scenarios, it can replace manual automated processing of read-only screenshots of test reports. For the text and image data in the obtained read-only screenshot test report images, it automatically uses an artificial intelligence model to collect data information, and then automatically fills it into the medical record generation template to generate the final medical record sent to the doctor for review. After the doctor's review, it is intelligently forwarded to the patient's mobile phone. This effectively avoids typos that occur when manually generating medical records and improves the efficiency of sending medical records to patients in large-scale scenarios with many patients and a large volume of medical records.

[0073] In this embodiment, the data information acquisition device 900 based on visual recognition further includes a read-only screenshot image caching module. The read-only screenshot image caching module is used to cache the read-only image in a preset queryable image library. The preset queryable image library is connected to the target image query acquisition end in an end-to-end bridging mode.

[0074] In this embodiment, the visual recognition-based data acquisition device 900 further includes a multi-level screenshot initial strategy acquisition module, a read-only screenshot batch processing module, a read-only image classification and organization module, and a read-only screenshot knowledge optimization module. Wherein: The multi-level screenshot initial strategy acquisition module is used to pre-acquire the multi-level screenshot initial strategies set for different pages of information to be screenshotted; The read-only screenshot batch processing module is used to perform read-only screenshot processing according to the multi-level screenshot initial strategy corresponding to different pages of information to be screenshotted, and obtain all read-only images of all screenshots corresponding to all pages of information to be screenshotted. The read-only image classification and organization module is used to classify and organize all the read-only images after screenshotting each page according to the page address information corresponding to each page to be screenshotted, and obtain a read-only image set with the page address information as the classification identifier and the read-only images after screenshotting as the classification elements. The read-only screenshot knowledge optimization module is used to reverse optimize the multi-level initial screenshot strategy corresponding to different pages of information to be screenshotted through the read-only image set, and obtain the multi-level screenshot optimization strategy corresponding to different pages of information to be screenshotted as read-only screenshot knowledge feedback to the preset read-only screenshot processing terminal.

[0075] In this embodiment, the read-only screenshot knowledge optimization module includes an input target recognition region extraction model unit, a region calibration processing unit, a screenshot content missing recognition unit, a backtracking and optimization adjustment unit, and a read-only screenshot knowledge feedback unit. Wherein: The input target recognition region extraction model unit is used to input the read-only image set corresponding to all page address information into the pre-built target recognition region extraction model, wherein the pre-built target recognition region calibration model includes a region calibration component; The region labeling processing unit is used to label the text data region and image data region in each read-only image by using the region labeling component in the target recognition region extraction model. The screenshot content missing recognition unit is used to identify whether there is missing screenshot content in the text data area and image data area of ​​each read-only image based on the area calibration result of the area calibration component; The backtracking and optimization adjustment unit is used to mark the current read-only image if there is missing screenshot content in the current read-only image, and backtrack the corresponding page address information and the multi-level screenshot initial strategy adopted based on the current read-only image, and optimize and adjust the multi-level screenshot initial strategy adopted. The read-only screenshot knowledge feedback unit is used to feed back the multi-level screenshot optimization strategy corresponding to different page address information as read-only screenshot knowledge to the preset read-only screenshot processing terminal until there is no missing text data and image data in each read-only image. The multi-level screenshot initial strategy and the multi-level screenshot optimization strategy respectively specify the corresponding screenshot window position and size specifications.

[0076] In this embodiment, the visual recognition-based data acquisition device 900 further includes a non-target data blanking processing module. The non-target data blanking processing module is used to blank out the non-textual data and non-image data in each read-only image, retaining only the text data and image data in each read-only image.

[0077] In this embodiment, the region calibration processing unit includes a boundary information identification subunit and a region calibration processing subunit. Wherein: The boundary information recognition subunit is used to identify the boundary information corresponding to all text data and image data in each read-only image through the region calibration component. The boundary information includes vertex coordinate information, center point coordinate information, and coordinate information of all points constituting the boundary region box. The region labeling processing subunit is used to label the text data region and image data region in each read-only image based on the boundary information corresponding to all text data and image data respectively.

[0078] In this embodiment, the screenshot content missing identification unit includes a calibration region box integrity judgment subunit, a screenshot content non-missing determination subunit, and a screenshot content missing determination subunit. Wherein: The calibration region bounding box integrity judgment subunit is used to determine, based on the region calibration results, whether both the text data region and the image data region in the current read-only image have complete calibration region bounding boxes. The screenshot content non-missing determination subunit is used to determine that if both the text data area and the image data area in the current read-only image have complete bounding boxes, then there will be no missing screenshot content. The screenshot content missing determination subunit is used to determine if any text data area or image data area in the current read-only image does not have a complete bounding box, in which case there will be missing screenshot content.

[0079] In this embodiment, the visual recognition-based data acquisition device 900 further includes an information acquisition area knowledge feedback module and a model learning and training completion module. Wherein: The information collection area knowledge feedback module is used to feed back the screenshot window position and size information corresponding to the multi-level screenshot optimization strategy as information collection area knowledge to the multimodal data collection component. The model learning and training completion module is used to obtain a pre-trained target recognition region extraction model.

[0080] In this embodiment, the read-only screenshot processing module 902 includes a read-only screenshot instruction parsing unit, a page orientation unit for the information to be screenshotted, a screenshot-capturing status judgment unit, a page update request unit, and a page screenshot processing unit. Wherein: A read-only screenshot instruction parsing unit is used to parse the target read-only screenshot instruction and obtain the page address information and page rendering status information of the page to be screenshotted. A page redirection unit for screenshot information is used to redirect to the page to be screenshotted based on the page address information; A captureable state determination unit is used to determine whether the page to be captured is in a captureable state based on the page rendering state information, wherein the page rendering state information includes whether the page is in a black screen or blank rendering state. A page update request unit is used to send an instruction to update and load page content to the page to be captured if the page to be captured is in an uncaptureable state. The page screenshot processing unit is used to obtain a multi-level screenshot optimization strategy corresponding to the page address information from a preset read-only screenshot processing terminal until the page to be screenshotted is in a screenshotable state, and then perform screenshot processing on the page to be screenshotted in combination with the multi-level screenshot optimization strategy to obtain a read-only image after the screenshot.

[0081] In this embodiment, the page screenshot processing unit includes a screenshot processing subunit, a first page screenshot image acquisition subunit, a second page screenshot image acquisition subunit, a third page screenshot image acquisition subunit, and a read-only image processing subunit. Wherein: The sequential screenshot processing subunit is used to sequentially perform screenshot processing on the page to be screenshotted using the full-page screenshot strategy, the multi-window non-nested screenshot strategy, and the multi-window nested screenshot strategy, respectively. The first sub-unit for obtaining page screenshot images is used to obtain the entire page screenshot image captured under the full-page screenshot strategy; The second sub-unit for obtaining page screenshot images is used to obtain multiple combined and stitched page screenshot images captured under the multi-window non-nested screenshot strategy; The third sub-unit for obtaining page screenshot images is used to obtain multi-layered nested page screenshot images captured under the multi-layered window nesting screenshot strategy; The read-only image processing subunit is used to combine the entire page screenshot, the multiple combined and stitched page screenshots, and the multi-layered nested page screenshots as the read-only image after the screenshot is taken.

[0082] In this embodiment, the identification region calibration module 903 includes a region bounding calibration processing unit and a target identification region acquisition unit. Wherein: The region bounding and labeling processing unit is used to perform region bounding and labeling processing on all target recognition regions contained in all read-only images using the region labeling component in the pre-trained target recognition region extraction model. The region labeling component includes a region labeling component based on OCR optical recognition technology or a region labeling component based on CNN convolutional neural network. The target recognition region acquisition unit is used to obtain all target recognition regions contained in each of the read-only images.

[0083] In this embodiment, the multimodal data acquisition module 904 includes a data information extraction unit and a categorization and processing unit. Wherein: The data information extraction unit is used to extract data information from all calibrated target recognition regions using the multimodal data acquisition component in the target recognition region calibration model, wherein the multimodal data acquisition component includes an LLM-based multimodal data acquisition component. The categorization and sorting unit is used to categorize and sort the extracted data information according to the different types of data information, thereby completing the data information collection.

[0084] In this embodiment, the visual recognition-based data acquisition device 900 further includes a recognition area size range conformity judgment module and a proportional scaling adjustment module. Wherein: The recognition region size range judgment module is used to determine whether the size of the current target recognition region conforms to the preset recognition region size range; The proportional scaling adjustment module is used to process the current target recognition area to meet the preset recognition area size range if the current target recognition area size does not conform to the preset recognition area size range.

[0085] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by instructing related hardware through computer-readable instructions. These computer-readable instructions can be stored in a computer-readable storage medium. When the program is executed, it can include the processes of the embodiments of the methods described above. The aforementioned storage medium can be a non-volatile storage medium such as a magnetic disk, optical disk, or read-only memory (ROM), or random access memory (RAM).

[0086] It should be understood that although the steps in the flowcharts of the accompanying figures are shown sequentially as indicated by the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowcharts of the accompanying figures may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times, and their execution order is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0087] To address the aforementioned technical problems, embodiments of this application also provide a computer device. Please refer to [link / reference needed]. Figure 10 , Figure 10 This is a basic structural block diagram of the computer device in this embodiment.

[0088] The computer device 10 includes a memory 10a, a processor 10b, and a network interface 10c, which are interconnected via a system bus. It should be noted that... Figure 10 Only a computer device 10 with component memory 10a, processor 10b, and network interface 10c is shown. However, it should be understood that it is not required to implement all the components shown, and more or fewer components may be implemented instead. Those skilled in the art will understand that the computer device described herein is a device capable of automatically performing numerical calculations and / or information processing according to pre-set or stored instructions. Its hardware includes, but is not limited to, microprocessors, application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), digital signal processors (DSPs), embedded devices, etc.

[0089] The computer device can be a desktop computer, laptop, handheld computer, or cloud server, etc. The computer device can interact with the user via a keyboard, mouse, remote control, touchpad, or voice control.

[0090] The memory 10a includes at least one type of readable storage medium, including flash memory, hard disk, multimedia card, card-type memory (e.g., SD or DX memory), random access memory (RAM), static random access memory (SRAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), programmable read-only memory (PROM), magnetic memory, magnetic disk, optical disk, etc. In some embodiments, the memory 10a may be an internal storage unit of the computer device 10, such as the hard disk or memory of the computer device 10. In other embodiments, the memory 10a may also be an external storage device of the computer device 10, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc. Of course, the memory 10a may include both internal storage units and external storage devices of the computer device 10. In this embodiment, the memory 10a is typically used to store the operating system and various application software installed on the computer device 10, such as computer-readable instructions for data acquisition methods based on visual recognition. In addition, the memory 10a can also be used to temporarily store various types of data that have been output or will be output.

[0091] In some embodiments, the processor 10b may be a central processing unit (CPU), controller, microcontroller, microprocessor, or other data processing chip. The processor 10b is typically used to control the overall operation of the computer device 10. In this embodiment, the processor 10b is used to execute computer-readable instructions stored in the memory 10a or to process data, for example, to execute computer-readable instructions of the vision-based data acquisition method.

[0092] The network interface 10c may include a wireless network interface or a wired network interface, which is typically used to establish communication connections between the computer device 10 and other electronic devices.

[0093] The computer device proposed in this embodiment belongs to the field of big data technology and is applied in scenarios involving the collection of page display content. This application receives a target read-only screenshot instruction; in response to the target read-only screenshot instruction, performs read-only screenshot processing on the page to be screenshotted to obtain a read-only image; inputs all read-only images into a pre-trained target recognition region extraction model to label all target recognition regions contained in all read-only images; and uses the multimodal data acquisition component in the target recognition region labeling model to extract data information from all labeled target recognition regions, thus completing data information collection. Applying this method to the digital service processes of medical institutions or hospitals, such as in the intelligent digital medical record generation scenario, can replace manual automated processing of read-only screenshots of test reports. For the text and image data in the obtained read-only screenshot test report images, an artificial intelligence model is automatically used to collect data information, which is then automatically filled into the medical record generation template to generate the final medical record sent to the doctor for review. After the doctor's review, the record is intelligently forwarded to the patient's mobile phone. This effectively avoids typos that occur when manually generating medical records and improves the efficiency of sending medical records to patients in large-scale scenarios with many patients and a large volume of medical records.

[0094] This application also provides another embodiment, namely, providing a computer-readable storage medium storing computer-readable instructions that can be executed by a processor to cause the processor to perform the steps of the visual recognition-based data acquisition method described above.

[0095] The computer-readable storage medium proposed in this embodiment belongs to the field of big data technology and is applied in scenarios involving the collection of page display content. This application involves receiving a target read-only screenshot instruction; responding to the target read-only screenshot instruction, performing read-only screenshot processing on the page to be screenshotted to obtain a read-only image; inputting all read-only images into a pre-trained target recognition region extraction model to calibrate all target recognition regions contained in all read-only images; and using the multimodal data acquisition component in the target recognition region calibration model to extract data information from all calibrated target recognition regions, thus completing data information collection. Applying this method to the digital service processes of medical institutions or hospitals, such as in the intelligent digital medical record generation scenario, can replace manual automated processing of read-only screenshots of test reports. For the text and image data in the obtained read-only screenshot test report images, an artificial intelligence model is automatically used to collect data information, which is then automatically filled into the medical record generation template to generate the final medical record sent to the doctor for review. After the doctor's review, the record is intelligently forwarded to the patient's mobile phone. This effectively avoids typos that occur when manually generating medical records and improves the efficiency of sending medical records to patients in large-scale scenarios with many patients and a large volume of medical records.

[0096] Through the above description of the embodiments, those skilled in the art can clearly understand that the methods of the above embodiments can be implemented by means of software plus necessary general-purpose hardware platforms. Of course, they can also be implemented by hardware, but in many cases the former is a better implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a storage medium (such as ROM / RAM, magnetic disk, optical disk) and includes several instructions to cause a terminal device (which may be a mobile phone, computer, server, air conditioner, or network device, etc.) to execute the methods described in the various embodiments of this application.

[0097] Obviously, the embodiments described above are only some embodiments of this application, not all embodiments. The accompanying drawings show preferred embodiments of this application, but do not limit the patent scope of this application. This application can be implemented in many different forms; rather, the purpose of providing these embodiments is to make the disclosure of this application more thorough and comprehensive. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art can still modify the technical solutions described in the foregoing specific embodiments, or make equivalent substitutions for some of the technical features. Any equivalent structures made using the content of this application's specification and drawings, directly or indirectly applied to other related technical fields, are similarly within the scope of patent protection of this application. Software tools or components not belonging to this company that appear in the embodiments of this application are merely illustrative examples and do not represent actual use.

[0098] It should be noted that if any AI model software tools or components not belonging to this company appear in the embodiments of this application, they are merely illustrative examples and do not represent actual use. All user personal information involved in the embodiments of this application is authorized (with the knowledge and consent) by the relevant parties or fully authorized by all parties, and the executing entity may obtain it through various legal and compliant means. The collection, storage, use, transmission, provision, and disclosure of the information, data, and signals involved all comply with relevant laws and regulations and do not violate public order and good morals.

Claims

1. A data information acquisition method based on visual recognition, characterized in that, Includes the following steps: Receive a target read-only screenshot instruction, wherein the target read-only screenshot instruction provides the page address information and page rendering status information of the page to be screenshotted; In response to the target read-only screenshot instruction, the page to be screenshotted is processed for read-only screenshotting to obtain a read-only image after screenshotting; All read-only images are input into a pre-trained target recognition region extraction model to identify all target recognition regions contained in all read-only images; The multimodal data acquisition component in the target recognition region calibration model is used to extract data information from all calibrated target recognition regions, thus completing the data information acquisition.

2. The data information acquisition method based on visual recognition according to claim 1, characterized in that, Before executing the step of performing a read-only screenshot on the page to be screenshotted in response to the target read-only screenshot instruction to obtain a read-only image after screenshotting, the method further includes: Pre-obtain the multi-level initial screenshot strategies set for different pages of information to be screenshotted; Based on the multi-level initial screenshot strategy corresponding to different pages of information to be screenshotted, read-only screenshot processing is performed separately to obtain all read-only images of all pages of information to be screenshotted. Based on the page address information corresponding to each page to be screenshotted, all the read-only images after being screenshotted are classified and organized to obtain a read-only image set with the page address information as the classification identifier and the read-only images after being screenshotted as the classification elements. By reverse-optimizing the initial multi-level screenshot strategy corresponding to different pages of information to be screenshotted using the read-only image set, the final multi-level screenshot optimization strategy corresponding to different pages of information to be screenshotted is obtained and fed back as read-only screenshot knowledge to the preset read-only screenshot processing terminal.

3. The data information acquisition method based on visual recognition according to claim 2, characterized in that, The step of reversely optimizing the initial multi-level screenshot strategy corresponding to different pages of information to be screenshotted using the read-only image set, and obtaining the final multi-level screenshot optimization strategy corresponding to different pages of information to be screenshotted as read-only screenshot knowledge feedback to the target read-only screenshot processing end, includes: The read-only image set corresponding to all page address information is input into the pre-built target recognition region extraction model, wherein the pre-built target recognition region calibration model includes a region calibration component; The region labeling component in the target recognition region extraction model is used to label the text data region and image data region in each read-only image; Based on the region calibration results of the region calibration component, identify whether there are missing screenshot content in the text data region and image data region of each read-only image; If the current read-only image contains missing screenshot content, then mark the current read-only image, and backtrack the corresponding page address information and the multi-level screenshot initial strategy adopted based on the current read-only image, and optimize and adjust the multi-level screenshot initial strategy adopted. Until there is no missing text or image data in each read-only image, the multi-level screenshot optimization strategy corresponding to different page address information is fed back as read-only screenshot knowledge to the preset read-only screenshot processing terminal. The multi-level screenshot initial strategy and the multi-level screenshot optimization strategy respectively specify the corresponding screenshot window position and size specifications.

4. The data information acquisition method based on visual recognition according to claim 3, characterized in that, Before executing the step of calibrating the text data region and image data region in each read-only image using the region calibration component in the target recognition region extraction model, the method further includes: The non-textual and non-image data in each read-only image are blanked out, and only the text and image data in each read-only image are retained; The step of calibrating the text data region and image data region in each read-only image using the region calibration component in the target recognition region extraction model includes: The region calibration component identifies the boundary information corresponding to all text data and image data in each read-only image. The boundary information includes vertex coordinate information, center point coordinate information, and coordinate information of all points that constitute the boundary region box. Based on the boundary information corresponding to all text data and image data, the text data region and image data region in each read-only image are marked.

5. The data information acquisition method based on visual recognition according to claim 3, characterized in that, The pre-built target recognition region extraction model also includes a multimodal data acquisition component. After executing the step of feeding back the multi-level screenshot optimization strategy corresponding to different page address information as read-only screenshot knowledge to the preset read-only screenshot processing terminal, the method further includes: The screenshot window position and size information corresponding to each of the multi-level screenshot optimization strategies are fed back to the multimodal data acquisition component as information collection area knowledge. Obtain a pre-trained target recognition region extraction model.

6. The data information acquisition method based on visual recognition according to claim 1, characterized in that, The step of performing read-only screenshot processing on the page to be screenshotted in response to the target read-only screenshot command, and obtaining a read-only image after screenshotting, includes: Parse the target read-only screenshot command to obtain the page address information and page rendering status information of the page to be screenshotted; Based on the page address information, the user is redirected to the page containing the screenshot information. The page rendering status information is used to determine whether the page to be screenshotted is in a screenshot-capturing state. The page rendering status information includes whether the page is in a black screen or blank rendering state. If the page to be screenshotted is in an uncaptureable state, then send a command to the page to be screenshotted to update and load the page content; Until the page to be screenshotted is in a screenshot-capturing state, the multi-level screenshot optimization strategy corresponding to the page address information is obtained from the preset read-only screenshot processing terminal, and the page to be screenshotted is processed in combination with the multi-level screenshot optimization strategy to obtain a read-only image after screenshotting.

7. The data information acquisition method based on visual recognition according to claim 6, characterized in that, The multi-level screenshot optimization strategy includes a full-page screenshot strategy, a multi-window non-nested screenshot strategy, and a multi-layered window nested screenshot strategy. The step of combining the multi-level screenshot optimization strategy to perform screenshot processing on the page to be screenshotted and obtaining a read-only image after screenshotting includes: The full-page screenshot strategy, the multi-window non-nested screenshot strategy, and the multi-window nested screenshot strategy are used sequentially to process the screenshot of the information page to be captured; Retrieve a full-page screenshot image taken under the full-page screenshot strategy; Retrieves multiple combined and stitched screenshots of the page taken under a multi-window non-nested screenshot strategy; Retrieve screenshots of multi-layered nested pages captured under a multi-layered window nesting screenshot strategy; The entire page screenshot, the multiple combined and stitched page screenshots, and the multi-layered nested page screenshots are collectively used as the read-only image after the screenshot is taken.

8. A data acquisition device based on visual recognition, characterized in that, include: A screenshot command receiving module is used to receive a target read-only screenshot command, wherein the target read-only screenshot command provides the page address information and page rendering status information of the page to be screenshotted; The read-only screenshot processing module is used to respond to the target read-only screenshot instruction, perform read-only screenshot processing on the information page to be screenshotted, and obtain a read-only image after the screenshot is taken. The recognition region labeling module is used to input all read-only images into the pre-trained target recognition region extraction model and label all target recognition regions contained in all read-only images; The multimodal data acquisition module is used to extract data information from all calibrated target recognition regions using the multimodal data acquisition component in the target recognition region calibration model, thereby completing the data information acquisition.

9. A computer device, characterized in that, The method includes a memory and a processor, wherein the memory stores computer-readable instructions, and the processor executes the computer-readable instructions to implement the steps of the data acquisition method based on visual recognition as described in any one of claims 1 to 7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-readable instructions, which, when executed by a processor, implement the steps of the data acquisition method based on visual recognition as described in any one of claims 1 to 7.

Images