Visual language model interaction method and device based on screenshot and medium

Abstract

The invention discloses a visual language model interaction method and device based on screenshot and a medium, and relates to the technical field of multi-mode large models.The method comprises the steps that on the basis of a preset screenshot directory, timestamp traversal analysis is conducted on directory timestamps of the screenshot directory, and current screenshot data of a user is determined; performing analysis model recommendation evaluation on the current screenshot data of the user to obtain a preferred analysis model; according to the optimization analysis model, determining a calling function of the optimization analysis model through calling function configuration; based on the calling function, inputting the current screenshot of the user into the preferable analysis model to obtain text analysis data; and obtaining user instruction data, and performing comprehensive semantic analysis on the user instruction data and the text analysis data to obtain a screenshot analysis result. By means of the method, the technical problem that the user screenshot image information search operation process is tedious is solved.

Description

Technical Field

[0001] The present application relates to the technical field of multimodal large models, and in particular to a screenshot-based visual language model interaction method, device and medium. Background Art

[0002] In recent years, with the rapid development of deep learning and artificial intelligence technology, vision-language models have become one of the important tools for multimodal information processing. In the subdivided application scenarios of vision-language models, the demand for automatic analysis based on screenshots has gradually increased. Through the rapid understanding and information extraction of screenshot content, individual users who frequently use computers for study and work can realize convenient and efficient functions such as screenshot content acquisition, questioning and analysis, thereby improving daily use experience and information retrieval efficiency.

[0003] Many computer users often need to search for content on the screen, including text and images, in daily operations. However, during the text search process, users usually need to manually copy the text first, then open the browser to perform the search step by step. If image content is involved, especially if you want to obtain further explanation of certain details in the pattern, even with the help of existing multimodal large model tools, users still need to copy the image and open a specific website. The operation process is cumbersome, the user experience is poor, and the efficiency of the workflow is reduced. Summary of the invention

[0004] The embodiments of the present application provide a screenshot-based visual language model interaction method, device and medium, which solve the technical problem of complicated operation procedures for users to search for screenshot image information.

[0005] In a first aspect, an embodiment of the present application provides a screenshot-based visual language model interaction method, characterized in that the method includes: based on a preset screenshot directory, performing a timestamp traversal analysis on the directory timestamps of the screenshot directory to determine the user's current screenshot data; performing an analysis model recommendation evaluation on the user's current screenshot data to obtain a preferred analysis model; according to the preferred analysis model, determining a calling function of the preferred analysis model by calling a function configuration; based on the calling function, inputting the user's current screenshot into the preferred analysis model to obtain text analysis data; obtaining user instruction data, and performing a comprehensive semantic analysis on the user instruction data and the text analysis data to obtain a screenshot analysis result.

[0006] In one implementation of the present application, based on a preset screenshot directory, a timestamp traversal analysis is performed on the directory timestamp of the screenshot directory to determine the user's current screenshot data, specifically including: obtaining the current timestamp, and based on the current timestamp, periodically traversing the screenshot directory to obtain the last modification time of the screenshot file; determining the screenshot update data based on the last modification time of the screenshot file by identifying the last check time; updating the current timestamp to determine the detection timestamp after the current timestamp, and performing a storage path query on the screenshot update data to obtain the user's current screenshot data; wherein the user's current screenshot data includes: the user's current screenshot, and the user's current screenshot storage path.

[0007] In one implementation of the present application, an analysis model recommendation evaluation is performed on the user's current screenshot data to obtain a preferred analysis model, specifically including: performing image type feature analysis on the current screenshot data to obtain a first recommended evaluation parameter; performing image information volume feature analysis on the current screenshot data to obtain a second recommended evaluation parameter; based on the first recommended evaluation parameter and the second recommended evaluation parameter, determining the preferred analysis model through screenshot data processing performance analysis.

[0008] In one implementation of the present application, according to the preferred analysis model, the calling function of the preferred analysis model is determined by calling the function configuration, specifically including: when the preferred analysis model is a cloud model, calling the completion function in the preset LiteLLM library to determine the first calling function; when the preferred analysis model is a local model, calling the chat function in the preset Ollama library to determine the second calling function; based on the first calling function or the second calling function, determining the calling function of the preferred analysis model.

[0009] In one implementation of the present application, based on a calling function, the user's current screenshot is input into a preferred analysis model to obtain text analysis data, specifically including: when the calling function is a completion function, encapsulating the user's current screenshot data; calling a preset API, and inputting the encapsulated user's current screenshot data into the API to obtain API parsing parameters; based on the API parsing parameters, performing a screenshot content analysis on the user's current screenshot data to obtain first text analysis data; when the calling function is a chat function, loading an analysis model in a preset local environment, and obtaining second text analysis data through screenshot content analysis according to the analysis model; obtaining text analysis data according to the first text analysis data or the second text analysis data.

[0010] In one implementation of the present application, a comprehensive semantic analysis is performed on user command data and text analysis data to obtain a screenshot analysis result, specifically including: performing user intent analysis on the user command data to determine the user command impact parameters; based on the user command impact parameters, performing simulated command analysis on the text analysis data to obtain command analysis data of the text analysis data; and performing command intent verification on the command analysis data to obtain a screenshot analysis result.

[0011] In one implementation of the present application, after performing comprehensive semantic analysis on user command data and text analysis data to obtain screenshot analysis results, the method also includes: based on a preset display interface configuration, typeset the screenshot analysis results to obtain an initial display interface of the screenshot analysis results; according to the initial display interface of the screenshot analysis results, obtain a display interface of the screenshot analysis results through real-time configuration of the display interface.

[0012] In one implementation of the present application, after obtaining the display interface of the screenshot analysis results through real-time configuration of the display interface based on the initial display interface of the screenshot analysis results, the method also includes: storing the screenshot analysis results to a preset cache address, and periodically scanning the cache address for storage space to determine a cache clearing node; when the remaining storage space of the cache address is lower than a preset storage threshold, resetting the cache space of the cache address.

[0013] In a second aspect, an embodiment of the present application also provides a screenshot-based visual language model interaction device, characterized in that the device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can: based on a preset screenshot directory, perform a timestamp traversal analysis on the directory timestamps of the screenshot directory to determine the user's current screenshot data; perform an analysis model recommendation evaluation on the user's current screenshot data to obtain a preferred analysis model; according to the preferred analysis model, determine the calling function of the preferred analysis model by calling the function configuration; based on the calling function, input the user's current screenshot into the preferred analysis model to obtain text analysis data; obtain user command data, and perform a comprehensive semantic analysis on the user command data and the text analysis data to obtain a screenshot analysis result.

[0014] In a third aspect, an embodiment of the present application further provides a non-volatile computer storage medium for a screenshot-based visual language model interaction method, storing computer executable instructions, characterized in that the computer executable instructions are configured to: based on a preset screenshot directory, perform a timestamp traversal analysis on the directory timestamps of the screenshot directory to determine the user's current screenshot data; perform an analysis model recommendation evaluation on the user's current screenshot data to obtain a preferred analysis model; according to the preferred analysis model, determine a calling function of the preferred analysis model by calling a function configuration; based on the calling function, input the user's current screenshot into the preferred analysis model to obtain text analysis data; obtain user command data, and perform a comprehensive semantic analysis on the user command data and the text analysis data to obtain a screenshot analysis result.

[0015] The embodiments of the present application provide a screenshot-based visual language model interaction method, device and medium, which solve the technical problem of cumbersome image information search operation process through real-time updating of screenshot files to be analyzed, dynamic recommendation of analysis algorithms and simultaneous analysis of user instructions, realize the automation of screenshot analysis, improve the accuracy of screenshot analysis results, support Ollama framework to call local reasoning models, reduce cloud dependence and improve the security of user privacy. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] The drawings described herein are used to provide a further understanding of the present application and constitute a part of the present application. The illustrative embodiments of the present application and their descriptions are used to explain the present application and do not constitute an improper limitation on the present application. In the drawings: Figure 1 A flowchart of a screenshot-based visual language model interaction method provided in an embodiment of the present application; Figure 2 A schematic diagram of the internal structure of a screenshot-based visual language model interaction device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0017] In order to make the purpose, technical solution and advantages of the present application clearer, the technical solution of the present application will be clearly and completely described below in combination with the specific embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of the present application.

[0018] The embodiments of the present application provide a screenshot-based visual language model interaction method, device and medium, which solve the technical problem of cumbersome image information search operation process through real-time updating of screenshot files to be analyzed, dynamic recommendation of analysis algorithms and simultaneous analysis of user instructions, realize the automation of screenshot analysis, improve the accuracy of screenshot analysis results, support Ollama framework to call local reasoning models, reduce cloud dependence and improve the security of user privacy.

[0019] The technical solution proposed in the embodiments of the present application is described in detail below with reference to the accompanying drawings.

[0020] Figure 1 A flowchart of a screenshot-based visual language model interaction method provided in an embodiment of the present application. Figure 1 As shown, a screenshot-based visual language model interaction method provided in an embodiment of the present application specifically includes the following steps: Step 101: Based on a preset screenshot directory, perform a timestamp traversal analysis on directory timestamps of the screenshot directory to determine the user's current screenshot data.

[0021] Specifically, it includes: obtaining the current timestamp, and based on the current timestamp, periodically traversing the screenshot directory to obtain the last modification time of the screenshot file; determining the screenshot update data according to the last modification time of the screenshot file through the last inspection time identification; updating the current timestamp to determine the detection timestamp after the current timestamp, and performing a storage path query on the screenshot update data to obtain the user's current screenshot data; wherein, the user's current screenshot data includes: the user's current screenshot and the user's current screenshot storage path.

[0022] Based on a preset screenshot directory, this application performs timestamp traversal analysis on the directory timestamps of the screenshot directory to determine the user's current screenshot data, so that each newly added screenshot data in the screenshot directory can be identified and stored in sequence. Path query improves the recognition accuracy of the newly added screenshot data.

[0023] In the examples of the present application, detailed explanation is given by the following Example 1.

[0024] Example 1: The screenshot detection module continuously monitors the screenshot storage directory specified by the user, promptly detects newly added screenshot files, and passes their paths to the image analysis module and the user interaction interface module.

[0025] First, the module is set to monitor the screenshot storage directory once a second. Each time it is checked, the system will get the current time and save it as the "last checked time".

[0026] Traverse all files in the screenshot storage directory and obtain the last modification time of each file one by one.

[0027] Compare the last modification time of the file with the "last check time". If the file is created or modified after the last check, it is considered a new screenshot file.

[0028] Then, for files that meet the conditions (i.e., new screenshot files), the module sends their file paths to the image analysis module and the user interaction interface module for subsequent processing and display.

[0029] Finally, the system updates the “last check time” to the current time to ensure that the next check can accurately identify the newly added files.

[0030] Step 102: perform analysis model recommendation and evaluation on the user's current screenshot data to obtain a preferred analysis model.

[0031] Specifically, it includes: performing image type feature analysis on the current screenshot data to obtain a first recommended evaluation parameter; performing image information quantity feature analysis on the current screenshot data to obtain a second recommended evaluation parameter; and determining a preferred analysis model through screenshot data processing performance analysis based on the first recommended evaluation parameter and the second recommended evaluation parameter.

[0032] This application recommends and evaluates the analysis model of the user's current screenshot data to obtain the optimal analysis model, thereby realizing the demand assessment of screenshot analysis, improving the efficiency and accuracy of screenshot analysis, and meeting the user's security needs and real-time usage needs.

[0033] In the examples of the present application, detailed explanation is given by the following Example 2.

[0034] Example 2: Image type feature analysis can distinguish image types with different color distributions by calculating the color histogram.

[0035] Using the Sobel edge detection algorithm, the edge information in the image is analyzed. Charts and graphs usually have a high edge density.

[0036] Analyzing the texture features of an image using GLCM (Gray Level Co-occurrence Matrix) helps distinguish natural textures (such as trees and grass) from artificial textures (such as buildings and lines on a graph).

[0037] Image information feature analysis analyzes the entropy of the image and calculates the spatial frequency of the image in the horizontal and vertical directions to reflect the richness of the image details. The color distribution variance of each RGB channel of the user screenshot is analyzed to evaluate the color richness. Finally, Harris corner detection is used to count the number of feature points in the image.

[0038] First, according to the image type, some models that are not effective for specific types of images are selected or excluded, and then the preferred analysis model is determined.

[0039] Step 103: According to the preferred analysis model, by calling function configuration, determine the calling function of the preferred analysis model.

[0040] Specifically, it includes: when the preferred analysis model is a cloud model, calling the completion function in the preset LiteLLM library to determine the first calling function; when the preferred analysis model is a local model, calling the chat function in the preset Ollama library to determine the second calling function; based on the first calling function or the second calling function, determining the calling function of the preferred analysis model.

[0041] This application determines the calling function of the preferred analysis model based on the preferred analysis model by calling function configuration, realizes real-time calling of the screenshot analysis model and local model reasoning, and improves the flexibility and accuracy of screenshot analysis.

[0042] In the examples of the present application, this is explained in detail through the following Example 3.

[0043] Example 3: If the user chooses to use a cloud model (such as OpenAI, Qwen), the module will call the completion function in the LiteLLM library.

[0044] If the user chooses to use a local model (such as the model in the Ollama framework), the module calls the chat function in the Ollama library.

[0045] Step 104: Based on the calling function, the user's current screenshot is input into the optimal analysis model to obtain text analysis data.

[0046] Specifically, it includes: when the calling function is a completion function, encapsulating the user's current screenshot data; calling a preset API, and inputting the encapsulated user's current screenshot data into the API to obtain API parsing parameters; based on the API parsing parameters, performing a screenshot content analysis on the user's current screenshot data to obtain first text analysis data; when the calling function is a chat function, loading an analysis model in a preset local environment, and obtaining second text analysis data through screenshot content analysis based on the analysis model; obtaining text analysis data based on the first text analysis data or the second text analysis data.

[0047] Based on calling functions, this application inputs the user's current screenshot into the preferred analysis model to obtain text analysis data, realizes flexible calling analysis of local large model inference frameworks (such as Ollama) and cloud large model APIs (such as LiteLLM), and improves the efficiency of user screenshot analysis.

[0048] In the examples of the present application, a detailed explanation is given by the following Example 4.

[0049] Example 4: When the calling function is the completion function, the system will pass the image path from the screenshot detection module and the question entered by the user to the API, and the cloud-based large model will analyze and generate text results.

[0050] When the calling function is a chat function, the model is loaded and run in the local environment. The image and text input are sent to the local large model for inference and generate response results.

[0051] Step 105: Obtain user instruction data, and perform comprehensive semantic analysis on the user instruction data and text analysis data to obtain screenshot analysis results.

[0052] Specifically, it includes: performing user intent analysis on user instruction data to determine the user instruction impact parameters; based on the user instruction impact parameters, performing simulated instruction analysis on the text analysis data to obtain instruction analysis data of the text analysis data; and performing instruction intent verification on the instruction analysis data to obtain screenshot analysis results.

[0053] After performing comprehensive semantic analysis on the user command data and the text analysis data to obtain the screenshot analysis results, the method also includes: based on a preset display interface configuration, typeset the screenshot analysis results to obtain an initial display interface of the screenshot analysis results; according to the initial display interface of the screenshot analysis results, obtain a display interface of the screenshot analysis results through real-time configuration of the display interface.

[0054] After obtaining the display interface of the screenshot analysis results through real-time configuration of the display interface according to the initial display interface of the screenshot analysis results, the method also includes: storing the screenshot analysis results to a preset cache address, and periodically scanning the storage space of the cache address to determine the cache clearing node; when the remaining storage space of the cache address is lower than a preset storage threshold, resetting the cache space of the cache address.

[0055] This application obtains user command data and performs comprehensive semantic analysis on the user command data and text analysis data to obtain screenshot analysis results, thereby realizing the automation of screenshot analysis, improving the accuracy of screenshot analysis results, supporting the Ollama framework to call local inference models, reducing cloud dependence and improving the security of user privacy.

[0056] In the examples of the present application, this is explained in detail through the following Example 5.

[0057] Example 5: After the image analysis is completed, the module will return the generated text results to the user interface for user viewing and further operation.

[0058] The user interaction interface module is developed based on PyQt6 and is mainly composed of the "main page" and "settings page", which makes it easy for users to view analysis results and configure system parameters.

[0059] The top of the main page is used to display the images passed in by the screenshot detection module in real time, ensuring that the user can clearly view the image content currently being analyzed.

[0060] There is an input box at the bottom of the main page where users can directly ask questions or send instructions. The system will send the user's text input and screenshot content to the image analysis module. The text analysis results generated by the module are displayed in Markdown format, supporting mixed text and image layout, which improves the intuitiveness of interaction and reading experience.

[0061] The settings page is used to configure the API key, model ID, screenshot directory path, and the activation of the Ollama local model. The settings made by the user will be saved to the .env file to ensure that the system can still use the same configuration after restart.

[0062] The settings page also provides a memory reset button, so users can manually clean up the system cache to maintain system stability during long-term operation.

[0063] To ensure system stability and efficiency, the memory management module provides memory cleaning and configuration reset functions. This module effectively prevents memory leaks and keeps the system configuration clear during long-term operation by periodically or manually cleaning cache and resetting memory resources.

[0064] Users can choose to manually clean up the cache on the settings page to release unnecessary memory resources and optimize system performance. The module allows users to restore the configuration parameters on the settings page to their initial state at any time to solve problems caused by configuration changes during operation.

[0065] The above is an embodiment of the method proposed in this application. Based on the same inventive concept, the embodiment of this application also provides a visual language model interaction device based on screenshots, and its structure is as follows Figure 2 shown.

[0066] Figure 2 A schematic diagram of the internal structure of a visual language model interaction device based on screenshots provided in an embodiment of the present application. Figure 2 As shown, the device includes: at least one processor 201; and, a memory 202 communicatively connected to the at least one processor; The memory 202 stores instructions that can be executed by at least one processor, and the instructions are executed by at least one processor 201 to enable at least one processor 201 to: Based on the preset screenshot directory, perform timestamp traversal analysis on the directory timestamps of the screenshot directory to determine the user's current screenshot data; perform analysis model recommendation evaluation on the user's current screenshot data to obtain the preferred analysis model; according to the preferred analysis model, determine the calling function of the preferred analysis model by calling the function configuration; based on the calling function, input the user's current screenshot into the preferred analysis model to obtain text analysis data; obtain user command data, and perform comprehensive semantic analysis on the user command data and text analysis data to obtain the screenshot analysis result.

[0067] Some embodiments of the present application provide corresponding Figure 1 A non-volatile computer storage medium for interactive visual language model based on screenshots, storing computer executable instructions, wherein the computer executable instructions are set to: Based on the preset screenshot directory, perform timestamp traversal analysis on the directory timestamps of the screenshot directory to determine the user's current screenshot data; perform analysis model recommendation evaluation on the user's current screenshot data to obtain the preferred analysis model; according to the preferred analysis model, determine the calling function of the preferred analysis model by calling the function configuration; based on the calling function, input the user's current screenshot into the preferred analysis model to obtain text analysis data; obtain user command data, and perform comprehensive semantic analysis on the user command data and text analysis data to obtain the screenshot analysis result.

[0068] Each embodiment in this application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the IoT device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0069] The system and medium provided in the embodiments of the present application correspond one-to-one to the method. Therefore, the system and medium also have similar beneficial technical effects to the corresponding methods. Since the beneficial technical effects of the methods have been described in detail above, the beneficial technical effects of the system and medium will not be repeated here.

[0070] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware. Moreover, the present application may adopt the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program codes.

[0071] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the embodiments of the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to generate a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0072] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0073] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0074] In a typical configuration, a computing device includes one or more processors (CPU), input / output interfaces, network interfaces, and memory.

[0075] Memory may include non-permanent storage in a computer-readable medium, in the form of random access memory (RAM) and / or non-volatile memory, such as read-only memory (ROM) or flash RAM. Memory is an example of a computer-readable medium.

[0076] Computer readable media include permanent and non-permanent, removable and non-removable media that can be implemented by any method or technology to store information. Information can be computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disk read-only memory (CD-ROM), digital versatile disk (DVD) or other optical storage, magnetic cassettes, magnetic tape magnetic disk storage or other magnetic storage devices or any other non-transmission media that can be used to store information that can be accessed by a computing device. As defined herein, computer readable media does not include temporary computer readable media (transitory media), such as modulated data signals and carrier waves.

[0077] It should also be noted that the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, commodity or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, commodity or device. In the absence of more restrictions, the elements defined by the sentence "comprises a ..." do not exclude the existence of other identical elements in the process, method, commodity or device including the elements.

[0078] The above are only embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various changes and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application should be included within the scope of the claims of the present application.

Claims

1. A screenshot-based visual language model interaction method, characterized in that: The method comprises: Based on a preset screenshot directory, perform a timestamp traversal analysis on the directory timestamps of the screenshot directory to determine the user's current screenshot data; Performing an analysis model recommendation evaluation on the user's current screenshot data to obtain a preferred analysis model; According to the preferred analysis model, determining the calling function of the preferred analysis model by calling function configuration; Based on the calling function, the user's current screenshot is input into the preferred analysis model to obtain text analysis data; User instruction data is obtained, and a comprehensive semantic analysis is performed on the user instruction data and the text analysis data to obtain a screenshot analysis result.

2. The screenshot-based visual language model interaction method according to claim 1, characterized in that: Based on a preset screenshot directory, a timestamp traversal analysis is performed on the directory timestamps of the screenshot directory to determine the user's current screenshot data, specifically including: Obtaining a current timestamp, and based on the current timestamp, periodically traversing the screenshot directory to obtain the last modification time of the screenshot file; According to the last modification time of the screenshot file, the screenshot update data is determined by identifying the last check time; The current timestamp is updated to determine the detection timestamp after the current timestamp, and a storage path query is performed on the screenshot update data to obtain the user's current screenshot data; wherein the user's current screenshot data includes: the user's current screenshot and the user's current screenshot storage path.

3. The screenshot-based visual language model interaction method according to claim 1, characterized in that: Performing an analysis model recommendation evaluation on the user's current screenshot data to obtain a preferred analysis model, specifically including: Performing image type feature analysis on the current screenshot data to obtain a first recommended evaluation parameter; Performing image information feature analysis on the current screenshot data to obtain a second recommended evaluation parameter; Based on the first recommended evaluation parameter and the second recommended evaluation parameter, the preferred analysis model is determined through screenshot data processing performance analysis.

4. The screenshot-based visual language model interaction method according to claim 1, characterized in that: According to the preferred analysis model, determining the calling function of the preferred analysis model by calling function configuration specifically includes: In the case where the preferred analysis model is a cloud model, calling a completion function in a preset LiteLLM library to determine a first calling function; In the case where the preferred analysis model is a local model, calling a chat function in a preset Ollama library to determine a second calling function; A calling function of the preferred analysis model is determined based on the first calling function or the second calling function.

5. The screenshot-based visual language model interaction method according to claim 1, characterized in that: Based on the calling function, the user's current screenshot is input into the preferred analysis model to obtain text analysis data, specifically including: When the calling function is a completion function, encapsulating the current screenshot data of the user; Calling a preset API, and inputting the user's current screenshot data after data encapsulation into the API to obtain API parsing parameters; Based on the API parsing parameters, performing screenshot content analysis on the user's current screenshot data to obtain first text analysis data; In the case where the calling function is a chat function, loading an analysis model in a preset local environment, and obtaining second text analysis data by analyzing the screenshot content according to the analysis model; The text analysis data is obtained according to the first text analysis data or the second text analysis data.

6. The screenshot-based visual language model interaction method according to claim 1, characterized in that: Performing a comprehensive semantic analysis on the user instruction data and the text analysis data to obtain a screenshot analysis result, specifically including: Performing user intention analysis on the user instruction data to determine user instruction impact parameters; Based on the user instruction impact parameter, performing simulated instruction analysis on the text analysis data to obtain instruction analysis data of the text analysis data; The instruction analysis data is subjected to instruction intent verification to obtain the screenshot analysis result.

7. The screenshot-based visual language model interaction method according to claim 1, characterized in that: After performing comprehensive semantic analysis on the user instruction data and the text analysis data to obtain a screenshot analysis result, the method further includes: Based on a preset display interface configuration, the screenshot analysis results are typeset to obtain an initial display interface of the screenshot analysis results; According to the initial display interface of the screenshot analysis result, the display interface of the screenshot analysis result is obtained through real-time configuration of the display interface.

8. The screenshot-based visual language model interaction method according to claim 1, characterized in that: After obtaining the display interface of the screenshot analysis result through real-time configuration of the display interface according to the initial display interface of the screenshot analysis result, the method further includes: The screenshot analysis result is stored in a preset cache address, and a periodic storage space scan is performed on the cache address to determine a cache cleaning node; When the remaining storage space of the cache address is lower than a preset storage threshold, the cache space of the cache address is cache reset.

9. A screenshot-based visual language model interaction device, characterized in that: The device comprises: at least one processor; and, a memory communicatively coupled to the at least one processor; The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to: Based on a preset screenshot directory, perform a timestamp traversal analysis on the directory timestamps of the screenshot directory to determine the user's current screenshot data; Performing an analysis model recommendation evaluation on the user's current screenshot data to obtain a preferred analysis model; According to the preferred analysis model, determining the calling function of the preferred analysis model by calling function configuration; Based on the calling function, the user's current screenshot is input into the preferred analysis model to obtain text analysis data; User instruction data is obtained, and a comprehensive semantic analysis is performed on the user instruction data and the text analysis data to obtain a screenshot analysis result.

10. A non-volatile computer storage medium for a screenshot-based visual language model interaction method, storing computer executable instructions, characterized in that: The computer executable instructions are configured to: Based on a preset screenshot directory, perform a timestamp traversal analysis on the directory timestamps of the screenshot directory to determine the user's current screenshot data; Performing an analysis model recommendation evaluation on the user's current screenshot data to obtain a preferred analysis model; According to the preferred analysis model, determining the calling function of the preferred analysis model by calling function configuration; Based on the calling function, the user's current screenshot is input into the preferred analysis model to obtain text analysis data; User instruction data is obtained, and a comprehensive semantic analysis is performed on the user instruction data and the text analysis data to obtain a screenshot analysis result.

Images