Artificial intelligence intent real-time visualization method and system

Abstract

The application provides an artificial intelligence intention real-time visualization method and system, which is applied to a server, and the method comprises the following steps: acquiring a video stream and transmitting the video stream to a client; inputting frame images of the video stream into a preset visual model to acquire intention text and target physical coordinates of the frame images generated by inference of the visual model; and pushing the intention text and the target physical coordinates to the client through a server sending event channel. Through the establishment of a one-way high-frequency pushing mechanism, the client can synchronously render intention identification on the video stream, real-time space-time synchronous mapping of an artificial intelligence implicit decision process and a front-end video picture is realized, and the transparency and debuggability of an automatic execution process are effectively improved.

Description

Technical Field

[0001] This invention relates to the fields of artificial intelligence and human-computer interaction technology, and in particular to a method and system for real-time visualization of artificial intelligence intentions. Background Technology

[0002] In current AI vision automation and human-machine collaboration scenarios, visual models typically operate as black-box engines. Existing automated testing or assisted operation technologies rely on the backend acquiring screenshots of the device screen, inferring information from the model, and then directly outputting the final operation commands (such as clicks or swipes).

[0003] However, this processing mechanism has significant technical flaws: because the intermediate semantic information (i.e., why the action is performed) and spatial information (i.e., the specific physical region the model focuses on) generated by the visual model's inference only exist in the backend memory, and the traditional HTTP request-response mechanism has significant round-trip time (RTT), the video stream displayed on the front end and the inference logic on the backend cannot be decoupled and synchronized in real time along the time axis and spatial dimension. Users can only observe the final screen transition result, but cannot obtain the AI's decision-making basis before executing the operation or the physical landing point mapping of the target. This opacity of the execution logic leads to a lack of observability and error correction in complex UI environments, thereby reducing the efficiency of automated debugging and the degree of human-computer interaction trust. Summary of the Invention

[0004] The technical problem to be solved by the present invention is to provide a method and system for real-time visualization of artificial intelligence intent, which can realize real-time spatial mapping and semantic synchronous display of AI reasoning and decision-making process on video stream, and improve the observability of artificial intelligence execution logic.

[0005] To solve the above-mentioned technical problems, the technical solution adopted by the present invention is as follows: A method for real-time visualization of artificial intelligence intent, applied on a server-side, the method comprising: Receive video stream, raw resolution data, and intent text and target physical coordinates pushed by the server; Compare the original resolution data with the current playback container size data to calculate the video stream's scaling ratio and edge offset; The target physical coordinates are converted into screen coordinates that are adapted to the screen display area using the screen scaling ratio and the edge offset. Rendering of the intent identifier component is performed based on the intent text and the screen coordinates.

[0006] A method for real-time visualization of artificial intelligence intent, applied to a client, the method comprising: Receive video stream, raw resolution data, and intent text and target physical coordinates pushed by the server; Compare the original resolution data with the current playback container size data to calculate the video stream's scaling ratio and edge offset; The target physical coordinates are converted into screen coordinates that are adapted to the screen display area using the screen scaling ratio and the edge offset. Rendering of the intent identifier component is performed based on the intent text and the screen coordinates.

[0007] To solve the above-mentioned technical problems, another technical solution adopted by the present invention is as follows: A real-time visualization system for artificial intelligence intent includes a memory, a processor, and a computer program stored in the memory and executable on the processor. When the processor executes the computer program, it implements the steps of the real-time visualization method for artificial intelligence intent described above.

[0008] The beneficial effects of this invention are as follows: By constructing a parallel dual-channel data pipeline on the server side, decoupled transmission of underlying video data and high-level semantic metadata is achieved. By introducing a Server Send Event Channel (SSE), leveraging its unidirectional persistent connection characteristics, the structured intent text generated by the visual model and the target physical coordinates are pushed to the client in real time at a throughput matching the inference frequency. This solves the problem of spatiotemporal asynchrony between inference logic and rendering in traditional interaction modes. Spatially, by mapping discrete physical coordinates onto the video stream, visual anchoring of the AI's area of ​​interest is achieved. Semantically, the potential decision-making process of the black-box model is transformed into observable text labels. The two work synergistically, achieving real-time "white-box" display of AI behavior through rendering with a dynamic intent identifier component above the video playback layer. Compared to existing technologies, this solution eliminates the invisibility of decisions during automated execution, significantly reduces debugging feedback latency in human-machine collaboration scenarios, and improves the determinism and transparency of the system in complex UI operations. Attached Figure Description

[0009] Figure 1 This is a flowchart of the server-side steps of a real-time visualization method for artificial intelligence intent according to an embodiment of the present invention; Figure 2 This is a flowchart of the client-side steps of a real-time visualization method for artificial intelligence intent according to an embodiment of the present invention; Figure 3 This is a system architecture diagram of a real-time visualization system for artificial intelligence intent according to an embodiment of the present invention; Figure 4 This is a timing diagram of a real-time visualization method for artificial intelligence intent according to an embodiment of the present invention; Figure 5 This is a schematic diagram of an artificial intelligence intent real-time visualization system according to an embodiment of the present invention. Detailed Implementation

[0010] To explain in detail the technical content, objectives, and effects of the present invention, the following description is provided in conjunction with the embodiments and accompanying drawings.

[0011] In existing technologies, large-scale AI vision models have been widely applied in scenarios such as automated game testing, remote drone-assisted control, and automated inspection of smart terminals. In these typical applications, the system usually uses computer vision technology to capture real-time video streams from the device screen or camera, which are then processed by a large-scale vision model in the backend for image analysis and reasoning. Subsequently, the large-scale model directly outputs specific execution control commands (such as clicking a coordinate or swiping a certain area).

[0012] However, as the complexity of business logic increases, this black-box execution architecture exposes serious technical problems. Before the visual big model outputs the final action, the semantic decision-making basis (i.e., "why do it this way") and intermediate spatial information (i.e., "what target boundaries were specifically identified") generated internally are completely hidden in the server-side memory. Due to the latency caused by network and computing power between the real-time playback of the front-end video stream and the back-end inference actions, users can only observe sudden changes in the UI interface (the final execution result) on the front end, and cannot monitor the big model's thinking process in real time. When automated tasks produce recognition deviations or logical fallacies, testers and debuggers find it difficult to trace the root cause of the failure (unable to determine whether it is a semantic understanding error or a coordinate alignment deviation), resulting in extremely high troubleshooting costs and low system transparency in human-machine collaboration scenarios.

[0013] To address at least some of the aforementioned issues, this invention proposes a real-time visualization method for artificial intelligence intent. This method simultaneously captures and transmits video streams to the client from the server, feeding captured video frames into a visual model for inference, generating intent text and target physical coordinates that characterize the model's decision-making process. Subsequently, the server pushes this data to the client in real-time at high frequency through a one-way server event transmission channel (SSE), enabling the client to dynamically overlay and render intent identification components on the currently playing video stream based on this data.

[0014] In this way, real-time spatiotemporal decoupling and synchronous mapping between the underlying inference logic of the large model and the dynamic video feed at the front end can be achieved. By making implicit decision-making intentions explicit as highlighted boxes and text labels on the video stream, this system completely breaks the black-box state of AI automated execution, providing R&D and supervisory personnel with a zero-latency decision visualization and monitoring method, and significantly improving the efficiency of model error correction and the certainty of human-computer interaction.

[0015] The following describes in detail a real-time visualization method for artificial intelligence intent according to the present invention, applied to the server side. Please refer to [link / reference]. Figure 1 The method 100 includes steps 110 to 130: Step 110: Obtain the video stream and transmit it to the client.

[0016] For example, in a game UI automation testing scenario, the server continuously captures the screen of the target game application window to generate a video stream, and pushes the video stream to the client front-end where the tester is located for playback via protocols such as WebSocket.

[0017] Step 120: Extract the frame images of the video stream and input them into the preset visual model to obtain the intent text and target physical coordinates of the frame images generated by the visual model inference.

[0018] For example, the server extracts the latest frame from the video stream queue and inputs it into a multimodal vision model (such as Google's Gemini model) capable of processing both images and text simultaneously for real-time inference. After analyzing the current screen, the model outputs a result, where the intent text contains the natural language reason for the model's decision (e.g., "The 'Confirm' button in the dialog box is green, so clicking it will be performed"), and the target physical coordinates are the bounding box parameters of the button at the original screen resolution.

[0019] Step 130: Push the intent text and target physical coordinates to the client through the server's event channel.

[0020] For example, the server establishes a channel through the SSE (Server-Sent Events) protocol and pushes the generated intent text and target physical coordinates to the client front end in real time at a frequency consistent with the inference frame rate of the visual model (such as 10-15 FPS).

[0021] As described above, this embodiment constructs separate video transmission streams and model data analysis streams on the server side, and structures the invisible black-box inference logic into intent text and target physical coordinates, utilizing a unidirectional server-side event sending channel for high-frequency delivery. This architecture solves the latency and overhead problems caused by HTTP polling from the engineering level, ensuring that the decision semantics and spatial location of large models can reach the front end with near real-time throughput, providing a low-latency data foundation for subsequent visualization mapping.

[0022] In one embodiment of the present invention, step 110, acquiring the video stream and transmitting the video stream to the client, includes steps 111 to 112: Step 111: Extract the size information of the video frames in the video stream and use the size information as the original resolution data.

[0023] For example, when encapsulating each frame of video data, the server extracts the original width and height (such as W_src, H_src) of the currently captured image and establishes them as the original resolution data.

[0024] Step 112: Send the raw resolution data to the client along with the video stream.

[0025] For example, the server may use the raw resolution data as metadata and write it into a preset field in the header or message body of a custom transport protocol, and send it to the client for parsing along with the video stream image data.

[0026] As described above, since the physical coordinates output by the visual model are generated based on the original dimensions during video capture, severe positional misalignment will occur when the client zooms in and out if this size reference is not passed. By extracting and transmitting the original resolution data along with the stream, a unified spatial measurement benchmark is established between the server and the client, ensuring the absolute accuracy of subsequent coordinate transformations.

[0027] In one embodiment of the present invention, step 120, which extracts frame images from the video stream and inputs them into a preset visual model, includes steps 121 to 122: Step 121: Extract frame images from the video stream at a preset frequency.

[0028] For example, the server extracts a single screenshot as a frame image from the continuously generated video stream queue according to a preset sampling frequency (such as 10 FPS).

[0029] Step 122: Obtain the preset task test instructions and input the frame image and task test instructions into the preset visual model.

[0030] For example, the server obtains the currently preset automated execution command (such as the command "click the store button"), and submits the text command as a prompt word along with the captured frame image to the multimodal vision model for joint analysis.

[0031] As described above, by combining fixed-frequency frame sampling with specific task test instructions, the system introduces clear contextual constraints to the visual model, thereby guiding the model to focus on UI elements that are strongly related to the task, avoiding the model's attention divergence in complex interfaces, and improving the effectiveness of generating target physical coordinates and the semantic accuracy of intent text.

[0032] In one embodiment of the present invention, step 120, which involves obtaining the intent text and target physical coordinates of the frame image generated by visual model inference, includes steps 123 to 125: Step 123: Obtain the structured format data generated by the visual model inference.

[0033] For example, the server receives data packets in JSON format output by the visual model.

[0034] Step 124: Extract the natural language string of action decision from the structured data as the intent text.

[0035] For example, the server parses the JSON data and extracts the value of the field with the key name ai_reasoning (i.e., the decision description directly generated by the model) as the intent text.

[0036] Step 125: Extract the coordinate set of the target boundary position from the structured format data as the target physical coordinates.

[0037] For example, the server parses the JSON data, extracts the coordinate array with the key name target_box (such as [ymin, xmin, ymax, xmax]), and confirms it as the target physical coordinates of the UI element.

[0038] As described above, the raw output of large visual models often contains irregular information. By limiting the model output to a structured format and performing targeted key-value pair extraction, semantic logic (text) and spatial geometry (coordinates) are decoupled in engineering. This data preprocessing eliminates data noise, providing a clear and standardized payload for efficient encapsulation of subsequent signaling transmission.

[0039] In one embodiment of the present invention, before pushing the intent text and target physical coordinates to the client through the server's event channel in step 130, steps 101 to 102 are further included: Step 101: Receive the connection request initiated by the client.

[0040] For example, when the client detects that the webpage has finished loading, its frontend actively initiates an EventSource request via JavaScript, and the server receives the connection request.

[0041] Step 102: In response to the connection request, establish a transmission channel with the client as a channel for the server to send events.

[0042] For example, the server receives the request and maintains a persistent connection, establishing a dedicated server event channel for the server to send data unidirectionally to the client.

[0043] As described above, compared to the traditional request-response mechanism, the unidirectional long-lived connection channel, which is actively triggered by the client and kept alive on the server, avoids the network overhead caused by frequent handshakes. This channel is specifically designed for high-frequency, unidirectional real-time intent push, ensuring high data throughput and low-latency delivery.

[0044] In one embodiment of the present invention, step 130, which involves pushing the intent text and target physical coordinates to the client via a server-side event channel, further includes step 131: Step 131: Obtain the preset display duration parameter, and push the preset display duration parameter, along with the intent text and target physical coordinates, to the client.

[0045] For example, the server obtains the system's preset highlight duration (e.g., a specified duration of 2 seconds), and pushes it to the client along with the generated JSON format intent and coordinates in the SSE message body.

[0046] As described above, the server sends the preset display duration parameter along with the target physical coordinates, transferring the rendering control logic of the intent identifier component to the client for local execution. This mechanism eliminates the need for the server to initiate subsequent network requests to control the component's disappearance, effectively reducing the concurrent signaling overhead of the server's event channel, while also improving the independence of client interface rendering state management and system fault tolerance.

[0047] In one embodiment of the present invention, steps 141 to 144 are further included: Step 141: Establish a history queue.

[0048] For example, the server initializes a fixed-capacity data structure in memory (such as storing the most recent 50 intent data by default, and discarding the oldest record if the limit is exceeded) as a history queue.

[0049] Step 142: Obtain the generation timestamp when the visual model infers and generates the intent text and target physical coordinates, and persistently store the intent text, target physical coordinates and generation timestamp in the history queue.

[0050] For example, when generating intent data, the server records the current absolute time as the generation timestamp, and writes the intent text, target physical coordinates, and generation timestamp together into the aforementioned historical record queue for long-term storage.

[0051] Step 143: When a backtracking request is received from the client, the target timestamp is parsed from the backtracking request.

[0052] For example, when a user reviews their history on the client, the client sends a request containing a time parameter, which the server receives and extracts as the target timestamp.

[0053] Step 144: Extract the intent text and target physical coordinates that match the target timestamp from the history queue and return them to the client.

[0054] For example, the server can accurately find the original coordinates and intent text corresponding to that frame from a fixed-capacity queue based on the target timestamp and return it to the client.

[0055] As described above, real-time streaming is characterized by its fleeting nature. By introducing timestamps to anchor transient AI decisions onto the timeline and establishing a persistent storage queue with fixed capacity, the system is provided with time-slice retrieval capabilities. This allows testers to backtrack the input and output states of the large model when errors occur in the automated process, shortening the troubleshooting process.

[0056] The following details a method for real-time visualization of artificial intelligence intent according to the present invention. Please refer to [link / reference]. Figure 2 The method 200 is applied to the client and includes steps 210 to 240: Step 210: Receive the video stream, raw resolution data, and intent text and target physical coordinates pushed from the server.

[0057] For example, the client-side front-end player module continuously receives and renders the video stream, obtaining the original resolution (W_src, H_src) carried by the stream, while the SSE receiving module listens for new data and parses out the intent text and physical coordinates.

[0058] Step 220: Compare the original resolution data with the current playback container size data, and calculate the video stream's scaling ratio and edge offset.

[0059] For example, the client obtains the current actual physical size (W_cont, H_cont) of the player container and compares it with the aspect ratio of the original resolution. If the two are inconsistent, the client calculates the scaling ratio parameter of the actual video image to adapt to the container, as well as the possible width of the white space and black border, as the edge offset.

[0060] Step 230: Use the screen scaling ratio and edge offset to convert the target physical coordinates into screen coordinates that fit the screen display area.

[0061] For example, the client uses a coordinate mapping method in this field to multiply the target physical coordinates sent by the server by the screen scaling ratio and add edge offset compensation caused by the black border to calculate the accurate pixel position of the target element on the current browser screen, that is, the screen coordinates (x_scr, y_scr, w_scr, h_scr).

[0062] Step 240: Render the intent identifier component on the video stream based on the intent text and screen coordinates.

[0063] For example, the client renders a highlighted box at the corresponding position on the video screen based on the calculated precise screen coordinates, and attaches intent text to form an intent identifier component.

[0064] As described above, due to dynamic changes in the size of the playback container on the terminal device (such as window scaling or dragging) and display white space caused by aspect ratio mismatch, directly applying the absolute physical coordinates generated by the server to the client will inevitably lead to spatial mapping misalignment. By obtaining the original video resolution and the current size of the playback container, a two-dimensional coordinate transformation mechanism based on scaling ratio calculation and white space offset compensation is introduced. This mechanism not only eliminates scale and displacement errors caused by different display terminals, but also ensures that the dynamically generated intent identifier components can accurately and adaptively overlay the corresponding interactive targets in the video stream without changing the underlying video pixels.

[0065] In one embodiment of the present invention, the rendering of the intent identification component based on the intent text and screen coordinates in step 240 includes steps 241 to 242: Step 241: Create a separate transparent canvas layer above the video stream playback layer.

[0066] For example, the client-side front-end maintains a transparent Canvas, which is absolutely positioned and overlaid on top of the video player element using CSS (Cascading Style Sheets) technology.

[0067] Step 242: Draw the intent identifier component on the transparent canvas layer based on screen coordinates, and write the intent text into the intent identifier component.

[0068] For example, the client-side visualization rendering module draws a semi-transparent highlighted rectangle on the screen coordinate area of ​​the transparent Canvas, and attaches a label containing the text in the intent text above the rectangle. If a transition animation is introduced, the component can also fade out. Each component has an independent lifecycle and does not interfere with each other.

[0069] As described above, by constructing a transparent overlay layer independent of the video playback layer, the system achieves physical separation between the UI marker rendering and the underlying video hardware decoding pipelines. This non-intrusive rendering mechanism not only ensures that the original video quality is not compromised, but also avoids screen chaos during long-term occlusion and concurrent multi-intent processing by relying on independent component lifecycle management, thus achieving a high-quality human-computer interaction visual experience.

[0070] The following is mainly based on Figure 4 (A sequence diagram of a real-time visualization method for artificial intelligence intent) shows the execution mainline, combined with... Figure 3The module flow in (an architecture diagram of a real-time visualization system for artificial intelligence intent) is explained in detail step by step, outlining the specific implementation logic of this invention: Step 1: Preparation for video stream transmission and playback (corresponding to...) Figure 4 Step 1).

[0071] server (corresponding) Figure 3 The video capture module continuously captures real-time footage of the target application process, generating a continuous video stream in a standard format (equivalent to...). Figure 3 In step 1), during the video stream construction phase, the server encapsulates the absolute size of the captured video frame as raw resolution data (metadata) into the streaming channel. The server then continuously transmits this video stream to the client's video playback module (equivalent to...). Figure 3 Step 6), thus providing the underlying media environment for subsequent physical mapping and image display.

[0072] Step 2: Analyze video resolution and calculate rendering parameters (corresponding) Figure 4 Step 2).

[0073] After receiving the video stream, the client's front-end video playback module parses out the raw resolution data sent with the stream and obtains the actual rendering size of the current client player container (equivalent to...). Figure 3 Step 7); subsequently, the client (corresponding to Figure 3 The rendering positioning module in the middle compares these two sets of size data, and calculates the screen scaling ratio of the video stream during adaptive rendering in the current container based on the difference in their aspect ratios, as well as the edge offset caused by whitespace (equivalent to...). Figure 3 (Step 8) This calculation process provides compensation parameters for screen adaptive mapping of different terminal devices.

[0074] Step 3: AI analyzes video frames to generate operation intent text and outputs the target's physical coordinates (corresponding to...). Figure 4 Step 3).

[0075] While video streaming is running in parallel, the server (corresponding to...) Figure 3 The AI ​​inference module extracts frame images from the video stream according to a preset sampling frequency (equivalent to...). Figure 3 Step 2) inputs the image into a preset visual model; the visual model performs real-time analysis of the frame image based on preset instructions, and after inference, outputs a structured result containing the intent text representing the operation decision and the bounding box parameters surrounding the target interactive element, i.e., the server obtains the target's physical coordinates (equivalent to...). Figure 3 Step 3) completes the transformation of implicit execution logic into explicit structured data.

[0076] Step 4: SSE pushes the intent text and physical coordinates (corresponding to...) Figure 4 (SSE push after step 3).

[0077] server (corresponding) Figure 3 The SSE service module in the system pre-establishes a one-way persistent connection channel with the client. When the server generates the latest inference data, it encapsulates the aforementioned intent text and target physical coordinates, continuously injects them into the downlink through the established server-side event transmission channel, and pushes them one-way to the client (corresponding to...). Figure 3 (Step 4). By utilizing the streaming unidirectional transmission characteristic of this channel, the server ensures that the data output by the model closely follows the playback progress of the underlying video stream in terms of timing.

[0078] Step 5: Convert physical coordinates to screen coordinates based on rendering parameters (corresponding to...) Figure 4 Step 4).

[0079] The client's SSE module passes the received intent data to the visualization module (equivalent to...). Figure 3 (Step 5) The client uses the screen scaling ratio and edge offset obtained in Step 2 above to perform a mapping transformation on the received target physical coordinates, converting them into screen coordinates that are suitable for the current screen display area; then, the client draws a highlighted bounding box and rendering intent text label on a transparent overlay layer independent of the video layer, based on the screen coordinates, to form an intent identification component.

[0080] Step 6: Intent History Review (corresponding) Figure 4 (The opt intention history review process).

[0081] The server establishes a historical record queue with a fixed capacity strategy (e.g., storing a few most recent records and discarding the oldest records when the limit is reached). When retrieving the visual model's inference results, the server persistently stores the intent text, target physical coordinates, and generation timestamp in this queue. In exception handling and debugging scenarios, the client responds to the user's "pause" command by immediately freezing the current video stream and sending a backtracking request to the server. The server receives the request, parses the target timestamp, extracts the matching intent data from the historical record queue, and returns it to the client. After receiving the data, the client expands a debugging panel on the side of the interface (displaying the timestamp, intent text, coordinate data, and inference log for that moment). Simultaneously, the client temporarily redraws the intent identifier component for that historical moment on the frozen video frame, allowing testers to compare with expectations and quickly locate problems.

[0082] As described above, the server extracts video frames as input to the visual model to generate intent text and target physical coordinates. The server then establishes a server-sending event channel to independently push this structured data to the client. Simultaneously, combined with the original resolution data, the client calculates the screen scaling ratio and edge offset and performs coordinate transformation. This mechanism decouples the underlying video media stream from the model's high-level inference data stream on the transmission link. It utilizes the high-frequency unidirectional transmission characteristics of the event channel to maintain temporal alignment between the two and eliminates spatial errors in multi-terminal display interfaces through an adaptive coordinate compensation algorithm. Compared to a single execution mode that only outputs the final execution result, this invention can accurately and in real-time map the model's implicit semantic decisions and spatial coordinates onto the corresponding dynamic video frames, improving the system observability and anomaly debugging efficiency of the automated execution process.

[0083] In summary, the real-time visualization method and system for artificial intelligence intent provided in this application continuously captures video streams and extracts frame images as input to a visual model on the server side, generating structured intent text and target physical coordinates. The server then uses a server-side event channel to unidirectionally push this inference data to the client, simultaneously sending raw resolution data along with the video stream. This allows the client to calculate the screen scaling ratio and edge offset based on the raw resolution and the current playback container size. The client then converts the target physical coordinates into screen coordinates and renders the intent identifier component on top of the video stream. In this implementation, the solution separates the video image transmission link from the model data push link in the transmission architecture, utilizing the persistent connection characteristics of the event channel to maintain temporal synchronization between intent data and video images. In terms of image mapping logic, an adaptive coordinate compensation calculation mechanism is introduced to correct the relative positional deviations caused by size scaling and black borders on different terminal displays, achieving accurate conversion from the server's raw image coordinates to the client's current screen coordinates. This solution effectively solves the problems of the inability to track the inference process in real time and the difficulty in aligning the screen display with the backend data during the automated execution of existing visual models. It transforms the internal decision-making logic into intuitive markers on the interface in real time, improving the system transparency and anomaly detection efficiency in automated operation scenarios.

[0084] Please refer to Figure 5 The present invention also provides a real-time visualization system for artificial intelligence intent, including a server device 510 and a client device 520: The server device 510 includes a memory 511, a processor 512, and a computer program stored in the memory and executable on the processor. When the processor 512 executes the computer program, it implements the various steps of the above-described real-time visualization method for artificial intelligence intent on the server side.

[0085] The client device 520 includes a memory 521, a processor 522, a display 523, and a computer program stored in the memory and executable on the processor. When the processor 522 executes the computer program, it implements the various steps of the aforementioned real-time visualization method client for artificial intelligence intent and displays the synthesized video image through the display 523.

[0086] The above description is merely an embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the content of the present invention specification and drawings, or direct or indirect applications in related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. A method for real-time visualization of artificial intelligence intent, characterized in that, Applied to the server side, the method includes: Acquire the video stream and transmit the video stream to the client; The frame images of the video stream are extracted and input into a preset visual model to obtain the intent text and target physical coordinates of the frame images generated by the visual model inference. The intent text and the target physical coordinates are pushed to the client via the server's event sending channel.

2. The method for real-time visualization of artificial intelligence intent according to claim 1, characterized in that, The step of acquiring the video stream and transmitting the video stream to the client includes: Extract the size information of the video frames in the video stream and use the size information as the original resolution data; The original resolution data is sent to the client along with the video stream.

3. The method for real-time visualization of artificial intelligence intent according to claim 1, characterized in that, The step of extracting frame images from the video stream and inputting them into a preset visual model includes: The frame images are extracted from the video stream at a preset frequency; Obtain a preset task test instruction, and input the frame image and the task test instruction into a preset visual model.

4. The method for real-time visualization of artificial intelligence intent according to claim 1, characterized in that, The step of obtaining the intent text and target physical coordinates of the frame image generated by the visual model inference includes: Obtain the structured format data generated by the visual model inference; Extract the natural language string of the action decision from the structured data as the intent text; The coordinate set of the target boundary position is extracted from the structured format data and used as the target physical coordinates.

5. The method for real-time visualization of artificial intelligence intent according to claim 1, characterized in that, Before pushing the intent text and the target physical coordinates to the client via the server's event channel, the method further includes: Receive the connection request initiated by the client; In response to the connection request, a transmission channel is established with the client as the server's event sending channel.

6. The method for real-time visualization of artificial intelligence intent according to claim 1, characterized in that, When pushing the intent text and the target physical coordinates to the client via the server's event channel, the method further includes: Obtain the preset display duration parameter, and push the preset display duration parameter, the intent text, and the target physical coordinates to the client.

7. The method for real-time visualization of artificial intelligence intent according to claim 1, characterized in that, The method further includes: Establish a historical record queue; Obtain the generation timestamp when the visual model infers and generates the intent text and the target physical coordinates, and persistently store the intent text, the target physical coordinates, and the generation timestamp in the history queue; When a backtracking request is received from the client, the target timestamp is parsed from the backtracking request; The intent text matching the target timestamp and the target physical coordinates are extracted from the historical record queue and returned to the client.

8. A method for real-time visualization of artificial intelligence intent, characterized in that, Applied to a client, the method includes: Receive video stream, raw resolution data, and intent text and target physical coordinates pushed by the server; Compare the original resolution data with the current playback container size data to calculate the video stream's scaling ratio and edge offset; The target physical coordinates are converted into screen coordinates that are adapted to the screen display area using the screen scaling ratio and the edge offset. Rendering of the intent identifier component is performed based on the intent text and the screen coordinates.

9. The method for real-time visualization of artificial intelligence intent according to claim 8, characterized in that, The step of rendering the intent identifier component based on the intent text and the screen coordinates includes: Create a separate transparent canvas layer above the playback layer of the video stream; The intent identifier component is drawn on the transparent canvas layer based on the screen coordinates, and the intent text is written into the intent identifier component.

10. A real-time visualization system for artificial intelligence intent, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements each step of the real-time visualization method for artificial intelligence intent as described in any one of claims 1 to 7.

Images