Remote assistance method and system based on AI visual identification and interactive labeling

Abstract

The invention relates to a remote assistance method and system based on AI visual identification and interactive annotation. The method comprises the following steps: a target expert terminal establishes a remote assistance session with a target client; the target client side captures a screen image frame of a local screen and sends the screen image frame to the target expert side and the AI visual recognition module in a screen image flow mode; the AI visual identification module receives and processes the screen image flow to obtain an AI visual identification result; the target expert end generates a background layer according to the screen image stream and the AI visual recognition result, and constructs an interactive whiteboard superposition layer on the background layer; when labeling is carried out, the target expert end converts a labeling action into labeling instruction data and sends the labeling instruction data to the target client; and rendering the corresponding annotation content on the whiteboard superposition layer above the local screen of the target client.

Description

TECHNICAL FIELD

[0001] The application belongs to the technical field of computer networks, and particularly relates to a remote assistance method and system based on AI visual recognition and interactive labeling. BACKGROUND

[0002] In a remote assistance scenario, the traditional remote assistance method is for the assisted object to rely on screen recording, voice guidance or static shared documents to let the remote assistance party assist, which may cause problems such as information asynchronization, difficulty in intuitively conveying operation steps, and low communication efficiency.

[0003] When the assisted object is a non-technical user, the traditional remote assistance method cannot adapt to dynamic changes in screen content in real time, for example, software interface jumping, error pop-up windows, etc., resulting in delayed assistance guidance or misunderstanding; and the existing remote assistance tools lack intelligent analysis capability for screen content, and cannot automatically generate a visual operation path, further reducing collaboration efficiency. SUMMARY

[0004] To solve the above technical problems, the application provides a remote assistance method and system based on AI visual recognition and interactive labeling.

[0005] The technical solution adopted by the application is as follows:

[0006] In a first aspect, a remote assistance method based on AI visual recognition and interactive labeling is provided, which is applied to a remote assistance system based on AI visual recognition and interactive labeling. The remote assistance system includes a plurality of client terminals, a plurality of expert terminals, a central server and an AI visual recognition module. The remote assistance method includes the following steps:

[0007] When a client using a target client terminal needs remote assistance, the target client terminal sends a remote assistance request to the corresponding target expert terminal through the central server;

[0008] The target expert terminal establishes a remote assistance session with the target client terminal according to the remote assistance request;

[0009] The target client terminal captures a screen image frame of the local screen and sends it to the target expert terminal and the AI visual recognition module in the form of a screen image stream;

[0010] The AI visual recognition module receives and processes the screen image stream to obtain AI visual recognition results, and sends the AI visual recognition results to the target expert terminal;

[0011] The target expert terminal generates a background layer according to the screen image stream and the AI visual recognition results, and constructs an interactive whiteboard overlay layer on the background layer;

[0012] When the target expert makes annotations on the interactive whiteboard overlay layer, the target expert end converts the annotation action into annotation instruction data and sends it to the target client;

[0013] The target client receives and parses the annotation instruction data and renders the corresponding annotation content on the whiteboard overlay layer above the local screen.

[0014] Further, the target client sends the remote assistance request to the corresponding target expert end through the central server, including:

[0015] The target client receives the remote assistance instruction initiated by the customer and generates a remote assistance request according to the remote assistance instruction; the remote assistance request contains the client identifier and the customer identity identifier;

[0016] The target client sends the remote assistance request to the central processor;

[0017] The central processor determines the remote assistance task level according to the customer identity identifier in the remote assistance request; according to the remote assistance task level, the pre-set task level-expert level relationship table is queried to determine the expert identity identifier and the expert end identifier corresponding to the remote assistance task level;

[0018] The central processor sends the remote assistance request to the target expert end corresponding to the expert end identifier.

[0019] Further, the target expert end establishes a remote assistance session with the target client according to the remote assistance request, including:

[0020] The target expert end generates a remote assistance acceptance notification according to the remote assistance request;

[0021] After the remote assistance is accepted, the target expert end establishes a secure transmission channel with the target client through the central server;

[0022] The target expert end sends a screen sharing request to the target client through the secure transmission channel;

[0023] The target client generates a screen sharing authorization notification according to the screen sharing request;

[0024] After the screen sharing authorization, the remote assistance session between the target client and the target expert end is successfully established.

[0025] Further, the method further includes:

[0026] The target client receives the screen sharing authorization instruction corresponding to the screen sharing authorization notification and acquires the screen sharing area range; the screen sharing area range is one or more screen areas in the local screen of the target client.

[0027] Further, the AI visual recognition module is deployed on the target client, the target expert terminal or the central server,

[0028] The target client captures a screen image frame of a local screen and sends the screen image frame to the target expert terminal and the AI visual recognition module in the form of a screen image stream, including:

[0029] The target client captures a plurality of screen image frames of a local screen, and the screen image frames include a client original logical resolution, a client DPI scaling ratio and image encoding.

[0030] The target client performs image compression and fragmentation on the plurality of screen image frames to form a screen image stream and sends the screen image stream to the target expert terminal.

[0031] The target client copies a copy of the screen image stream and sends the copy to the AI visual recognition module.

[0032] Further, the AI visual recognition module receives and processes the screen image stream to obtain an AI visual recognition result and sends the AI visual recognition result to the target expert terminal, including:

[0033] The AI visual recognition module receives the screen image stream to obtain an original screen image frame in the screen image stream.

[0034] The AI visual recognition module pre-processes the original screen image frame to obtain an optimized screen image frame.

[0035] The AI visual recognition module performs UI element recognition and text extraction on the optimized screen image frame based on a preset AI visual recognition model to obtain UI element information and text information.

[0036] The AI visual recognition module generates an AI visual recognition result according to the UI element information and the text information and sends the AI visual recognition result to the target expert terminal.

[0037] Further, the AI visual recognition module performs UI element recognition and text extraction on the optimized screen image frame based on a preset AI visual recognition model to obtain UI element information and text information, including:

[0038] The AI visual recognition module performs UI element recognition on the optimized screen image frame based on a preset AI visual recognition model to obtain UI elements and a bounding box coordinate.

[0039] The AI visual recognition module analyzes the original screen image frame to obtain a client original logical resolution and a client DPI scaling ratio and determines a client original logical pixel coordinate system.

[0040] The AI visual recognition module reversely maps the bounding box coordinate to the client original logical pixel coordinate system to obtain UI element coordinates and forms UI element information according to the UI elements and the UI element coordinates.

[0041] The AI visual recognition module extracts text strings of the text region and text region coordinates according to a preset file detection model or UI element information;

[0042] The AI visual recognition module reversely maps the text region coordinates to a client original logical pixel coordinate system to obtain text coordinates, and forms text information according to the text strings and the text coordinates.

[0043] Further, the target expert end generates a background layer according to the screen image stream and the AI visual recognition result, and constructs an interactive whiteboard superimposed layer on the background layer, including:

[0044] The target expert end obtains a rendering area size of a display area in which a client screen sharing picture is displayed;

[0045] The target expert end calculates an expert end scaling ratio according to the rendering area size and a client original logical resolution;

[0046] The target expert end processes UI element coordinates and text coordinates in the AI visual recognition result according to the expert end scaling ratio to obtain UI element pixel coordinates and text pixel coordinates corresponding to the display area;

[0047] The target expert end displays the screen image stream in the display area according to the rendering area size, the expert end scaling ratio, the UI element pixel coordinates and the text pixel coordinates to form the background layer;

[0048] The target expert end constructs a transparent interactive whiteboard superimposed layer on the background layer.

[0049] Further, after the target expert end constructs the transparent interactive whiteboard superimposed layer on the background layer, the target expert end further includes:

[0050] The target expert end draws special marks in the interactive whiteboard superimposed layer according to the UI element pixel coordinates and the text pixel coordinates to prompt a border of the UI element and the text region.

[0051] In a second aspect, a remote assistance system based on AI visual recognition and interactive labeling is provided, including:

[0052] A plurality of client ends, a plurality of expert ends, a central server and an AI visual recognition module;

[0053] When a client using a target client end needs remote assistance, the target client end is configured to send a remote assistance request to a corresponding target expert end through the central server;

[0054] The target expert end is configured to establish a remote assistance session with the target client end according to the remote assistance request;

[0055] The target client is also used to capture screen image frames of the local screen and send them to the target expert terminal and the AI ​​visual recognition module respectively as screen image streams;

[0056] The AI ​​visual recognition module is used to receive and process the screen image stream to obtain the AI ​​visual recognition result, and then send the AI ​​visual recognition result to the target expert terminal.

[0057] On the target expert side, it is also used to generate a background layer based on the screen image stream and AI visual recognition results, and to build an interactive whiteboard overlay layer on the background layer;

[0058] When the target expert makes annotations on the interactive whiteboard overlay, the target expert's end is also used to convert the annotation action into annotation instruction data and send it to the target client;

[0059] The target client is also used to receive and parse annotation instruction data, and render the corresponding annotation content on the whiteboard overlay at the top of the local screen.

[0060] The beneficial effects achieved by this invention are as follows:

[0061] When a client using the target client needs remote assistance, the target client sends a remote assistance request to the corresponding target expert terminal through a central server. The target expert terminal establishes a remote assistance session with the target client based on the request. The target client captures screen image frames from its local screen and sends them as screen image streams to both the target expert terminal and the AI ​​visual recognition module. The AI ​​visual recognition module receives and processes the screen image stream to obtain the AI ​​visual recognition result, which is then sent to the target expert terminal. The target expert terminal generates a background layer based on the screen image stream and the AI ​​visual recognition result, and builds an interactive whiteboard overlay on top of the background layer. When the target expert annotates on the interactive whiteboard overlay, the target expert terminal converts the annotation action into annotation instruction data and sends it to the target client. The target client receives and parses the annotation instruction data, rendering the corresponding annotation content on the whiteboard overlay above its local screen. Utilizing AI visual recognition to identify screen images helps experts quickly understand the client's local screen content. The target expert terminal generates a background layer based on the screen image stream and the AI ​​visual recognition result, and builds an interactive whiteboard overlay on top of the background layer, allowing for real-time and visual display of the expert's remote assistance content. Attached Figure Description

[0062] Figure 1 This is a flowchart of the remote assistance method based on AI visual recognition and interactive annotation of the present invention;

[0063] Figure 2 This is a result diagram of the remote assistance system based on AI visual recognition and interactive annotation of the present invention. DETAILED DESCRIPTION

[0064] The application will be further described below with reference to the drawings. The following examples are only used to more clearly illustrate the technical solutions of the application, and cannot be used to limit the protection scope of the application.

[0065] The remote assistance system based on AI visual recognition and interactive labeling includes a plurality of client terminals, a plurality of expert terminals, a central server and an AI visual recognition module. The AI visual recognition module can be deployed on a local client terminal, an expert terminal or the central server. The plurality of client terminals and the plurality of expert terminals are network-connected to the central server.

[0066] A stable and sufficient-bandwidth network connection must be established between the expert terminal and the client terminal, so as to ensure that the screen content can be smoothly transmitted from the client terminal to the expert terminal, the AI visual recognition result can be timely fed back, and the labeling of the expert terminal can be synchronously displayed on the local screen of the client terminal. Network instability or insufficient bandwidth will cause screen freezing, labeling delay and AI recognition lag, which seriously affects the user experience and assistance efficiency.

[0067] The expert terminal and the client terminal need to use a hardware device (such as a personal computer, a notebook computer, a tablet computer, which can be extended to a smart phone in the future) and a compatible operating system (such as Windows, macOS, Linux, Android, iOS, etc.) that can support the software (or web application) running of the remote assistance client terminal / expert terminal of the application. The normal installation and running of the software, as well as the functions of screen content capture, AI module calling and whiteboard overlay layer rendering, all depend on the support of the underlying hardware and operating system.

[0068] As shown in FIG. 1, the embodiment of the application provides a remote assistance method based on AI visual recognition and interactive labeling, which includes the following steps: Figure 1

[0069] 101. When a client using a target client terminal needs remote assistance, the target client terminal sends a remote assistance request to a corresponding target expert terminal through the central server.

[0070] In this embodiment, when a client using a target client terminal needs remote assistance, the client needs to initiate a request to the expert terminal. The specific process is as follows:

[0071] The target client terminal receives the remote assistance instruction initiated by the client, and generates a remote assistance request according to the remote assistance instruction. The remote assistance request includes a client terminal identifier and a client identity identifier. The client terminal identifier can be the MAC address of a personal computer, which is used to uniquely identify the physical device of the client terminal. The client identity identifier is a user ID.

[0072] ​The target client sends the remote assistance request to the central processor;

[0073] The central processor determines the remote assistance task level according to the customer identity in the remote assistance request, for example, retrieves a user database using a user ID, determines the consumption amount of the user ID, and determines the remote assistance task level according to the size of the amount; queries a preset task level-expert level relationship table according to the remote assistance task level to determine the expert identity corresponding to the remote assistance task level and the expert end identifier; the preset task level-expert level relationship table is to improve the use experience of customers of different levels and increase the service provided by experts of different levels;

[0074] The central processor sends the remote assistance request to the target expert end corresponding to the expert end identifier.

[0075] 102, the target expert end establishes a remote assistance session with the target client according to the remote assistance request;

[0076] The target expert end generates a remote assistance acceptance notification according to the remote assistance request; the remote assistance acceptance notification is displayed on the screen of the target expert end in the form of a pop-up window, so that the expert can click to accept;

[0077] After the remote assistance is accepted, the target expert end establishes a secure transmission channel with the target client through the central server; the secure transmission channel can be a WebSocket or a WebRTC data channel based on the Transport Layer Security (TLS / DTLS) data packet transmission layer security protocol;

[0078] The target expert end sends a screen sharing request to the target client through the secure transmission channel;

[0079] The target client generates a screen sharing authorization notification according to the screen sharing request; the screen sharing authorization notification is displayed on the screen of the target client in the form of a pop-up window to prompt the client to click;

[0080] After the screen sharing authorization, the remote assistance session between the target client and the target expert end is successfully established;

[0081] The target client receives a screen sharing authorization instruction corresponding to the screen sharing authorization notification, and obtains a screen sharing area range; the screen sharing area range is one or more screen areas in the local screen of the target client.

[0082] 103, the target client captures a screen image frame of the local screen and sends it to the target expert end and an AI visual recognition module in the form of a screen image stream;

[0083] The target client should capture the screen content of the local screen as much as possible in the native logical resolution of the current local screen, for example, if the logical resolution of the local screen is 1920x1080 and the system scaling is 100%, then capture an image of 1920x1080 pixels; if the system scaling is 150%, the logical resolution is still 1920x1080, but the actual pixels may be 2880x1620, at this time the logical resolution should be captured;

[0084] The target client must send the native logical resolution of the local screen (e.g., client_native_width, client_native_height) and the client DPI scaling factor (e.g., client_dpi_scale_factor, such as: 1.0, 1.25, 1.5) to the target expert and AI recognition module during session establishment or screen sharing, which is the key to accurate coordinate mapping;

[0085] The captured image frames are encoded (such as H.264), but the original pixel size represented internally is clear;

[0086] Therefore, the target client captures multiple screen image frames of the local screen, and the screen image frames include the native logical resolution, the client DPI scaling factor, and image encoding;

[0087] The target client sends the screen image stream formed after image compression and fragmentation of the multiple screen image frames to the target expert;

[0088] The target client sends a copy of the screen image stream to the AI visual recognition module.

[0089] 104, the AI visual recognition module receives and processes the screen image stream to obtain AI visual recognition results, and sends the AI visual recognition results to the target expert;

[0090] The AI visual recognition module receives the screen image stream to obtain the original screen image frame in the screen image stream;

[0091] The AI visual recognition module pre-processes the original screen image frame to obtain an optimized screen image frame; the process of pre-processing the original screen image frame is:

[0092] Resizing: Mainly to adapt to the fixed input size requirements of the preset AI visual recognition model, such as 224x224, 416x416, 640x640, etc.; receive the original screen image frame Iraw (size WrawxHraw); the input size of the preset AI visual recognition model is WmodelxHmodel; use the resize function of the image processing library (such as OpenCV, Pillow) for scaling; output the image frame Iresized (size WmodelxHmodel);

[0093] Color Space Conversion: The preset AI visual recognition model usually has specific requirements for the color channel order and range of the input image (such as RGB, BGR, or grayscale image); check the color channel order expected by the preset AI visual recognition model (such as OpenCV default BGR, TensorFlow / PyTorch usually expect RGB); if the input is RGBA, the Alpha channel needs to be removed; use the color conversion function of the image processing library, such as OpenCV's cvtColor; optionally, pixel value normalization is also needed: scale the pixel value from 0-255 to the floating-point number range of 0-1, i.e. Pixelnormalized=Pixeloriginal / 255.0;

[0094] Denoising: During the screen sharing process, slight noise may occur in the image due to network transmission or capture device, and denoising operation can smooth the image and reduce the interference of noise on recognition accuracy; select appropriate filter according to noise type and degree, common denoising methods include Gaussian blur, median filter or bilateral filter; use the filter function of the image processing library, such as OpenCV's GaussianBlur or medianBlur; output the denoised image frame Idenoised;

[0095] The AI visual recognition module performs UI element recognition on the optimized screen image frame based on the preset AI visual recognition model, and obtains the UI element and bounding box coordinates;

[0096] (1) UI element recognition

[0097] The mainstream model selected by the preset AI visual recognition model includes: Faster R-CNN, YOLO (You Only Look Once, such as YOLOv5, YOLOv7, YOLOv8), SSD (Single Shot MultiBox Detector), and Mask R-CNN (if pixel-level segmentation is required); the basis for selecting the model is that the YOLO series model is particularly suitable for real-time or near-real-time screen content recognition due to its balance between speed and accuracy; Mask R-CNN has an advantage when identifying complex and irregular-shaped UI elements (such as custom graphical buttons);

[0098] The preset AI visual recognition model needs to be trained in advance, and a large number of screenshots with annotations are needed as training data; the annotations should include:

[0099] Image itself: screenshots of various operating systems, applications, and web interfaces;

[0100] Bounding Box: the exact rectangular area [xmin, ymin, xmax, ymax] of each UI element in the image;

[0101] Class Label: the type of UI element corresponding to each bounding box, such as Button, InputField, Dropdown, Checkbox, RadioButton, Icon, MenuItem, Scrollbar, Label, Link, AlertBox, Image, etc.;

[0102] The recognition process of the preset AI visual recognition model is as follows:

[0103] The preprocessed image frame Iprocessed is input into the trained deep learning model; the model extracts image features through convolution layers; the bounding box coordinates of each potential UI element are predicted through the regression layer; the category of the element inside each bounding box is predicted through the classification layer; non-maximum suppression (NMS) is applied to filter overlapping low-confidence bounding boxes and retain the best detection results;

[0104] The bounding box coordinates [xmin', ymin', xmax', ymax'] output by the preset AI visual recognition model are pixel coordinates relative to the input image (i.e., the image after preprocessing and scaling); in order to maintain uniformity in subsequent processing, the coordinates are mapped back to the original logical pixel coordinate system when outputting the results.

[0105] The calculation formula is:

[0106] The size of the input image of the preset AI visual recognition model is WmodelxHmodel; the original logical resolution of the client is WclientxHclient; the bounding box coordinates of the UI element recognized by the AI visual recognition are [xmin', ymin', xmax', ymax'] (WmodelxHmodel in the original logical pixel coordinate system of the client); the reverse scaling factor: Sx=Wclient / Wmodel, Sy=Hclient / Hmodel; the converted UI element coordinates: xmin=xmin'xSx, ymin=ymin'xSy, xmax=xmax'xSx, ymax=ymax'xSy;

[0107] For each recognized UI element, the AI visual recognition module will output a structured data list, usually in JSON format, and each UI element entry contains:

[0108] type: string type, indicating the category of the recognized UI element, such as "Button", "InputField", "Dropdown", "Icon", etc.; this is a predefined enumeration value;

[0109] boundingBox: [x_min, y_min, x_max, y_max], an array containing four floating-point numbers or integers, indicating the top-left corner coordinates (xmin, ymin) and bottom-right corner coordinates (xmax, ymax) of the element in the original logical pixel coordinate system of the client;

[0110] confidence: float, indicating the confidence of the model for the recognition result, usually ranging from 0 to 1;

[0111] elementId: string, a unique identifier, used to track a specific UI element in subsequent processing (such as context understanding, label association);

[0112] additionalProperties: object, optional, may contain other useful information according to the element type; for example, for "text box", it may contain the internal text content (if extracted by OCR); for "icon", it may contain its semantic description;

[0113] (II) Text extraction

[0114] Optical Character Recognition (OCR) is used to extract readable text information from screen images, complementing UI element recognition, and providing richer contextual information.

[0115] Text Region Detection: Obtain rectangular regions that may contain text through specialized text detection models (such as EAST, CRAFT) or from the results of UI element recognition;

[0116] Image Cropping and Preprocessing: Crop each detected text region to form independent text image blocks, and further preprocessing may be needed to improve OCR accuracy, including:

[0117] Grayscale: Convert color images to grayscale;

[0118] Binarization: Convert grayscale images to black and white images to separate foreground text and background;

[0119] Deskew: If the text has slight skew, correct it to make it horizontal;

[0120] Character Enhancement: Adjust contrast, brightness, and sharpen the image to improve character clarity;

[0121] Character Recognition: Send the preprocessed text image block to the OCR engine for character recognition; mainstream OCR engines include Tesseract, PaddleOCR, Baidu OCR, Google Cloud Vision API, etc.

[0122] Text Post-processing: Post-process the OCR results, including:

[0123] Error Correction: Correct recognition errors based on dictionaries or language models;

[0124] Formatting: Remove extra spaces, line breaks, and organize the recognized text into readable strings;

[0125] For each recognized text region, the OCR module will output:

[0126] textContent: string, the extracted text string;

[0127] textBoundingBox: [x_min, y_min, x_max, y_max], an array containing four floating-point numbers or integers, representing the location of the text region in the client's original logical pixel coordinate system;

[0128] confidence: float, the confidence of the OCR engine in text recognition, usually a floating-point number between 0 and 1;

[0129] lineLevelInfo / wordLevelInfo: More detailed OCR results can include bounding boxes and confidence scores for each line or each word;

[0130] Note that to go beyond individual UI element recognition, understanding the logical relationships between UI elements on the screen and the user's current potential intent or application state requires more complex AI models and state management, thus increasing context understanding, the specific process is:

[0131] Based on the bounding box information of UI element recognition, use graph neural networks (GNNs) or heuristic algorithms (such as X-YCut, Voronoi diagram, etc.) to analyze the spatial position, relative distance, size, alignment, color, etc. Visual attributes of elements; organize discrete UI elements into logical groups (e.g. a form, a dialog box, a navigation bar), identify parent-child relationships and sibling relationships, for example, determine which "submit" button belongs to which form;

[0132] Semantic Association: Combine the type of UI element, the text content extracted by OCR, and the pre-defined knowledge graph or semantic dictionary to infer the semantic meaning of the element, for example, a text box labeled "username" and a "password" text box usually belong to a login form; A large number of application interface screenshots and their corresponding semantic labels and logical relationship annotations are required;

[0133] State Inference: By tracking the changes in the sequence of screen image frames, combining the state of UI elements (e.g. whether a button is clicked, whether a text box is filled, whether a certain popup appears), inferring the current application state or operation process phase of the user, recurrent neural networks (RNNs) or Transformer models can be used to process time series data to identify "login page", "third step of settings wizard", "error prompt state", etc.

[0134] History frame storage: Maintain a certain number of historical screen image frames and corresponding AI recognition results;

[0135] Operation sequence analysis: Analyze the changes of UI elements (appearance, disappearance, position movement, text content change), and possible user inputs (through OCR or virtual keyboard input simulation) to infer the user's operation trajectory;

[0136] State machine / flowchart mapping: Predefine common application operation flows (state machines), map the recognized UI states and user operation sequences into these predefined states and transitions, e.g., recognize "login button is clicked" and "username / password input boxes are not empty", infer that the user is trying to log in;

[0137] Knowledge graph / rules engine: Build a knowledge graph about common application interface patterns and operation flows, e.g., "clicking a button with 'next' label usually leads to the next stage of a wizard";

[0138] Recommendation system: Based on the current screen's context understanding, recommend the most likely operation steps, common annotation types, or related knowledge base articles to the expert, e.g., if recognizing "file download failed" error information, the system can recommend the expert to highlight the "retry" button or suggest "check network connection";

[0139] The output of context understanding is more abstract and high-level, which can be a description of the overall state of the screen or a revelation of the relationship between specific elements.

[0140] pageType: string type, identifies the common page type that the current screen belongs to, such as "LoginPage", "SettingsPage", "ErrorDialog", "FileExplorer", etc.;

[0141] mainTask: string type, the system infers the user's current main task, such as "UserLogin", "SoftwareInstallation", "NetworkTroubleshooting", etc.;

[0142] relatedElementGroups: array, a list containing multiple objects (or arrays), each object represents a logical UI element group and its relationship;

[0143] Optional, currentState: string type, describes the current application or operation flow state, such as "WaitingForLogin", "InstallationStep3", "NetworkError", etc.;

[0144] Optionally, suggestedActions: array, a list of possible action suggestions provided by the AI based on the context, e.g., "CheckNetworkSettings", "ClickNextButton", "ReinstallApplication", etc.

[0145] After obtaining the UI element information, text information and context information in the above description, the AI visual recognition module generates an AI visual recognition result and sends it to the target expert end.

[0146] 105. The target expert end generates a background layer based on the screen image stream and the AI visual recognition result, and constructs an interactive whiteboard overlay layer on the background layer.

[0147] The target expert end obtains the rendering area size (e.g., expert_display_width, expert_display_height) of the display area for displaying the client screen sharing picture, which is usually the display area for displaying the video stream in the expert end application program <video>Actual pixel size of the element, QGraphicsView or other rendering component; DPI scale of the expert's own screen, as expert_dpi_scale_factor, which will affect the rendering of the UI elements of the expert's application itself;

[0148] The client's original logical resolution is Wclient x Hclient;

[0149] The rendering area size of the display area of the expert's display of the client's screen sharing picture is Wexpert_render x Hexpert_render;

[0150] The bounding box of each UI element in the AI visual recognition result is [xmin_client, ymin_client, xmax_client, ymax_client];

[0151] Calculate the expert's scale: ScaleX = Wexpert_render / Wclient, ScaleY = Hexpert_render / Hclient, usually to maintain the aspect ratio, take Scale = min(ScaleX, ScaleY), and then center or fill on this basis, but the simplest implementation is to scale X and Y independently;

[0152] The pixel coordinates of the UI elements in the converted display area: xmin_expert = xmin_client x ScaleX, ymin_expert = ymin_client x ScaleY, xmax_expert = xmax_client x ScaleX, ymax_expert = ymax_client x ScaleY;

[0153] It should be noted that it also includes processing of black or gray edges (Letterboxing / Pillarboxing): if ScaleX and ScaleY are not consistent, in order to maintain the aspect ratio of the client's screen, the video player usually adds black edges at the edge of the picture, when calculating the coordinates, the effect of these black edges on the coordinate offset needs to be considered, if the video player is in "full" mode, the scale distortion caused by picture stretching needs to be handled, the best practice is to maintain the aspect ratio and calculate the offset of the actual display area;

[0154] For example, if Scale = min(ScaleX, ScaleY), the actual display width is Wclient x Scale, and the height is Hclient x Scale;

[0155] Offsetx = (Wexpert_render - Wclient x Scale) / 2;

[0156] Offsety = (Hexpert_render - Hclient x Scale) / 2;

[0157] xfinal = xmin_expert + Offsetx; yfinal = ymin_expert + Offsety; etc.

[0158] Visual overlay hint rendering: the expert-side application uses these computed xfinal, yfinal coordinates to draw highlight boxes, text labels, etc. on the overlay layer.

[0159] Drawing API: depends on the expert-side's tech stack (e.g. Web app uses HTML5 Canvas 2D Context's strokeRect, fillText; desktop app uses Qt's QPainter or WPF's DrawingContext;

[0160] Style control: border color, thickness, transparency, text font, color, etc. should be configurable, and can be dynamically adjusted according to the AI's confidence (e.g. lower-confidence boxes have thinner lines).

[0161] Animation effect: to attract the expert's attention, a slight animation effect (like flashing, fading in) can be added to the corresponding overlay hint when new AI recognition result arrives.

[0162] Interactive whiteboard overlay layer: on top of the expert-side display of the client screen share video stream, efficiently render a transparent canvas (like HTML5 Canvas, Qt Graphics View, WPF DrawingContext) that is used to draw all AI recognition hints and expert manual annotations, without affecting the display of the underlying shared content.

[0163] Annotation toolset provides pens, highlighters, arrows, geometric shapes (rectangles, circles), text boxes, number labels, local zoom, etc. multiple annotation tools, supports customizing color, thickness, transparency, etc. properties.

[0164] 106. When the target expert makes annotations on the interactive whiteboard overlay layer, the target expert-side converts the annotation actions into annotation instruction data and sends them to the target client.

[0165] When the expert makes annotations on the interactive whiteboard overlay layer at the target expert end, the coordinates generated by the mouse operation are in the pixel coordinate system at the expert end. The ScaleX, ScaleY, and possibly Offset_x and Offset_y calculated in step 105 are used for reverse calculation;

[0166] xop_client = (xop_expert - Offsetx) / ScaleX;

[0167] yop_client = (yop_expert - Offsety) / ScaleY;

[0168] Annotation data transmission: The converted client original logical pixel coordinates are packaged into annotation instruction data and sent to the target client. This way, regardless of the actual display of the target client's screen, the original logical coordinates can be used for accurate alignment when rendering locally.

[0169] It should be noted that when handling annotation conflicts in multi-person collaboration (if supported), the consistency of views of all parties can be ensured by using optimistic locking or CRDT (Conflict-free Replicated Data Types).

[0170] 107. The target client receives and parses the annotation instruction data and renders the corresponding annotation content on the whiteboard overlay layer on the local screen.

[0171] The target client receives the annotation instruction data, parses it, and accurately renders the corresponding annotation graphics and text on the whiteboard overlay layer on the local screen. The client can visually see the annotation image and text, thereby knowing how to perform subsequent operations, thereby realizing the expert's remote assistance to the client.

[0172] It should be noted that in order to guide the client to operate more accurately, the expert can also provide operation guidance to the client through voice or text chat combined with real-time annotations on the screen, and the client can perform actual operations on the client according to the voice or text chat guidance.

[0173] The client's operation causes changes in the content of the client's screen, which are transmitted back to the expert end and the AI visual recognition module in real time through the screen image stream.

[0174] The expert observes the screen changes after the client's operation and judges whether the operation is correct. The AI visual recognition module also continuously analyzes the new screen content and updates the AI recognition result and auxiliary prompt at the expert end.

[0175] If the expert finds that the problem is not solved or needs further operation, the expert needs to clear the old annotation (or partially retain), make a new annotation, and form a closed loop of guidance-operation-feedback;

[0176] When the problem is solved or either party wants to end, an instruction to end the session is initiated;

[0177] The secure transmission channel is closed, the screen sharing is stopped, and the client and the expert end clear the temporary data related to the session, the whiteboard overlay layer, etc.;

[0178] The client, the expert end and the central server can record and archive the key information of this session (such as session duration, expert, customer, AI-identified UI elements, main annotation screenshots, customer satisfaction feedback, etc.) according to the configuration and user consent, for service quality analysis, knowledge base construction or audit.

[0179] The beneficial effects achieved by the embodiments of the present application are:

[0180] When a target client needs remote assistance, the target client sends a remote assistance request to the corresponding target expert end through the central server; the target expert end establishes a remote assistance session with the target client according to the remote assistance request; the target client captures a screen image frame of the local screen and sends it to the target expert end and the AI visual recognition module in the form of a screen image stream; the AI visual recognition module receives and processes the screen image stream to obtain an AI visual recognition result, and sends the AI visual recognition result to the target expert end; the target expert end generates a background layer according to the screen image stream and the AI visual recognition result, and constructs an interactive whiteboard overlay layer on the background layer; when the target expert makes annotations on the interactive whiteboard overlay layer, the target expert end converts the annotation actions into annotation instruction data and sends them to the target client; the target client receives and parses the annotation instruction data and renders the corresponding annotation content on the whiteboard overlay layer above the local screen. By using AI visual recognition to identify the screen image, the expert can quickly understand the content of the local screen of the client, and the target expert end generates a background layer according to the screen image stream and the AI visual recognition result, and constructs an interactive whiteboard overlay layer on the background layer, which can display the content of the expert's remote assistance in real time and visually.

[0181] In combination with the remote assistance method based on AI visual recognition and interactive annotation described in the above embodiments, the remote assistance system based on AI visual recognition and interactive annotation is described by embodiments as follows.

[0182] As shown in Figure 2 , the embodiment of the present application provides a remote assistance system based on AI visual recognition and interactive annotation, which comprises:

[0183] A plurality of clients 201, a plurality of expert terminals 202, a central server 203 and an AI visual recognition module 204;

[0184] When a client using a target client 201 needs remote assistance, the target client 201 is configured to send a remote assistance request to a corresponding target expert terminal 202 through the central server 203;

[0185] The target expert terminal 202 is configured to establish a remote assistance session with the target client 201 according to the remote assistance request;

[0186] The target client 201 is further configured to capture screen image frames of a local screen and send the screen image frames to the target expert terminal 202 and the AI visual recognition module 204 in the form of a screen image stream;

[0187] The AI visual recognition module 204 is configured to receive and process the screen image stream to obtain AI visual recognition results and send the AI visual recognition results to the target expert terminal 202;

[0188] The target expert terminal 202 is further configured to generate a background layer according to the screen image stream and the AI visual recognition results and construct an interactive whiteboard overlay layer on the background layer;

[0189] When the target expert makes annotations on the interactive whiteboard overlay layer, the target expert terminal 202 is further configured to convert the annotation actions into annotation instruction data and send the annotation instruction data to the target client 201;

[0190] The target client 201 is further configured to receive and parse the annotation instruction data and render corresponding annotation content on a whiteboard overlay layer above the local screen.

[0191] The embodiment of the present application has the following beneficial effects:

[0192] The AI visual recognition of the screen image can help the expert quickly understand the content of the local screen of the client, and the target expert terminal generates a background layer according to the screen image stream and the AI visual recognition results and constructs an interactive whiteboard overlay layer on the background layer, which can display the content of the remote assistance of the expert in real time and visually.

[0193] Those skilled in the art should understand that the embodiments of the present application can be provided as a method, a system, or a computer program product. Therefore, the present application can take the form of a complete hardware embodiment, a complete software embodiment, or an embodiment combining software and hardware aspects. Moreover, the present application can take the form of a computer program product implemented on one or more computer usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer usable program code.

[0194] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flows and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart or flows and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart or flows and / or block diagram block or blocks.

[0195] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the flowchart or flows and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart or flows and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart or flows and / or block diagram block or blocks.

[0196] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer-implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart or flows and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart or flows and / or block diagram block or blocks. Figure 1 one or more functions specified in the flowchart or flows and / or block diagram block or blocks.

[0197] The above merely provides an embodiment of the present application, but is not intended to limit the present application, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall fall within the scope of the claims of the present application.< / video>

Claims

1. A remote assistance method based on AI visual recognition and interactive annotation, characterized in that, A remote assistance system based on AI visual recognition and interactive annotation is applied, the remote assistance system including multiple clients, multiple expert clients, a central server and an AI visual recognition module, the remote assistance method including: When a client using the target client needs remote assistance, the target client will send the remote assistance request to the corresponding target expert through the central server; The target expert terminal establishes a remote assistance session with the target client based on the remote assistance request; The target client captures screen image frames of the local screen and sends them to the target expert terminal and the AI ​​visual recognition module respectively in the form of screen image streams; The AI ​​visual recognition module receives and processes the screen image stream to obtain the AI ​​visual recognition result, and sends the AI ​​visual recognition result to the target expert terminal; The target expert terminal generates a background layer based on the screen image stream and the AI ​​visual recognition result, and constructs an interactive whiteboard overlay layer on the background layer; When the target expert makes annotations on the interactive whiteboard overlay, the target expert's terminal converts the annotation action into annotation instruction data and sends it to the target client; The target client receives and parses the annotation instruction data, and renders the corresponding annotation content on the whiteboard overlay layer above the local screen.

2. The remote assistance method based on AI visual recognition and interactive annotation according to claim 1, characterized in that, The target client sends the remote assistance request to the corresponding target expert through the central server, including: The target client receives a remote assistance instruction initiated by the client and generates a remote assistance request based on the remote assistance instruction; the remote assistance request includes a client identifier and a client identity identifier; The target client sends the remote assistance request to the central processing unit; The central processing unit determines the remote assistance task level based on the customer identity identifier in the remote assistance request; and queries a preset task level-expert level relationship table based on the remote assistance task level to determine the expert identity identifier and expert terminal identifier corresponding to the remote assistance task level. The central processing unit sends the remote assistance request to the target expert terminal corresponding to the expert terminal identifier.

3. The remote assistance method based on AI visual recognition and interactive annotation according to claim 2, characterized in that, The target expert terminal establishes a remote assistance session with the target client based on the remote assistance request, including: The target expert terminal generates a remote assistance acceptance notification based on the remote assistance request; Once remote assistance is accepted, the target expert establishes a secure transmission channel with the target client through the central server; The target expert terminal sends a screen sharing request to the target client through the secure transmission channel; The target client generates a screen sharing authorization notification based on the screen sharing request; Once screen sharing is authorized, a remote assistance session is successfully established between the target client and the target expert.

4. The remote assistance method based on AI visual recognition and interactive annotation according to claim 3, characterized in that, The method further includes: The target client receives a screen sharing authorization instruction corresponding to the screen sharing authorization notification and obtains the screen sharing area range; the screen sharing area range is one or more screen areas within the target client's local screen.

5. The remote assistance method based on AI visual recognition and interactive annotation according to claim 3, characterized in that, The AI ​​visual recognition module is deployed on the target client, the target expert client, or the central server. The target client captures screen image frames from its local screen and sends them as screen image streams to both the target expert client and the AI ​​visual recognition module, including: The target client captures multiple screen image frames of the local screen, and the screen image frames include the client's original logical resolution, the client's DPI scaling ratio, and image encoding; The target client compresses and segments multiple screen image frames to form a screen image stream, which is then sent to the target expert terminal. The target client copies the screen image stream and sends it to the AI ​​visual recognition module.

6. The remote assistance method based on AI visual recognition and interactive annotation according to claim 5, characterized in that, The AI ​​visual recognition module receives and processes the screen image stream to obtain an AI visual recognition result, and sends the AI ​​visual recognition result to the target expert terminal, including: The AI ​​visual recognition module receives the screen image stream and obtains the original screen image frames in the screen image stream; The AI ​​visual recognition module preprocesses the original screen image frame to obtain an optimized screen image frame; The AI ​​visual recognition module performs UI element recognition and text extraction on the optimized screen image frame based on a preset AI visual recognition model to obtain UI element information and text information. The AI ​​visual recognition module generates AI visual recognition results based on the UI element information and the text information, and sends them to the target expert terminal.

7. The remote assistance method based on AI visual recognition and interactive annotation according to claim 6, characterized in that, The AI ​​visual recognition module performs UI element recognition and text extraction on the optimized screen image frame based on a preset AI visual recognition model, obtaining UI element information and text information, including: The AI ​​visual recognition module performs UI element recognition on the optimized screen image frame based on a preset AI visual recognition model to obtain the UI element and bounding box coordinates. The AI ​​visual recognition module parses the original screen image frame to obtain the customer's original logical resolution and customer's DPI scaling ratio, and determines the customer's original logical pixel coordinate system. The AI ​​visual recognition module reverse maps the bounding box coordinates to the customer's original logical pixel coordinate system to obtain the UI element coordinates, and forms UI element information based on the UI element and the UI element coordinates. The AI ​​visual recognition module extracts the text string and text region coordinates of the text region based on the preset file detection model or the UI element information. The AI ​​visual recognition module reverse maps the text region coordinates to the customer's original logical pixel coordinate system to obtain text coordinates, and forms text information based on the text string and the text coordinates.

8. The remote assistance method based on AI visual recognition and interactive annotation according to claim 7, characterized in that, The target expert terminal generates a background layer based on the screen image stream and the AI ​​visual recognition result, and constructs an interactive whiteboard overlay layer on the background layer, including: The target expert terminal obtains the rendering area size of the display area of ​​the shared screen of the client; The target expert terminal calculates the expert terminal scaling ratio based on the rendering area size and the client's original logical resolution; The target expert terminal processes the UI element coordinates and text coordinates in the AI ​​visual recognition result according to the scaling ratio of the expert terminal to obtain the UI element pixel coordinates and text pixel coordinates corresponding to the display area; The target expert terminal displays the screen image stream in the display area to form a background layer based on the rendering area size, the expert terminal scaling ratio, the UI element pixel coordinates, and the text pixel coordinates; The target expert client constructs a transparent interactive whiteboard overlay layer on the background layer.

9. The remote assistance method based on AI visual recognition and interactive annotation according to claim 8, characterized in that, After the target expert client constructs a transparent interactive whiteboard overlay layer on the background layer, it also includes: The target expert terminal draws special markers in the interactive whiteboard overlay based on the pixel coordinates of the UI elements and the pixel coordinates of the text to indicate the borders of the UI elements and the text area.

10. A remote assistance system based on AI visual recognition and interactive annotation, characterized in that, include: Multiple clients, multiple expert clients, a central server, and an AI visual recognition module; When a client using the target client needs remote assistance, the target client is used to send the remote assistance request to the corresponding target expert terminal through the central server; The target expert terminal is used to establish a remote assistance session with the target client based on the remote assistance request. The target client is also used to capture screen image frames of the local screen and send them to the target expert terminal and the AI ​​visual recognition module respectively in the form of screen image streams; The AI ​​visual recognition module is used to receive and process the screen image stream to obtain AI visual recognition results, and send the AI ​​visual recognition results to the target expert terminal; The target expert terminal is also used to generate a background layer based on the screen image stream and the AI ​​visual recognition result, and to build an interactive whiteboard overlay layer on the background layer; When the target expert makes annotations on the interactive whiteboard overlay, the target expert terminal is also used to convert the annotation action into annotation instruction data and send it to the target client; The target client is also used to receive and parse the annotation instruction data, and render the corresponding annotation content on the whiteboard overlay above the local screen.

Images