Picture artificial intelligence interaction method based on mouse key and gesture track

Abstract

The application belongs to the technical field of computer human-computer interaction and image processing, and discloses a picture artificial intelligence interaction method based on mouse keys and gesture tracks, an electronic device, a system and a computer readable storage medium. The client software takes the pressing event of the mouse intelligent key as the time zero point of the unified time reference, synchronously collects the microphone audio stream and the mouse cursor position sequence based on the unified time reference, and takes the release event as the submission boundary; the original picture pixel data and the display transformation parameter are obtained, the screen coordinate track is inversely transformed into the track point sequence in the original picture coordinate system, and the action area is determined; the instruction text is obtained by identifying the audio stream, the original pixel data, the action area and the instruction text are packaged into a structured data packet and sent to an artificial intelligence processing module, and output is performed according to the return result type.

Description

Technical Field

[0001] This invention relates to the field of computer human-computer interaction and image processing technology, specifically to an image artificial intelligence interaction method, electronic device, system, and computer-readable storage medium based on mouse button and gesture trajectory. Background Technology

[0002] With the development of artificial intelligence technology, image processing capabilities based on large models are becoming increasingly mature, and users' demand for intelligent operations such as image editing, recognition, and question answering is growing rapidly. In typical application scenarios, users usually have an original image open or displayed on the screen of an electronic device through an image host application (such as image editing software, image viewer, document editing software, spreadsheet editing software, presentation editing software, browser, or instant messaging software), and want to perform intelligent processing such as deletion, replacement, modification, recognition, or question answering on a specific local area of ​​the original image. To achieve the above processing, users need to provide the following three types of input to the artificial intelligence processing module simultaneously: the original image itself, the local area that the user wants to process, and the user's processing intention.

[0003] In existing technologies, users provide the aforementioned three types of input to the artificial intelligence processing module primarily through the following methods: First, a step-by-step graphical user interface interaction method, where the user first draws a selection area on the image using a mouse or stylus, then enters a text description in a text box, and finally clicks the confirmation button to submit. Second, a voice assistant-based interaction method, where the user first wakes up the voice assistant and issues a voice command, then the voice assistant prompts the user to specify the area of ​​effect, or vice versa. Third, a screenshot-based interaction method, where the user first captures a visible area of ​​the image on the screen, then uploads the captured image and text description to the artificial intelligence processing module.

[0004] Research has revealed that the aforementioned existing technologies have at least the following problems when used for artificial intelligence interaction with raw images opened or displayed by image host applications.

[0005] First, the matching between voice intent and the local area of ​​effect in the image is inaccurate. In step-by-step graphical user interface interaction, drawing a selection area and inputting text description are two independent operation stages, lacking a strict temporal correlation. When a user wants to express different processing intentions for multiple different local areas on the same image, they need to repeatedly switch between area drawing and text input, making the operation process cumbersome and prone to mismatch between area and intent. Voice assistant-based interaction methods also have similar problems. There is no unified time reference between the time interval of the voice command and the specified time interval of the area of ​​effect, making it difficult for the artificial intelligence processing module to accurately establish the correspondence between local areas and voice intent.

[0006] Secondly, the scaling of the host application's display means that screen coordinates cannot be directly used for processing the original image. When the original image is opened or displayed by the image host application, it often undergoes display transformations such as translation, scaling, canvas offset, and device pixel ratio adaptation. The image presented on the screen is not consistent with the original image in the coordinate system. When a user draws a selection or circles an area on the screen, the resulting trajectory is in the screen coordinate system, which is difficult to directly use as the area of ​​application in the original image's coordinate system for artificial intelligence processing. Screenshot tools typically use visible pixels on the screen as input, making it difficult to preserve the complete pixel data and original coordinate relationships of the original image. The resolution, color, and detail of the processed result are limited by the display state, and when the original image has an invisible area outside the screen, that area cannot be captured by the screenshot tool.

[0007] Third, the submission boundaries of multimodal inputs are inconsistent. Existing multimodal input interactions typically require users to perform start and stop operations separately for each modality. For example, clicking the record button to start recording, clicking the stop button to stop recording, clicking the draw button to start drawing a selection, and clicking the confirm button to submit after drawing, or waking up the voice assistant, drawing a selection, and confirming the submission again. The boundaries of each modal input are controlled by independent user interface operations, and the overall submission process lacks a unified trigger and termination mechanism. This is not only complex and error-prone, but also results in significant deviations in the start and end times of different modal inputs, making it difficult to achieve temporal alignment between voice commands and gesture trajectories based on a common time reference.

[0008] Fourth, existing interaction methods, when dealing with mouse input devices, typically treat mouse buttons as ordinary clicks, selection drawing, or quick input triggers. Although mouse buttons can generate press events, continuous press processes, and release events, existing image AI interaction methods still struggle to uniformly organize and submit the same press cycle along with voice acquisition, gesture trajectory acquisition, raw pixel data acquisition, and AI processing requests. Therefore, it is difficult to complete the acquisition, association, and submission of multimodal inputs in a single continuous operation.

[0009] Therefore, it is necessary to provide an image AI interaction method and corresponding electronic devices, systems and storage media to improve problems such as inaccurate matching between voice intent and local image action areas, difficulty in directly using screen coordinates for original image processing due to host application scaling, and inconsistent boundaries of multimodal input submission. Summary of the Invention

[0010] To address the problems existing in the aforementioned background technology, the present invention aims to provide an image AI interaction method, electronic device, system, and computer-readable storage medium based on mouse button and gesture trajectories, in order to improve the following problems existing in the current image AI interaction methods: inaccurate matching between voice intent and local image action areas due to the lack of a unified time reference; display transformation processing such as scaling of the original image by the image host application makes it difficult to directly use the trajectory drawn by the user on the screen for AI processing in the original image coordinate system; and the acquisition and submission boundaries of multimodal inputs are controlled separately by the independent user interface operations of each modality, resulting in a lack of a unified triggering and termination mechanism for the overall submission process.

[0011] To achieve the above objectives, the technical solution adopted by the present invention is as follows.

[0012] In a first aspect, the present invention provides an image artificial intelligence interaction method based on mouse button and gesture trajectory. The method is applied to an electronic device that is communicatively connected to a mouse, the mouse including at least one smart button and a microphone, the electronic device having client software installed, and the screen of the electronic device displaying an original image opened or displayed by an image host application; characterized in that the client software executes steps S1 to S6 based on the same continuous pressing cycle of the smart button.

[0013] In step S1, in response to the press event of the smart button, the time when the press event occurs is taken as the time zero point of a unified time reference. Based on the unified time reference, the audio stream of the microphone is synchronously acquired, and the position sequence of the mouse cursor on the screen is recorded at a preset sampling rate to form the audio stream with timestamps and the free curve gesture trajectory with timestamps in the screen coordinate system.

[0014] In step S2, in response to the release event of the smart button, the acquisition of the audio stream and the recording of the position sequence are simultaneously terminated to obtain the audio stream and the free curve gesture trajectory, and the release event is used as the unique submission boundary of the continuous pressing cycle.

[0015] In step S3, the original pixel data of the original image is obtained from the image host application, and the display transformation parameters of the original image are obtained from the original image coordinate system to the screen coordinate system; based on the inverse transformation of the display transformation parameters, the free curve gesture trajectory is mapped to a sequence of trajectory points in the original image coordinate system; based on the sequence of trajectory points, the effective area is determined, and coordinate data or an image mask representing the effective area is generated.

[0016] In step S4, speech recognition is performed on the audio stream to obtain the instruction text.

[0017] In step S5, the original pixel data, the coordinate data or the image mask representing the active area, and the instruction text are encapsulated into a single structured data packet, which is sent to the artificial intelligence processing module through a single processing request, and the processing result returned by the artificial intelligence processing module is received; wherein, the coordinate data or the image mask and the instruction text establish a temporal association identifier based on the unified time reference.

[0018] In step S6, the data type of the processing result is identified; when the processing result is image data, a new image file is generated based on the image data and saved in a preset output location; when the processing result is file data in a non-image format, an output file in the corresponding format is generated based on the file data and saved in the preset output location; when the processing result is text data, the text data is displayed on the electronic device in the form of a floating window or dialog box.

[0019] In an optional implementation, the effective area in step S3 is determined as follows: when the distance between the first and last points of the trajectory point sequence in the original image coordinate system is less than a preset closure threshold, the trajectory point sequence is smoothed and resampled, and a closed polygonal region is generated based on the trajectory point sequence; when the distance is greater than or equal to the preset closure threshold, a strip region is generated with the trajectory point sequence as the center line and a preset width, or a closed polygonal region is generated by connecting the first and last points of the trajectory point sequence; the closed polygonal region or the strip region is used to generate the image mask; the preset closure threshold can be taken as 2% to 5% of the diagonal length of the original image.

[0020] In one optional implementation, the client software, based on the same continuous press cycle of the smart button and the unified time reference, divides the free curve gesture trajectory into m discontinuous sub-trajectories using a pause signal, where m is greater than or equal to 2; the pause signal is defined as the duration for which the mouse cursor's movement speed is continuously less than a preset speed threshold and greater than or equal to a preset pause duration threshold, where the preset pause duration threshold can be between 300 milliseconds and 1000 milliseconds; the client software, based on the time boundary points between adjacent sub-trajectories and combined with the start and end times of each sub-trajectories and a preset tolerance time, extracts n audio segments from the audio stream, where the preset tolerance time can be between 100 milliseconds and 1000 milliseconds, and n is greater than or equal to 1; each of the n audio segments establishes a correspondence with at least one of the sub-trajectories based on time interval overlap or the preset tolerance time range; step S3 performs the inverse transformation on each of the m sub-trajectories to obtain m sets of trajectory point sequences in the original image coordinate system, and accordingly determines m The m active regions; step S4 involves performing speech recognition on the n audio segments to obtain n instruction texts; step S5 involves a structured data packet including a temporal correspondence between the m active regions and the n instruction texts established based on the unified time reference. The temporal correspondence includes at least one of the following: one active region corresponds to one instruction text, multiple active regions correspond to the same instruction text, and one active region corresponds to multiple instruction texts. This allows the artificial intelligence processing module to process the m active regions on the original image according to the temporal correspondence in a single processing request.

[0021] In an optional implementation, the method further includes a key duration determination step: after the client software detects the release event of the smart key in step S2, it calculates the press duration of the smart key from the press event to the release event. When the press duration is greater than or equal to a preset duration threshold, the continuous press cycle is determined as a long press operation for image AI interaction processing. The client software continues to process the audio stream and free curve gesture trajectory collected in steps S1 to S2 according to steps S3 to S6. When the press duration is less than the preset duration threshold, the key operation is determined as a short press operation, which is different from the long press operation. The short press operation triggers the screenshot function provided by the operating system of the electronic device. The preset duration threshold can be between 300 milliseconds and 800 milliseconds.

[0022] In one optional implementation, the image host application includes at least one of image editing software, image viewer, document editing software, spreadsheet editing software, presentation editing software, browser, or instant messaging software; the raw pixel data in step S3 is obtained through at least one of the following methods: directly reading the raw pixel data of the original image through the image access interface provided by the image host application; obtaining the file path of the original image currently opened or displayed by the image host application through the operating system interface of the electronic device, and having the client software read the raw pixel data of the original image from the storage medium according to the file path; and, when the image host application is a browser, reading the source link or source data stream of the document object model image node corresponding to the original image through the browser extension interface, and obtaining the raw pixel data of the original image based on the source link or the source data stream.

[0023] In one optional implementation, the user processing intent represented by the instruction text is at least one of the following: deleting image content within the effective area and performing content filling on the effective area based on pixels surrounding the effective area in the original image; adding new image content specified by the instruction text to the effective area; replacing the image content within the effective area with target image content specified by the instruction text; modifying at least one of the color, style, or material within the effective area; performing optical character recognition on the image content within the effective area to obtain recognized text, and translating, summarizing, rewriting, or converting the recognized text into at least one of structured tabular data according to the instruction text; generating at least one of a document file, a table file, or a presentation file based on the image content within the effective area; and performing content-based question-and-answer based on the image content within the effective area and in conjunction with the instruction text to generate response text for the instruction text.

[0024] In one optional implementation, the preset sampling rate in step S1 can be between 60 Hz and 500 Hz; the preset output location in step S6 is the storage directory where the original image is located or the output directory pre-configured by the client software; the filename of the generated new image file or the output file is generated by adding a preset suffix to the filename of the original image; the new image file maintains the same image format as the original image or adopts a preset target image format.

[0025] In a second aspect, the present invention provides an electronic device comprising a processor, a memory, and a computer program stored in the memory and executable on the processor; the electronic device is communicatively connected to a mouse, the mouse comprising at least one smart button and a microphone; the computer program, when executed by the processor, implements the method as described in any of the first aspects above.

[0026] Thirdly, the present invention provides an image artificial intelligence interaction system based on mouse button and gesture trajectory. The system includes an electronic device as described in the second aspect above, a mouse communicatively connected to the electronic device, and an artificial intelligence processing module. The mouse includes at least one smart button and a microphone, and is communicatively connected to the electronic device via at least one of Bluetooth communication, 2.4 GHz wireless communication, or USB wired communication. The computer program on the electronic device constitutes the client software, which is communicatively connected to the artificial intelligence processing module. The client software and the artificial intelligence processing module cooperate to execute the method described in any one of the first aspects above.

[0027] Fourthly, the present invention provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the method described in any of the first aspects above.

[0028] Compared with the prior art, the present invention has at least the following beneficial effects.

[0029] First, within the same continuous pressing cycle of the smart button, this invention uses the moment of the pressing event as the zero point of the unified time reference. Based on the unified time reference maintained by the client software, the audio stream and the free-form gesture trajectory are timestamped, ensuring that the area of ​​effect and the voice command share a common time reference. This technical solution helps the artificial intelligence processing module establish a temporal correspondence between the local area of ​​effect and the voice intent within the structured data packet, thereby improving the problem of inaccurate matching between voice intent and local area of ​​effect in the image in the prior art.

[0030] Second, in step S3, this invention acquires the original pixel data of the original image and maps the free curve gesture trajectory in the screen coordinate system to the trajectory point sequence in the original image coordinate system based on the inverse transformation of the display transformation parameters, thereby determining the effective area in the original image coordinate system. This technical solution enables subsequent artificial intelligence processing to be performed based on the original pixel data and the effective area in the original image coordinate system, without relying on the display transformation states of the original image such as scaling, translation, canvas offset, and device pixel ratio adaptation by the image host application, thereby improving the problem in the prior art where the scaling display of the host application makes it difficult to directly use the screen coordinates for original image processing.

[0031] Third, this invention uses the press event of the smart button as the unified start event for multimodal input and the release event as the unique submission boundary. No additional user interface confirmation is required during the process from the press event to the release event and the subsequent submission. This allows audio stream acquisition and gesture trajectory acquisition to be completed based on the same continuous press cycle, with the release event uniformly triggering subsequent processing and submission. This technical solution provides a unified triggering and termination mechanism for multimodal input acquisition and submission, thereby improving the problem of inconsistent multimodal input submission boundaries in existing technologies.

[0032] Fourth, in the optional embodiment of the present invention, the free-form gesture trajectory is divided into m sub-trajectories by the pause signal and the audio stream is extracted into n audio segments. Based on the unified time reference, a temporal correspondence is established between the m active areas and the n instruction texts. This technical solution allows users to express different processing intentions to multiple different active areas on the same original image within the same continuous pressing cycle. It also supports at least one correspondence mode: one active area corresponds to one instruction text, multiple active areas correspond to the same instruction text, and one active area corresponds to multiple instruction texts. This improves the matching accuracy and interaction efficiency between areas and instructions in multi-area, multi-intention scenarios.

[0033] Fifth, the present invention encapsulates the original pixel data, the active area, and the instruction text into a single structured data packet and sends it to the artificial intelligence processing module in a single processing request. This allows the artificial intelligence processing module to obtain the multimodal input information required to perform the corresponding processing and the temporal correlation identifiers between the modalities in a single processing request, which is beneficial for the unified processing, temporal alignment, and regional processing of multimodal inputs.

[0034] Sixth, in step S6, the present invention outputs the processing result in three ways: image data, non-image format file data, and text data, according to the data type of the processing result. This makes the method of the present invention applicable to various processing scenarios such as image editing, file generation, and content question answering. Attached Figure Description

[0035] To more clearly illustrate the technical solutions in the embodiments of the present invention, the accompanying drawings used in the description of the embodiments are briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope of protection of the present invention.

[0036] Figure 1 A schematic diagram illustrating the composition of an image-based artificial intelligence interaction system based on mouse button and gesture trajectories provided in an embodiment of the present invention;

[0037] Figure 2 A flowchart illustrating the overall process of an image-based AI interaction method based on mouse button presses and gesture trajectories provided in this embodiment of the invention.

[0038] Figure 3 A timing diagram illustrating the synchronous acquisition of audio stream and mouse cursor position sequence based on a unified time reference within the same continuous pressing cycle of the smart button, provided for an embodiment of the present invention;

[0039] Figure 4 This is a schematic diagram illustrating the principle of the display transformation of the original image from the original image coordinate system to the screen coordinate system, and the inverse transformation of the free curve gesture trajectory based on the display transformation parameters back to the original image coordinate system, provided in an embodiment of the present invention.

[0040] Figure 5 This is a schematic diagram illustrating the method for determining the effective area provided in an embodiment of the present invention, wherein, Figure 5 (a) in the diagram is a schematic diagram of the generation of a closed polygon region when the distance between the first and last points of the trajectory point sequence is less than a preset closure threshold. Figure 5 (b) is a schematic diagram of a strip region generated with the trajectory point sequence as the center line when the distance is greater than or equal to the preset closure threshold. Figure 5 (c) is a schematic diagram of a closed polygon region generated by connecting the first and last points of the trajectory point sequence when the distance is greater than or equal to the preset closure threshold.

[0041] Figure 6 This invention provides a timing diagram illustrating how a free-form gesture trajectory is divided into m sub-trajectories based on pause signals, and n audio segments are extracted from the audio stream to establish a timing correspondence between m active regions and n instruction texts.

[0042] Figure 7 A flowchart of the key press duration determination steps provided in an embodiment of the present invention;

[0043] Figure 8 A schematic diagram illustrating various methods for acquiring the original pixel data provided in embodiments of the present invention;

[0044] Figure 9 This is a schematic diagram illustrating the data flow of structured data packet transmission, artificial intelligence processing module processing, and processing result output provided in an embodiment of the present invention.

[0045] Figure 10 This is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present invention. Detailed Implementation

[0046] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the invention. All other embodiments obtained by those skilled in the art based on the embodiments of this invention without inventive effort are within the scope of protection of this invention.

[0047] In the description of this invention, the terms "first," "second," etc., are used for descriptive purposes only and should not be construed as indicating or implying relative importance or implicitly specifying the number of indicated technical features. Unless otherwise expressly specified and limited, the terms "connection," "communication connection," etc., should be interpreted broadly, and can refer to wired connections or wireless connections; they can be direct connections or indirect connections through an intermediate medium.

[0048] like Figure 1 As shown, this embodiment of the invention provides an image artificial intelligence interaction system based on mouse button and gesture trajectory. The system includes an electronic device 100, a mouse 200 communicatively connected to the electronic device 100, and an artificial intelligence processing module 300 communicatively connected to the electronic device 100. The electronic device 100 can be any one of a desktop computer, laptop computer, tablet computer, or all-in-one computer. The electronic device 100 includes a processor 101, a memory 102, a display 103, and a communication module 104; the memory 102 stores a computer program that can run on the processor 101, and the computer program forms client software 110 when running; the screen of the electronic device 100 displays an original image 130 opened or displayed by an image host application 120.

[0049] The mouse 200 includes a mouse body and at least one smart button 201, a microphone 202, a main control chip 203, and a communication unit 204 integrated into the mouse body. The mouse 200 communicates with the electronic device 100 via at least one of Bluetooth, 2.4 GHz wireless communication, or USB wired communication. The artificial intelligence processing module 300 can be deployed on a cloud server, and the client software 110 communicates with the artificial intelligence processing module 300 via a network; alternatively, the artificial intelligence processing module 300 can be deployed locally on the electronic device 100, and the client software 110 communicates with the artificial intelligence processing module 300 via inter-process communication or function calls.

[0050] like Figure 2 As shown, the image AI interaction method based on mouse button and gesture trajectory provided in this embodiment of the invention includes steps S1 to S6. The implementation process of this method is described in detail below through several embodiments.

[0051] Example 1: Basic Process of Single Region Single Instruction

[0052] This embodiment describes the complete interactive process of a user deleting a local area of ​​an original image and filling it based on surrounding pixels in an image editing software.

[0053] Suppose that the user has opened an original image 130 with a resolution of 4000×3000 pixels in the image editing software of the electronic device 100. The original image 130 is currently displayed on the screen with a display scaling ratio of 0.5x, and its upper left corner is located at screen coordinates (200,100). The device pixel ratio of the electronic device 100 is 1.0.

[0054] Step S1: The user presses the smart button 201 of the mouse 200. The smart button 201 sends a press event signal to the client software 110 through the communication unit 204. In response to the press event, the client software 110 records the occurrence time t0 of the press event as the zero point of the unified time reference, and simultaneously activates the first channel and the second channel. The first channel acquires an audio stream through the microphone 202 at a sampling rate of 16000 Hz; the second channel continuously records the position sequence of the mouse cursor on the screen at a preset sampling rate, which in this embodiment is 120 Hz. The client software 110 maintains a unified time reference based on the monotonic clock of the electronic device 100, specifically by calling the high-precision timing interface of the Windows operating system, the monotonic clock interface of macOS, or the monotonic clock interface of Linux to obtain the monotonic clock value. Each position point in the audio stream acquired by the first channel and the position sequence recorded by the second channel carries a timestamp relative to the zero point t0.

[0055] In this embodiment, while pressing and holding the smart button 201, the user draws an approximately closed curve on the screen along the outline of an object (such as a cup) in the original image, and at the same time says "delete this cup" through the microphone 202.

[0056] like Figure 3 As shown, in this step, the audio stream of the first channel and the position sequence of the second channel are timestamped based on a unified time base. Both use t0 as the common time zero point, and each data point has a timestamp relative to t0.

[0057] Step S2: The user releases the smart button 201 of the mouse 200. The client software 110 responds to the release event, records the occurrence time t1 of the release event, and simultaneously terminates the audio acquisition of the first channel and the position sequence recording of the second channel, obtaining a timestamped audio stream and a timestamped free-form curve gesture trajectory. In this step, the release event serves as the unique submission boundary. During the submission process from the press event t0 to the release event t1 and after the release event t1, the user does not need to perform any additional user interface confirmation operations.

[0058] Step S3: The client software 110 obtains the original pixel data and size parameters of the original image 130 from the image editing software. In this embodiment, the size parameter is 4000×3000 pixels. It also obtains the display transformation parameters of the original image 130 mapped from the original image coordinate system to the screen coordinate system. In this embodiment, the display transformation parameters include: display position (200, 100), display scaling ratio 0.5, canvas offset (0, 0), and device pixel ratio 1.0. The display transformation parameters can also be equivalently represented as the following two-dimensional affine transformation matrix: | 0.5 0 200 | | 0 0.5 100 | | 0 0 1 |

[0059] like Figure 4 As shown, the client software 110 performs the following calculations on each screen coordinate point (x_screen, y_screen) in the free curve gesture trajectory based on the inverse transformation of the display transformation parameters, to obtain the corresponding coordinate point (x_image, y_image) in the original image coordinate system: x_image = (x_screen - 200) / 0.5 y_image = (y_screen - 100) / 0.5

[0060] After inverse transformation, the original sequence of trajectory points in the screen coordinate system is mapped to a sequence of trajectory points in the original image coordinate system. The client software 110 calculates the distance between the first and last points of the trajectory point sequence and compares this distance with a preset closure threshold. In this embodiment, the preset closure threshold is 3% of the diagonal length of the original image, approximately 150 pixels. In this embodiment, the first-to-last distance is approximately 80 pixels, which is less than the preset closure threshold, therefore the client software 110... Figure 5 The trajectory point sequence is smoothed and resampled as shown in (a) above. The smoothing can employ cubic Bézier curve fitting, moving average filtering, or other curve smoothing methods; the resampling can be performed using equal-interval sampling. The client software 110 generates a closed polygonal region as the effective region based on the smoothed and resampled trajectory point sequence. The client software 110 further generates a binary image mask corresponding to the effective region based on the closed polygonal region. The size of the image mask is the same as the original image, i.e., 4000×3000 pixels, where the pixel value within the effective region is 1, and the remaining pixel values ​​are 0.

[0061] Step S4: The client software 110 calls the speech recognition engine to recognize the audio stream and obtains the instruction text "Delete this cup". The instruction text and the area of ​​action constitute a multimodal input synchronously acquired within the same pressing cycle based on the unified time reference.

[0062] Step S5: As Figure 9 As shown, the client software 110 encapsulates the original pixel data, the image mask or coordinate data representing the active area, and the instruction text into a structured data packet, and sends it to the artificial intelligence processing module 300 through a single processing request.

[0063] In this embodiment, the client software 110 encapsulates the original pixel data, the image mask or equivalent polygon coordinate data representing the area of ​​effect, and the instruction text "Delete this cup" into a single structured data packet. The structured data packet also includes a temporal association identifier between the area of ​​effect and the instruction text established based on the unified time reference. The client software 110 sends the structured data packet to the artificial intelligence processing module 300 through a single processing request. For example, the single processing request can be a POST request based on the HTTPS protocol, a local inter-process communication request, or a local function call request.

[0064] In one embodiment, the structured data packet includes an image data field, an image size field, an effective area field, an image mask field, an instruction text field, a timestamp field, a time-series association identifier field, and a task type field. Specifically, the image data field records the original pixel data or its encoded data; the image size field records the width, height, and number of channels of the original image; the effective area field records the coordinates of the vertices of a closed polygon, the coordinates of the center line of a strip region, or the width parameter of a strip region; the image mask field records the binary mask or grayscale mask corresponding to the original image size; the instruction text field records the instruction text obtained from speech recognition; the timestamp field records the time information corresponding to the audio stream, trajectory point sequence, or instruction text; the time-series association identifier field records the one-to-one, many-to-one, or one-to-many correspondence between the effective area and the instruction text; and the task type field records processing types such as deletion, replacement, addition, modification, recognition, question answering, or file generation.

[0065] The artificial intelligence processing module 300 parses the structured data packet, identifies the cup image content within the effective area based on the original pixel data and the image mask, and determines the user's intent, based on the instruction text, to delete the image content within the effective area and fill it based on surrounding pixels. The artificial intelligence processing module 300 calls the corresponding image processing model to perform deletion on the effective area and fill the effective area with content based on the pixels surrounding the effective area in the original image, generating processed image data and returning it to the client software 110. The image processing model can be an image completion model, a diffusion model, an image restoration model, or other models capable of generating processing results based on image content, region mask, and text instructions; this invention does not limit this.

[0066] Step S6: The client software 110 receives the processing result returned by the artificial intelligence processing module 300, identifies the data type of the processing result as image data, and generates a new image file based on the image data. In this embodiment, the preset output location is the storage directory where the original image 130 is located. The filename of the new image file is generated by adding a preset suffix to the filename of the original image 130, such as adding the suffix "_ai_edited". The new image file maintains the same image format as the original image 130, such as JPEG format.

[0067] Thus, the user completed the deletion of the cup from the original image through a series of actions: pressing, drawing a circle, speaking, and releasing.

[0068] Example 2: Synchronous Processing Flow of Multiple Regions and Multiple Instructions

[0069] This embodiment describes the interaction process in which a user expresses different processing intentions to multiple different areas of the same original image within the same pressing cycle.

[0070] Suppose a user has opened a family photo in the image editing software of the electronic device 100 and wants to change the clothes of two different people in the photo to blue and red respectively.

[0071] Steps S1 to S2: The user presses the smart button 201, with the press event occurring at time t0. The user first draws a circle in the clothing area of ​​the first character and then says, "Change this to blue." Next, the user moves the mouse cursor to the vicinity of the clothing area of ​​the second character and pauses for approximately 600 milliseconds, during which time neither drawing a circle nor speaking is required. Then, the user draws another circle in the clothing area of ​​the second character and says, "Change this to red." Finally, the user releases the smart button 201, with the release event occurring at time t1. Throughout the process, the first channel continuously acquires the audio stream, and the second channel continuously records the mouse cursor position sequence.

[0072] Step S3: Segmentation: The client software 110 analyzes the instantaneous movement speed between each position point in the free curve gesture trajectory based on the unified time reference. In this embodiment, the preset speed threshold is 30 pixels per second, and the preset pause duration threshold is 500 milliseconds. The client software 110 detects that between t0+1.8 seconds and t0+2.4 seconds, the movement speed of the mouse cursor is continuously lower than 30 pixels per second, and the duration is 600 milliseconds, exceeding the preset pause duration threshold of 500 milliseconds, thus determining this time period as a pause signal. Based on this pause signal, the client software 110 divides the free curve gesture trajectory into m=2 discontinuous sub-trajectories: the first sub-trajectory corresponds to the trajectory of selecting the first person's clothes from t0 to t0+1.8 seconds, and the second sub-trajectory corresponds to the trajectory of selecting the second person's clothes from t0+2.4 seconds to t1.

[0073] Audio Segment Extraction: The client software 110 further extracts n=2 audio segments from the audio stream based on the time boundary between adjacent sub-trajectories, i.e., the midpoint of the pause signal t0+2.1 seconds, and in conjunction with the start and end times of each sub-trajectory and a preset tolerance time. In this embodiment, the preset tolerance time is 300 milliseconds. The first audio segment corresponds to the interval from t0 to t0+2.4 seconds, including the user's spoken "Change this to blue"; the second audio segment corresponds to the interval from t0+1.8 seconds to t1, including the user's spoken "Change this to red". The tolerance time ensures that each audio segment can cover the speech content that may extend near the sub-trajectory time interval, avoiding speech truncation due to non-strict alignment between speech boundaries and trajectory boundaries.

[0074] Due to the introduction of the preset tolerance time, there may be partial overlap between adjacent audio segments. The client software 110 can determine the valid instruction text corresponding to each audio segment by combining the voice activity detection results, the timestamp of the speech recognition text, or the semantic matching results. In this embodiment, the time interval of the first audio segment overlaps with that of the first sub-track and is within the tolerance range, and the time interval of the second audio segment overlaps with that of the second sub-track and is within the tolerance range.

[0075] Coordinate mapping and determination of the effective area: The client software 110 performs the inverse transformation of the display transformation parameters on the m=2 sub-trajectories respectively to obtain two sets of trajectory point sequences in the original image coordinate system, and determines two effective areas respectively in the manner described in Example 1.

[0076] Speech recognition and timing correspondence: The client software 110 performs speech recognition on the n=2 audio segments respectively, and obtains two instruction texts: "Change this to blue" and "Change this to red".

[0077] Structured data packet encapsulation: The client software 110 encapsulates the original pixel data, the two active regions, the two instruction texts, and the temporal correspondence established based on the unified time base into a single structured data packet. The temporal correspondence is as follows: the first active region corresponds to the first instruction text, and the second active region corresponds to the second instruction text. This embodiment uses a one-to-one correspondence mode.

[0078] Artificial intelligence processing and result output: The artificial intelligence processing module 300 receives the structured data packet in a processing request, and performs color modification processing on the two active areas of the original image according to the time sequence correspondence, changing the color of the clothing in the first active area to blue and the color of the clothing in the second active area to red. The client software 110 receives the processing result and generates a new image file in the manner described in Embodiment 1.

[0079] like Figure 6 As shown, in this embodiment, a one-to-one correspondence is formed between m=2 active regions and n=2 instruction texts. In other embodiments of the present invention, the temporal correspondence can also be a many-to-one correspondence or a one-to-many correspondence. For example, when m=2 and n=1, the user can draw two circles and say "Change both of these to blue." In this case, the time intervals of the sub-trajectories all fall within the time interval of the same continuous speech, thus determining a many-to-one correspondence. As another example, when m=1 and n=2, the user can draw a circle and say "Delete this and then add a white background." In this case, there are two segments of speech content separated by silence intervals within the time interval of a single sub-trajectory, which can be further determined as a one-to-many correspondence through semantic analysis after speech recognition.

[0080] Example 3: Key press duration judgment and short press screenshot branch

[0081] This embodiment describes the implementation method of the key press duration determination step.

[0082] Let the preset duration threshold be 500 milliseconds.

[0083] Scenario 1, Long Press Scenario: The user presses the smart button 201 and holds the press for more than 500 milliseconds, during which time they draw a circle and speak, and finally release the smart button 201. After detecting the release event in step S2, the client software 110 calculates the press duration to be 3200 milliseconds, which is greater than or equal to the preset duration threshold of 500 milliseconds. The client software 110 determines this as a long press operation and continues to process the audio stream and free curve gesture trajectory collected in steps S1 to S2 according to steps S3 to S6.

[0084] Scenario 2, Short Press Scenario: The user only briefly presses the smart button 201, for example, pressing for 200 milliseconds and then immediately releasing it, without performing any circling or speaking operations. After detecting the release event in step S2, the client software 110 calculates the press duration as 200 milliseconds, which is less than the preset duration threshold of 500 milliseconds. The client software 110 determines this as a short press operation, discards the audio stream and free curve gesture trajectory already collected in steps S1 to S2, to avoid meaningless processing of sporadic data accidentally collected during the short press; steps S3 to S6 are not executed, and the screenshot function provided by the operating system of the electronic device 100 is activated. Specifically, the client software 110 can activate the screenshot function by calling the operating system screenshot interface, simulating the system screenshot shortcut key, or calling the system screenshot tool. This invention does not limit the specific method of calling the screenshot function.

[0085] like Figure 7 As shown, the button duration judgment step expands the same physical button of the smart button 201 into two different functions: a long press is used to trigger the image artificial intelligence interaction process of the present invention, and a short press is used to trigger the screenshot function, thereby improving the usage efficiency of the smart button 201.

[0086] Example 4: Multiple methods for acquiring raw pixel data

[0087] This embodiment describes three typical methods for obtaining the raw pixel data in step S3.

[0088] Method 1: Direct Reading via Image Access Interface. When the image host application is image editing software, the image editing software can provide image access capabilities externally through plugin interfaces, script interfaces, or extension interfaces. The client software 110 can be registered as a plugin of the image editing software, or work in conjunction with plugins, scripts, or extension modules of the image editing software to read the original pixel data of the currently opened image, active document, current layer, or currently selected image object through the corresponding image access interface.

[0089] Method 2: File Path Reading. When the image host application is an image viewer, document editing software, spreadsheet editing software, or presentation editing software, the client software 110 can obtain the file path of the original image currently opened or displayed by the image host application through the operating system interface of the electronic device 100. For example, the client software 110 can obtain the file path corresponding to the currently active window, currently active document, or currently displayed object through the window management interface, process information interface, file association information interface, accessibility interface, or application document query interface provided by the operating system. The client software 110 reads the original pixel data of the original image from the storage medium according to the file path.

[0090] Method 3: Browser Extension Interface Reading. When the image host application is a browser, the client software 110 works in conjunction with a browser extension. The browser extension can be implemented based on a browser extension framework and reads the source link or source data stream of the document object model image node corresponding to the original image through content scripts. The source link can be the URL pointed to by the src attribute of the image node, and the source data stream can be a Blob object, an ArrayBuffer object, base64 encoded data, or a data URI. The browser extension further initiates a request based on the source link to obtain the image binary stream and decodes it into pixel data, or directly decodes it into pixel data based on the source data stream, and returns the obtained original pixel data of the original image to the client software 110.

[0091] like Figure 8 As shown, all three methods ultimately produce complete original pixel data of the original image in the original image coordinate system. The client software 110 can automatically select one or more of the above methods according to the type of the image host application.

[0092] Example 5: Electronic Equipment and System Architecture

[0093] This embodiment describes the hardware structure of the electronic device and system according to an embodiment of the present invention.

[0094] like Figure 10As shown, the electronic device 100 provided in this embodiment of the invention includes a processor 101, a memory 102, a display 103, and a communication module 104. The processor 101 can be at least one of a general-purpose central processing unit, an application-specific integrated circuit (ASIC), a programmable gate array (FPGA), or other circuit devices capable of executing computer programs. The memory 102 can be at least one of a read-only memory (ROM), a random access memory (RAM), a flash memory, a solid-state drive (SSD), or a hard disk drive (HDD). The memory 102 stores a computer program that can run on the processor 101, and when executed by the processor 101, the computer program implements the method described in this embodiment of the invention. The display 103 is used to display the original image 130 opened or displayed by the image host application 120. The communication module 104 is used to communicate with the mouse 200 and the artificial intelligence processing module 300.

[0095] like Figure 1 In the system shown, the mouse 200 establishes a communication connection with the communication module 104 of the electronic device 100 through its internal communication unit 204. In one embodiment, the mouse 200 and the electronic device 100 communicate via Bluetooth; in another embodiment, a 2.4 GHz proprietary wireless protocol is used; and in yet another embodiment, wired communication is conducted directly via a USB data cable. The communication connection between the client software 110 and the artificial intelligence processing module 300 can be based on HTTPS-based remote network communication or local communication based on local inter-process communication, such as Unix Domain Socket or Windows Named Pipe; this invention does not limit this.

[0096] Computer-readable storage medium embodiments

[0097] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the method described in any embodiment of this invention. The computer-readable storage medium may be at least one of a read-only memory, random access memory, magnetic disk, optical disk, flash memory, solid-state drive, or hard disk drive.

[0098] Several descriptions of embodiments of the present invention

[0099] The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions or improvements made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

[0100] In this embodiment of the invention, the execution order of the steps is not limited to the specific order given in the above description, unless there is a clear logical sequence. For example, the acquisition of the original pixel data and the acquisition of the display transformation parameters in step S3 can be performed in parallel or sequentially; the speech recognition in step S4 can be performed immediately after step S2 or delayed until any time before step S5.

[0101] The specific numerical values ​​given in the embodiments of this invention are merely exemplary values, and those skilled in the art can adjust them within a reasonable range according to the needs of specific application scenarios; for the numerical ranges already defined in the claims, the ranges recorded in the claims shall prevail. For example, the preset sampling rate, the preset closure threshold, the preset speed threshold, the preset pause duration threshold, the preset tolerance time, and the preset duration threshold can all be selected within the ranges disclosed in the claims and the specification.

[0102] The functional division between the client software 110 and the artificial intelligence processing module 300 in this embodiment of the invention is merely exemplary. In other embodiments of the invention, the client software 110 may undertake more local processing tasks, such as performing speech recognition locally, thereby reducing the load on the artificial intelligence processing module 300; conversely, the client software 110 may also only undertake data collection and packaging, delegating more processing tasks to the artificial intelligence processing module 300. The invention does not limit this to any particular approach.

Claims

1. A method for AI-powered image interaction based on mouse button presses and gesture trajectories, the method being applied to an electronic device communicatively connected to a mouse, the mouse including at least one smart button and a microphone, the electronic device having client software installed, and the screen of the electronic device displaying an original image opened or displayed by an image host application; characterized in that, The client software performs the following steps based on the same continuous press cycle of the smart button: S1: In response to the press event of the smart button, the time when the press event occurs is taken as the time zero point of the unified time reference. The audio stream of the microphone is synchronously collected based on the unified time reference, and the position sequence of the mouse cursor on the screen is recorded at a preset sampling rate to form the audio stream with timestamps and the free curve gesture trajectory with timestamps in the screen coordinate system. S2: In response to the release event of the smart button, the acquisition of the audio stream and the recording of the position sequence are simultaneously terminated to obtain the audio stream and the free curve gesture trajectory, and the release event is used as the unique submission boundary of the continuous pressing cycle; S3: Obtain the original pixel data of the original image from the image host application, and obtain the display transformation parameters of the original image mapped from the original image coordinate system to the screen coordinate system; based on the inverse transformation of the display transformation parameters, map the free curve gesture trajectory into a sequence of trajectory points in the original image coordinate system; determine the effective area based on the sequence of trajectory points, and generate coordinate data or an image mask representing the effective area; S4: Perform speech recognition on the audio stream to obtain the instruction text; S5: Encapsulate the original pixel data, the coordinate data or image mask representing the active area, and the instruction text into a single structured data packet, send it to the artificial intelligence processing module through a single processing request, and receive the processing result returned by the artificial intelligence processing module; wherein, the coordinate data or image mask and the instruction text establish a temporal association identifier based on the unified time reference; S6: Identify the data type of the processing result; when the processing result is image data, generate a new image file based on the image data and save it in a preset output location; when the processing result is file data in a non-image format, generate an output file in the corresponding format based on the file data and save it in the preset output location; when the processing result is text data, display the text data on the electronic device in the form of a floating window or dialog box.

2. The method according to claim 1, characterized in that, The effective area in step S3 is determined as follows: when the distance between the first and last points of the trajectory point sequence in the original image coordinate system is less than a preset closure threshold, the trajectory point sequence is smoothed and resampled, and a closed polygonal region is generated based on the trajectory point sequence; when the distance is greater than or equal to the preset closure threshold, a strip region is generated with the trajectory point sequence as the center line and a preset width, or a closed polygonal region is generated by connecting the first and last points of the trajectory point sequence; the closed polygonal region or the strip region is used to generate the image mask; The preset closure threshold is set to 2% to 5% of the diagonal length of the original image.

3. The method according to claim 1, characterized in that, The client software, based on the same continuous press cycle of the smart button and the unified time reference, divides the free curve gesture trajectory into m discontinuous sub-trajectories using a pause signal, where m is greater than or equal to 2; the pause signal is defined as the duration of the mouse cursor's movement speed being continuously less than a preset speed threshold and greater than or equal to a preset pause duration threshold, where the preset pause duration threshold is 300 milliseconds to 1000 milliseconds; the client software, based on the time boundary between adjacent sub-trajectories and combined with the start and end times of each sub-trajectories and a preset tolerance time, extracts n audio segments from the audio stream, where the preset tolerance time is 100 milliseconds to 1000 milliseconds and n is greater than or equal to 1; each of the n audio segments establishes a correspondence with at least one of the sub-trajectories based on time interval overlap or the preset tolerance time range; step S3 performs the inverse transformation on each of the m sub-trajectories to obtain m sets of trajectory point sequences in the original image coordinate system, and accordingly determines m active areas; step S4 performs the inverse transformation on the n Each audio segment undergoes speech recognition to obtain n instruction texts; the structured data packet in step S5 includes a temporal correspondence between the m active regions and the n instruction texts established based on the unified time reference. The temporal correspondence includes at least one of the following: one active region corresponds to one instruction text, multiple active regions correspond to the same instruction text, and one active region corresponds to multiple instruction texts. The artificial intelligence processing module processes the m active regions on the original image according to the temporal correspondence in the processing request.

4. The method according to claim 1, characterized in that, The method further includes a key duration determination step: after the client software detects the release event of the smart key in step S2, it calculates the press duration of the smart key from the press event to the release event; When the press duration is greater than or equal to a preset duration threshold, the continuous press cycle is determined as a long press operation for image AI interaction processing. The client software continues to process the audio stream and free curve gesture trajectory collected in steps S1 to S2 according to steps S3 to S6. When the press duration is less than the preset duration threshold, the button operation is determined as a short press operation, which is different from the long press operation. The short press operation triggers the screenshot function provided by the operating system of the electronic device. The preset duration threshold is 300 milliseconds to 800 milliseconds.

5. The method according to claim 1, characterized in that, The image host application includes at least one of image editing software, image viewer, document editing software, spreadsheet editing software, presentation editing software, browser, or instant messaging software; the raw pixel data in step S3 is obtained through at least one of the following methods: directly reading the raw pixel data of the original image through the image access interface provided by the image host application; obtaining the file path of the original image currently opened or displayed by the image host application through the operating system interface of the electronic device, and having the client software read the raw pixel data of the original image from the storage medium according to the file path; Furthermore, when the image host application is a browser, the source link or source data stream of the document object model image node corresponding to the original image is read through the browser extension interface, and the original pixel data of the original image is obtained based on the source link or the source data stream.

6. The method according to claim 1, characterized in that, The user processing intent represented by the instruction text is at least one of the following: (a) deleting image content within the effective area and performing content filling on the effective area based on the pixels surrounding the effective area in the original image; (b) adding new image content specified by the instruction text to the effective area; (c) replacing the image content within the effective area with target image content specified by the instruction text. (d) Modify at least one of the color, style, or material within the effective area; (e) Perform optical character recognition on the image content within the effective area to obtain recognized text, and translate, summarize, rewrite, or convert the recognized text into at least one of structured tabular data according to the instruction text; (f) Generate at least one of a document file, a table file, or a presentation file based on the image content within the effective area; (g) Perform content-based question-and-answer based on the image content within the effective area and in conjunction with the instruction text to generate response text for the instruction text.

7. The method according to claim 1, characterized in that, The preset sampling rate in step S1 is 60 Hz to 500 Hz; the preset output location in step S6 is the storage directory where the original image is located or the output directory pre-configured by the client software; the filename of the generated new image file or the output file is generated by adding a preset suffix to the filename of the original image; the new image file maintains the same image format as the original image or adopts a preset target image format.

8. An electronic device, characterized in that, The electronic device includes a processor, a memory, and a computer program stored in the memory and executable on the processor; the electronic device is communicatively connected to a mouse, the mouse including at least one smart button and a microphone; when the computer program is executed by the processor, it implements the method as described in any one of claims 1 to 7.

9. An image-based artificial intelligence interactive system based on mouse button presses and gesture trajectories, characterized in that, The system includes an electronic device as described in claim 8, a mouse communicatively connected to the electronic device, and an artificial intelligence processing module; the mouse includes at least one smart button and a microphone, and is communicatively connected to the electronic device via at least one of Bluetooth communication, 2.4 GHz wireless communication, or USB wired communication; the computer program on the electronic device constitutes client software, the client software is communicatively connected to the artificial intelligence processing module, and the client software and the artificial intelligence processing module cooperate to perform the method as described in any one of claims 1 to 7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7.

Images