Partition gesture and voice interaction method and system on dynamic wallpaper

Abstract

The application discloses a partition gesture and voice interaction method and system on a dynamic wallpaper and belongs to the technical field of dynamic wallpaper interaction. The method receives a touch event through a touch event callback interface in a dynamic wallpaper engine, analyzes touch coordinate information and a double-click gesture type, divides a visible display area of a wallpaper display surface into an upper half interaction area and a lower half interaction area in a longitudinal direction, sends an audio acquisition start instruction to an independently running foreground service process when the touch coordinate falls into the upper half interaction area and the gesture is a double click, and sends a voice content query instruction to the foreground service process when the touch coordinate falls into the lower half interaction area and the gesture is a double click. The application realizes double interaction of voice acquisition and voice content acquisition in a dynamic wallpaper scene through gesture partition response and cross-process cooperation, processes an audio resource occupation conflict, and expands the interaction capability of the dynamic wallpaper.

Description

Technical Field

[0001] This invention relates to the field of dynamic wallpaper interaction technology, specifically to a method and system for zoned gesture and voice interaction on dynamic wallpapers. Background Technology

[0002] Live wallpapers are applications that display dynamic images or videos on a device's home screen or lock screen. Users access the wallpaper preview or settings interface by long-pressing or using specific gestures. Interaction with the live wallpaper itself is mostly limited to simple touch-swipe transitions or clicks, resulting in relatively basic functionality. Some live wallpaper applications attempt to add audio input or output functionality to enrich the interaction, such as voice control for wallpaper switching or voiceprint recognition. However, these functions usually require additional auxiliary applications or separate interfaces, resulting in low integration of voice capture and audio content acquisition within the wallpaper itself. Currently, to achieve voice interaction within a live wallpaper, the wallpaper process often directly requests microphone permissions and creates an audio capture component. Live wallpapers typically use video playback as the primary visual presentation, and the video player component occupies audio hardware resources for extended periods. When the wallpaper process simultaneously starts audio capture, conflicts with audio hardware resources can lead to capture failures or abnormal video playback. Another approach is to use a third-party independent recording application to complete voice capture and then send the recording results back to the wallpaper service. This approach requires users to switch between the wallpaper interface and the recording application, resulting in fragmented operations and making it impossible to directly trigger the start and end of capture on the wallpaper surface. Meanwhile, when the wallpaper needs to retrieve voice content from the server and play it during wallpaper display, this is usually achieved through an audio player within the wallpaper process. Playing voice may request system audio focus, causing the currently playing wallpaper video audio to be interrupted or improperly mixed, affecting the smooth audio-visual experience of the wallpaper's dynamic visuals. Furthermore, on devices with large wallpaper display areas, there is a lack of a mechanism to establish a correspondence between gesture operations at different spatial locations and different voice functions. Users cannot intuitively choose to initiate voice capture or retrieve voice content by touching the area, resulting in low efficiency in judging interaction intent. To address these issues, it is necessary to solve the problems of using double-tap gestures on the wallpaper surface to initiate audio capture and voice content query respectively, handling the conflict between wallpaper video playback and audio hardware resource usage when initiating audio capture, and maintaining continuous rendering of the wallpaper video without preempting audio focus while playing the retrieved voice content. Summary of the Invention

[0003] The purpose of this invention is to provide a method and system for zoned gesture and voice interaction on live wallpapers, so as to solve the problem of audio resource conflict between voice acquisition and wallpaper video playback in live wallpaper scenarios, as well as the problem that users cannot directly trigger voice acquisition and voice content query on the wallpaper surface through zoned gestures.

[0004] To achieve the above objectives, the present invention provides the following technical solution: The present invention provides a method for zoned gesture and voice interaction on a live wallpaper, comprising: in the running environment of the live wallpaper engine, receiving touch events acting on the wallpaper display surface through a touch event callback interface, and having a gesture recognition module built into the live wallpaper engine parse touch coordinate information and double-click gesture type from the touch events; dividing the visible display area of ​​the wallpaper display surface into an upper interactive area and a lower interactive area based on the vertical height of the visible display area; when the touch coordinate information falls into the upper interactive area and the gesture type is double-click, the live wallpaper engine sends an audio acquisition start command to a foreground service process that runs independently of the live wallpaper engine; when the touch coordinate information falls into the lower interactive area and the gesture type is double-click, the live wallpaper engine sends a voice content query command to the foreground service process. By dividing the visible display area of ​​the live wallpaper into upper and lower halves and combining it with double-click gesture recognition, users can directly trigger audio acquisition or voice content query on the wallpaper interface without leaving the current wallpaper scene, resulting in a short operation path and timely response.

[0005] As a preferred embodiment of the present invention, during the process of receiving a touch event, the dynamic wallpaper engine obtains the original touch coordinates from the touch event and maps the original touch coordinates to a standardized coordinate system of the visible display area. Within this standardized coordinate system, the ratio of the vertical distance of the touch point relative to the top boundary of the visible display area to the total vertical height of the visible display area is calculated. The interactive area to which the touch point belongs is determined based on the relationship between this ratio and a preset partitioning threshold. This ratio-based partitioning method can adapt to different screen sizes and wallpaper scaling states, ensuring consistency in interactive area division and accuracy in gesture recognition.

[0006] Preferably, when the live wallpaper engine sends an audio capture start command to the foreground service process, it transmits the audio capture start command to the foreground service process through an inter-process communication interface. The foreground service process responds to the audio capture start command by creating an audio capture instance. The audio capture instance is configured with microphone hardware access permissions and audio encoding parameters, and begins capturing audio data frames from the microphone hardware. Thus, the audio capture work is handled by an independent foreground service process. Even if the process containing the live wallpaper engine is recycled or restarted by the system, the continuity of audio capture is guaranteed. Furthermore, the foreground service process has persistent running capabilities and can stably hold microphone hardware resources.

[0007] Furthermore, after the foreground service process creates an audio capture instance in response to the audio capture start command, it returns a confirmation message indicating that audio capture has started to the live wallpaper engine. Upon receiving the confirmation message, the live wallpaper engine pauses the decoding operation of the current wallpaper video's player component and releases the audio hardware resources occupied by the player component, while maintaining the surface ownership of the wallpaper display surface to the live wallpaper engine. During audio capture, the live wallpaper engine's release of audio hardware resources effectively avoids conflicts between the microphone capture loop and the wallpaper video audio output, preventing the introduction of echoes or noise; at the same time, maintaining surface ownership ensures the stability of the wallpaper view hierarchy, preventing the wallpaper interface from disappearing or flickering due to resource release.

[0008] Furthermore, after releasing the audio hardware resources occupied by the player component, the dynamic wallpaper engine continuously renders a preset static placeholder image on the wallpaper display surface and pauses the periodic video resource query task initiated to the server until it receives a touch event transmitted by the touch event callback interface again and the gesture recognition module parses the double-tap gesture type from the touch event. At this point, the dynamic wallpaper engine sends an audio acquisition end command to the foreground service process. In this way, the wallpaper displays a static image and stops network requests during audio acquisition, which reduces system resource consumption, avoids interference with the user experience caused by changes in wallpaper video content, and intuitively presents the "recording in progress" status to the user.

[0009] Preferably, after the live wallpaper engine sends an audio acquisition termination command to the foreground service process, the foreground service process responds to the command by stopping the acquisition operation of the audio collector instance, encapsulating the encoded audio data output by the audio collector instance into an audio file, writing it to a local cache, and uploading the audio file to the server via an independent network request module. After the upload is completed, the foreground service process sends a confirmation message to the live wallpaper engine indicating that acquisition and upload are complete. By having the foreground service process uniformly handle the encapsulation and uploading of the audio file, the live wallpaper engine and network transmission logic are decoupled, improving module independence and upload reliability.

[0010] Based on this, after the foreground service process sends a confirmation message to the live wallpaper engine indicating that the data acquisition and upload are complete, the live wallpaper engine responds to the confirmation message by reconstructing the player component, rebinding the player component to the surface holding object of the wallpaper display surface, and resuming video decoding and rendering operations from the last paused playback position of the player component. Simultaneously, it resumes the periodic video resource query task initiated to the server. Therefore, after the audio acquisition task is completely completed, the live wallpaper can seamlessly resume video playback and resource updates without the user perceiving any interruption to the wallpaper video, ensuring visual continuity after the interaction is complete.

[0011] As a technical solution of the present invention, when the dynamic wallpaper engine sends a voice content query instruction to the foreground service process, it transmits the voice content query instruction to the foreground service process via an inter-process communication interface. The foreground service process responds to the voice content query instruction by calling the network request module to query the server for the latest voice content network address corresponding to the current pet identifier, and returns the latest voice content network address to the dynamic wallpaper engine.

[0012] Furthermore, after the foreground service process returns the latest voice content network address to the live wallpaper engine, the live wallpaper engine passes the latest voice content network address to the audio player instance running in the foreground service process. The audio player instance retrieves the voice audio data from the latest voice content network address and outputs it through the speaker hardware. While the audio player instance is outputting the voice audio data, the live wallpaper engine maintains its own video playback component's continuous decoding and rendering operations on the wallpaper video, and the foreground service process does not request system audio focus. In this way, when playing the retrieved voice content, the wallpaper video continues to render normally and will not interrupt the audio focus of other applications or cause abnormal wallpaper video playback due to voice playback. This achieves parallel presentation of voice broadcasting and live wallpaper visual effects, resulting in a smooth and unified user experience.

[0013] This invention also provides a zoned gesture and voice interaction system for live wallpapers, including a memory, a processor, and a computer program stored in the memory and running on the processor. When the processor executes the computer program, it implements the aforementioned zoned gesture and voice interaction method for live wallpapers. This system can achieve deep integration of zoned gestures, voice acquisition, and voice content retrieval at the live wallpaper entry point, resulting in efficient overall interaction, reasonable resource coordination, and significantly improved user operation convenience and functional usability.

[0014] The technical effects and advantages provided by the present invention in the above technical solution are as follows:

[0015] The visible display area of ​​the live wallpaper is vertically divided into an upper interactive area and a lower interactive area. Audio capture initiation commands and voice content query commands are triggered based on the area touched by the touch coordinates and the type of double-tap gesture. A double-tap gesture in the upper interactive area initiates audio capture, while a double-tap gesture in the lower interactive area initiates voice content query. Users can quickly trigger two different voice interaction functions by touching different areas of the wallpaper surface without leaving the wallpaper display interface or switching to other applications. The mapping relationship of the partitioned gestures is simple and intuitive, reducing the probability of accidental operation during interaction and expanding the interactive dimension of the live wallpaper from a single visual display to a multi-channel interaction that includes both voice input and voice content acquisition. When the touch coordinates fall into the upper interactive area and the gesture is a double-tap, the live wallpaper engine sends an audio capture initiation command to the foreground service process. The foreground service process runs independently of the live wallpaper engine and creates an audio collector instance to capture audio data frames. This architecture, separating the foreground service process from the wallpaper engine, ensures that the audio capture operation is not affected by the playback status of the wallpaper engine itself, avoiding conflicts caused by the wallpaper video player component occupying audio hardware resources when directly initiating audio capture within the wallpaper process. After receiving a confirmation message from the foreground service process that audio capture has started, the dynamic wallpaper engine pauses the decoding operation of the current wallpaper video player component and releases the audio hardware resources occupied by the player component. Simultaneously, it maintains ownership of the wallpaper display surface, continuing to render a preset static placeholder image on the wallpaper display surface. After releasing the audio hardware resources, the foreground service process can independently and conflict-free use the microphone hardware to complete audio capture. The wallpaper display surface does not lose its image due to the pause in video decoding; the static placeholder image maintains the integrity of the interface. The user only perceives the wallpaper switching from a dynamic video to a static image, preserving visual continuity and avoiding the risk of the wallpaper application being reclaimed by the system when the dynamic wallpaper is exited.

[0016] After audio acquisition is complete, the foreground service process encapsulates and uploads the acquired audio data, sending a confirmation message to the live wallpaper engine indicating that acquisition and uploading are complete. The live wallpaper engine then rebuilds its player component and resumes video decoding and rendering operations. During audio acquisition, video decoding is paused and audio hardware resources are released. After acquisition, the player component is rebuilt and playback resumes from the paused position, allowing the live wallpaper to seamlessly continue playback from where it left off after the audio acquisition task is completed, avoiding the abruptness of reloading or starting from the beginning. For voice content queries triggered by double-clicking in the lower interactive area, the live wallpaper engine sends a voice content query command to the foreground service process. The foreground service process queries and returns the latest voice content network address. The audio player instance in the foreground service process then retrieves the voice audio data and outputs it through the speaker. During playback, the live wallpaper engine maintains its own video playback component's continuous decoding and rendering operations on the wallpaper video, and the foreground service process does not request system audio focus. Voice content playback and wallpaper video playback proceed in parallel, with the sound of the wallpaper video not interrupted or covered. The two audio outputs coexist on their respective independent audio tracks, enabling the simultaneous presentation of dynamic images and voice prompts. This avoids playback interruptions or mute issues caused by competing for audio focus, and enhances the composite expressive capabilities of the dynamic wallpaper as an information-carrying interface. Attached Figure Description

[0017] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments recorded in this invention. For those skilled in the art, other drawings can be obtained based on these drawings.

[0018] Figure 1 This is a flowchart of the gesture and voice interaction method for dynamic wallpaper partitions;

[0019] Figure 2 This is a flowchart of the voice query and playback process for the dynamic wallpaper engine;

[0020] Figure 3 This is a standardized vertical axis distribution chart of touch points for dynamic wallpaper zones;

[0021] Figure 4 This is a comparison chart of the raw PCM data from audio acquisition and the data after smoothing.

[0022] Figure 5 It is a curve showing the audio data retrieval and playback volume changes during the audio content playback process. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0024] See Figure 1 This invention provides a method for zoned gesture and voice interaction on a live wallpaper, comprising: in the running environment of a live wallpaper engine, receiving touch events acting on the wallpaper display surface through a touch event callback interface, and having a gesture recognition module built into the live wallpaper engine parse touch coordinate information and double-click gesture type from the touch events; dividing the visible display area of ​​the wallpaper display surface into an upper interactive area and a lower interactive area based on the vertical height of the visible display area; when the touch coordinate information falls into the upper interactive area and the gesture type is double-click, the live wallpaper engine sends an audio acquisition start command to a foreground service process that runs independently of the live wallpaper engine; when the touch coordinate information falls into the lower interactive area and the gesture type is double-click, the live wallpaper engine sends a voice content query command to the foreground service process.

[0025] Example 1:

[0026] In its implementation, the live wallpaper engine receives touch events applied to the wallpaper display surface through a touch event callback interface. The touch events contain raw touch coordinates passed from the system's underlying layer. The live wallpaper engine extracts the horizontal and vertical coordinates of these raw touch coordinates from the touch event's data structure. Simultaneously, the live wallpaper engine queries the boundary information of the visible display area of ​​the wallpaper display surface in the screen coordinate system. This boundary information includes the top and bottom vertical coordinates, the left and right horizontal coordinates, as well as the total vertical height and horizontal width of the visible display area. Based on this boundary information, the live wallpaper engine constructs a standardized coordinate system. This standardized coordinate system uses the top-left corner of the visible display area as its origin, defines the horizontal axis with the total horizontal width of the visible display area as the unit length, and defines the vertical axis with the total vertical height of the visible display area as the unit length.

[0027] In some embodiments, the live wallpaper engine maps the original touch coordinates to a normalized coordinate system to obtain the normalized ordinate of the touch point. The mapping method is as follows: subtract the ordinate of the top boundary of the visible display area from the original touch coordinate's ordinate value to obtain the vertical distance of the touch point relative to the top boundary of the visible display area; divide this vertical distance by the total vertical height of the visible display area to obtain the normalized ordinate of the touch point. The formula for calculating the normalized ordinate is expressed as:

[0028]

[0029] in, The standardized ordinate representing the touch landing point, This represents the vertical coordinate value of the original touch coordinates. This represents the ordinate of the top boundary of the visible display area in the screen coordinate system. This indicates the total vertical height of the visible display area.

[0030] The live wallpaper engine has a preset partitioning threshold, which is a fixed value greater than 0 and less than 1. In some embodiments, the partitioning threshold is set to 0.5. After obtaining the standardized ordinate of the touch point, the live wallpaper engine compares the standardized ordinate with the preset partitioning threshold. When the standardized ordinate is less than the preset partitioning threshold, the live wallpaper engine determines that the interactive area to which the touch point belongs is the upper interactive area; when the standardized ordinate is greater than or equal to the preset partitioning threshold, the live wallpaper engine determines that the interactive area to which the touch point belongs is the lower interactive area. The upper interactive area corresponds to the area in the visible display area extending downwards from the top boundary to half the total vertical height, and the lower interactive area corresponds to the area in the visible display area extending downwards from half the total vertical height to the bottom boundary. Through the above process, the live wallpaper engine completes the division of the interactive area based on vertical height, providing area determination results for subsequent determination of the correspondence between gesture type and interactive area.

[0031] See Figure 3 The graph shows the distribution of touch events on the dynamic wallpaper display surface. The vertical axis represents the standardized y-coordinate of the touch event, and the horizontal axis represents the touch event number. The blue solid lines and dots indicate touch points identified as belonging to the upper interactive area. Their standardized y-coordinates are concentrated between 0 and the preset partition threshold of 0.5, with fluctuations mainly within the range of 0.0 to 0.5, fully verifying the effectiveness of the interactive area division based on standardized y-coordinates and partition thresholds in Example 1. The red solid lines and squares indicate touch points identified as belonging to the lower interactive area. Their standardized y-coordinates are all greater than or equal to 0.5, and their value distribution basically covers the range of 0.5 to 1.0, demonstrating the dynamic wallpaper engine's accurate identification of touch events in the lower interactive area.

[0032] The dashed line in the figure represents the partition threshold of 0.5, clearly dividing the standardized vertical coordinate data of touch events into two distinct partitions. In the continuous distribution of touch event sequence numbers, the blue and red curves show no significant overlap, indicating that the dynamic wallpaper engine can reliably determine the touch point region based on the preset partition threshold, avoiding misclassification. This touch event sequence number corresponds to the continuous touch data stream obtained by the dynamic wallpaper engine through the touch event callback interface in Example 1, reflecting the continuous detection and accurate partitioning of the touch point vertical coordinate by the partition gesture recognition module.

[0033] Example 2:

[0034] In its implementation, the live wallpaper engine generates an audio capture start command when it determines that the touch coordinates fall within the upper interactive area and the gesture recognition module interprets the gesture type as a double-tap. The live wallpaper engine then transmits this audio capture start command to the foreground service process via an inter-process communication (IPC) interface. This IPC interface is implemented based on the Android operating system's binding service mechanism. The live wallpaper engine acts as the client of the binding service, and the foreground service process acts as the server. The communication protocol between the live wallpaper engine and the foreground service process is described using the Android Interface Definition Language (API). The IPC interface defines a set of audio control methods, including a method to start audio capture. The parameters of this method are the configuration information carried by the audio capture start command, which includes the target sampling rate, target number of channels, and target encoding format.

[0035] The foreground service process creates an audio acquisition instance in response to the audio acquisition start command. Before creating the audio acquisition instance, the foreground service process has declared and obtained microphone hardware access permissions through the Android system's permission check mechanism. The audio acquisition instance is instantiated based on the Android system's AudioRecord class. During instantiation, the audio encoding parameters configured for the audio acquisition instance are passed in, including sampling rate parameters, number of channels parameters, and encoding format parameters. In some embodiments, the sampling rate parameter is set to 16000 Hz, the number of channels parameter is set to mono, and the encoding format parameter is set to 16-bit pulse code modulation. The foreground service process calls the start recording method of the AudioRecord instance to cause the audio acquisition instance to start acquiring audio data frames from the microphone hardware. The audio acquisition instance internally maintains a looping read thread, which reads a fixed-length byte array from the AudioRecord instance's buffer at a preset period. Each segment of the byte array constitutes an audio data frame.

[0036] After the audio acquisition instance begins acquiring audio data frames from the microphone hardware, the foreground service process constructs an acknowledgment message confirming that audio acquisition has started. The message body of the acknowledgment message includes a result status field and an audio acquisition session identifier field. The foreground service process sends the acknowledgment message to the dynamic wallpaper engine via a callback channel defined in the inter-process communication interface or by returning a message object.

[0037] After receiving confirmation that audio capture has started, the live wallpaper engine executes the player component control flow. The live wallpaper engine calls the pause method of the wallpaper video player component to pause the video decoding operation. The video decoding operation includes reading compressed video frames from the video source and decoding the compressed video frames. After pausing the decoding operation, the live wallpaper engine releases the audio hardware resources occupied by the player component. This release includes calling the release method of the audio output object within the player component and sending an audio focus release request to the system audio service. After releasing the audio hardware resources occupied by the player component, the live wallpaper engine maintains ownership of the wallpaper display surface; the Surface object corresponding to the wallpaper display surface is not destroyed, and the wallpaper window's hierarchy and attributes remain unchanged.

[0038] See Figure 4 In the graph, the horizontal axis represents time in milliseconds (ms), and the vertical axis represents amplitude in least significant bits (LSB). The legend indicates that the two curves represent "raw PCM data" and "smoothed PCM data," respectively. The curves exhibit periodic fluctuations, with an amplitude range of approximately -15000 to 15000.

[0039] As can be seen from the figure, the blue raw PCM data curve has more high-frequency noise and subtle fluctuations, manifested as sharp undulations and irregular jitters. The orange smooth PCM data curve, on the other hand, is significantly smoother, with gentler peak and trough changes, removing most of the high-frequency interference, but its overall waveform trend and periodic characteristics are highly consistent with the raw PCM data.

[0040] In terms of time interval, the two curves exhibit amplitude variations of approximately six complete cycles within the range of 0 to 30 milliseconds. The waveforms maintain a good correspondence between the peaks and troughs of each cycle, indicating that the smoothing process did not alter the main periodic characteristics and amplitude range of the PCM signal. By filtering out sharp fluctuations in the original data, the smoothing curves effectively reduce local noise in the signal, improving the stability and accuracy of subsequent audio acquisition and processing.

[0041] Example 3:

[0042] In its implementation, after releasing the audio hardware resources occupied by the player component, the dynamic wallpaper engine performs a switching operation on the rendering content of the wallpaper display surface. The dynamic wallpaper engine has pre-set static placeholder image resources, which are bitmap files stored in the engine's resource directory. The pixel size of the bitmap file matches the display resolution of the wallpaper display surface. Responding to the status signal indicating that the audio hardware resources have been released, the dynamic wallpaper engine calls the canvas locking method associated with the surface holding object to obtain the drawing canvas object corresponding to the wallpaper display surface. The dynamic wallpaper engine decodes the static placeholder image resource into a bitmap object and draws the bitmap object onto the wallpaper display surface using the canvas object's bitmap drawing method, completing the rendering of the static placeholder image. The dynamic wallpaper engine unlocks the wallpaper display surface and submits the drawing content, allowing the static placeholder image to remain continuously displayed on the wallpaper display surface. During the continuous display of the static placeholder image, the dynamic wallpaper engine no longer submits new video frame data to the wallpaper display surface, maintaining the static placeholder image as the display content of the wallpaper display surface.

[0043] After rendering the static placeholder image, the live wallpaper engine pauses the periodic video resource query task. This periodic video resource query task is a scheduled task within the live wallpaper engine, and its execution cycle is controlled by a preset query interval parameter. In some embodiments, the query interval parameter is set to 300,000 milliseconds. Each time the periodic video resource query task executes, it sends a Hypertext Transfer Protocol (HTTP) request to the server via the network request module integrated within the live wallpaper engine. This HTTP request carries wallpaper identifier parameters and the current timestamp parameter. The server responds to the HTTP request by returning wallpaper video resource update information. The live wallpaper engine removes the periodic video resource query task from the scheduled task queue by calling the task cancellation method of the scheduled task manager, thus pausing the periodic video resource query task initiated to the server. After pausing the periodic video resource query task, the live wallpaper engine no longer actively requests new wallpaper video resources from the server.

[0044] After pausing periodic video resource query tasks, the live wallpaper engine enters a touch event listening and waiting state. The live wallpaper engine keeps its touch event callback interface active, continuously listening for touch events acting on the wallpaper display surface. When a user's finger performs a touch operation on the wallpaper display surface, the system's underlying layer distributes the touch event to the live wallpaper engine's touch event callback interface. After receiving the touch event, the live wallpaper engine's built-in gesture recognition module parses the gesture type. The gesture recognition module determines the gesture type by analyzing the time interval and position offset between press and release events in the touch event sequence. The gesture recognition module has preset recognition parameters for double-click gestures, including a maximum time interval threshold and a maximum position offset threshold between two clicks. In some embodiments, the maximum time interval threshold is set to 400 milliseconds, and the maximum position offset threshold is set to 50 pixels. When the gesture recognition module detects two press and release operations in the touch event sequence, and the time interval between the two operations is less than the maximum time interval threshold, and the distance between the touch points of the two operations is less than the maximum position offset threshold, the gesture recognition module determines the current gesture type to be a double-click.

[0045] After confirming that the gesture recognition module has identified the double-tap gesture type from the touch event, the live wallpaper engine generates an audio capture termination command. The live wallpaper engine then transmits this command to the foreground service process via an inter-process communication interface, triggering subsequent audio capture stop and audio data upload processes.

[0046] Example 4:

[0047] In its implementation, the dynamic wallpaper engine transmits the audio acquisition termination command to the foreground service process via an inter-process communication interface (IPC). This IPC is based on a service binding mechanism, and the audio acquisition termination command is transmitted through the stop audio acquisition method defined within the IPC. Upon receiving the audio acquisition termination command, the foreground service process immediately executes the stop operation on the audio acquisition instance. The foreground service process calls the stop recording method of the AudioRecord object maintained internally by the audio acquisition instance, causing the AudioRecord object to cease reading audio data from the microphone hardware. After calling the stop recording method, the foreground service process sends a thread interruption signal to the loop reading thread within the audio acquisition instance. After completing the byte array reading operation for the current loop cycle, the loop reading thread exits the loop by detecting the thread interruption status, stopping the reading of audio data frames from the AudioRecord object's buffer.

[0048] The foreground service process acquires the encoded audio data output by the audio acquisition instance from startup to shutdown. The audio acquisition instance internally maintains an audio data buffer queue; each audio data frame read by the looping read thread is appended to the audio data buffer queue in chronological order. The foreground service process iterates through the audio data buffer queue, sequentially concatenating each audio data frame into a continuous Pulse Code Modulation (PCM) audio data block. Based on the sampling rate, number of channels, and encoding format parameters in the audio encoding parameters, the foreground service process constructs the audio file header information. The audio file header information includes a Resource Interchange File System (RISS) block and a format block. The RISS block records the file type identifier and total file length, while the format block records fields such as the audio encoding format identifier, number of channels, sampling rate, byte rate, and bit depth. The foreground service process merges the constructed audio file header information with the concatenated PCM audio data blocks and encapsulates them into an audio file. The audio file encapsulation format uses a waveform audio file format.

[0049] In some embodiments, the foreground service process calculates the total length of the pulse code modulation audio data block in the following manner to fill the total file length field of the audio file header information:

[0050]

[0051] in, This indicates the total length of the pulse code modulation audio data block, in bytes. This represents the value of the sampling rate parameter configured for the audio acquisition instance, in Hertz (Hz). In some embodiments... Set to 16000; This indicates the bit depth corresponding to the encoding format parameters, in bits. In some embodiments... Set to 16; The value of the channel number parameter, which represents the configuration of the audio acquisition instance, is shown in some embodiments. Set to 1; This indicates the duration of audio acquisition, in seconds. The duration of audio acquisition is calculated by the foreground service process by recording the time difference between the start and stop times of the audio acquisition instance. The time difference is obtained by calling the system clock's millisecond value method, subtracting the start time from the stop time's millisecond value, and then dividing the difference by 1000 to convert it into seconds.

[0052] The foreground service process writes the packaged audio file to the local cache directory. The local cache directory is obtained through the Android system's context object; specifically, the context object's method for obtaining the cache directory is called to get the file path object of the cache directory. The foreground service process creates a file named after the audio capture session identifier and timestamp in the cache directory, and writes the byte data of the audio file to this file through a file output stream.

[0053] After the audio file is successfully written to the local cache, the foreground service process starts a separate network request module to upload the audio file. This separate network request module is implemented using a Hypertext Transfer Protocol (HTTP) client library and runs in a thread pool, without blocking the main thread of the foreground service process. The module constructs a multi-part form request body, which includes a file portion carrying the audio file's input stream and filename. The module then sends a POST request to the server's upload interface address, uploading most of the form request body as the request content. The server's upload interface address is pre-configured in the foreground service process's configuration parameters.

[0054] After the independent network request module completes the audio file upload, the foreground service process receives the Hypertext Transfer Protocol (HTTP) response status code returned by the server. When the foreground service process determines that the HTTP response status code indicates successful upload, it constructs a confirmation message indicating that the acquisition and upload are complete. This confirmation message includes the server-side storage path information for the audio file. The foreground service process then sends this confirmation message to the dynamic wallpaper engine via the inter-process communication interface.

[0055] In response to the confirmation message indicating completion of data acquisition and upload, the dynamic wallpaper engine executes the reconstruction process of the player component. The engine calls the release method of the player component manager to ensure that the resources occupied by the previously paused player component are completely reclaimed. The engine then creates a new player component instance, whose specific implementation type is a wrapper object for the video decoding and rendering component. The engine calls the initialization method of the newly created player component, passing the surface holding object of the wallpaper display surface as a parameter, thus rebinding the new player component to the surface holding object. The newly created player component obtains the corresponding Surface object through the surface holding object and sets the Surface object as the rendering target after video decoding. The engine reads the playback position information of the player component from persistent storage at the time of the last pause. The playback position information is the timestamp offset value of the current video file before the pause operation, in milliseconds. The engine calls the positioning method of the newly created player component, setting the playback position parameter to the read timestamp offset value, so that the video decoding and rendering operation resumes from the last paused playback position.

[0056] After the newly created player component resumes video decoding and rendering operations, the live wallpaper engine resumes the periodic video resource query task. The live wallpaper engine calls the scheduling method of the scheduled task manager to add the periodic video resource query task back to the scheduled task queue, maintaining the query interval parameter unchanged from its previous configuration value. After the periodic video resource query task resumes, the live wallpaper engine periodically sends Hypertext Transfer Protocol requests to the server according to the preset query interval to obtain wallpaper video resource update information.

[0057] Example 5:

[0058] In specific implementation, please refer to Figure 2When the dynamic wallpaper engine determines that the touch coordinates fall within the lower half of the interactive area and the gesture recognition module parses the double-tap gesture type, it generates a voice content query command. The dynamic wallpaper engine then transmits this command to the foreground service process via an inter-process communication (IPC) interface. This IPC interface is implemented based on the Android operating system's binding service mechanism, with the dynamic wallpaper engine acting as the client and the foreground service process as the server. The IPC interface defines a voice query method, whose method signature includes a string parameter used to pass the current pet identifier. The dynamic wallpaper engine internally maintains a pet identifier string corresponding to the current wallpaper theme. This pet identifier string is read from a local configuration file during wallpaper theme initialization or provided by the server through a resource description field when the wallpaper resource is distributed. When generating the voice content query command, the dynamic wallpaper engine retrieves the current pet identifier string from memory, fills it as a parameter into the request object of the voice query method, and initiates a remote call to the voice query method via the IPC interface.

[0059] The foreground service process responds to a voice content query command by extracting the current pet identifier string from the received request object. The foreground service process calls its internally integrated network request module, which is implemented based on a Hypertext Transfer Protocol (HTTP) client library and runs in the foreground service process's thread pool. The network request module constructs an HTTP GET request, the target address of which is the server's voice content query interface address, pre-configured in the foreground service process's remote configuration parameters. The query parameters of the HTTP GET request include a pet identifier field, the value of which is the current pet identifier string extracted from the voice content query command. In some embodiments, the query parameters also include a client version number field and a timestamp field. The client version number field is used for server-side compatibility checks, and the timestamp field is used to avoid HTTP caching.

[0060] The network request module sends a constructed Hypertext Transfer Protocol (HTTP) GET request to the server. Upon receiving the GET request, the server queries its database for the network address of the latest voice content corresponding to the current pet's identifier. This latest voice content network address is a Uniform Resource Locator (URI) string pointing to the storage location of the latest pet voice audio file on the server. The server returns an HTTP response with a JSON body containing a voice address field, the value of which is the latest voice content network address. The network request module in the foreground service process receives the HTTP response, parses the JSON data, extracts the voice address field value, and obtains the latest voice content network address string. The network request module then returns this string to the main thread of the foreground service process. The foreground service process, through a callback method defined in the inter-process communication interface, encapsulates the latest voice content network address string in a returned result object and sends it to the dynamic wallpaper engine.

[0061] After receiving the latest voice content network address from the foreground service process, the live wallpaper engine passes it to the audio player instance running in the foreground service process. The audio player instance is initialized when the foreground service process starts, and it is instantiated based on the Android system's MediaPlayer or ExoPlayer class. The live wallpaper engine calls the audio playback control method through an inter-process communication interface. This method includes a playback address parameter, which the live wallpaper engine passes as the value of the latest voice content network address. Upon receiving the playback command and the latest voice content network address, the audio player instance in the foreground service process calls its data source setting method, setting the latest voice content network address as its data source. Internally, the audio player instance performs an asynchronous network data retrieval operation, initiating a Hypertext Transfer Protocol (HTTP) GET request to the latest voice content network address to retrieve voice and audio data from the server. The voice and audio data can be encoded in MP3 or AAC format.

[0062] After retrieving the initial buffered data of the audio data, the audio player instance begins outputting the audio data through the speaker hardware. When playing audio data, the audio player instance does not request system audio focus from the system audio service. This is achieved by: setting the audio purpose type to "Media" and the audio content type to "Speech" in the audio property configuration of the audio player instance; explicitly calling the "Set Audio Focus" method of the audio property builder, passing in the parameter value for the audio focus property as "No Focus Required"; or skipping the call to the method requesting audio focus during the playback preparation phase of the audio player instance. Because the foreground service process does not hold system audio focus, other audio playback sessions in the system will not be paused or have their volume reduced due to the foreground service process's audio playback.

[0063] Throughout the playback of audio data output by the audio player instance through the speaker hardware, the live wallpaper engine maintains its own video playback component's continuous decoding and rendering operations on the wallpaper video. The video playback component has already been initialized and bound to the surface holding object of the wallpaper display surface in the pre-processing. The video playback component continuously reads compressed video frames from the locally cached wallpaper video file, decodes the compressed video frames to obtain pixel data, and renders the pixel data onto the wallpaper display surface. The decoding and rendering operations of the video playback component do not pause or change playback position due to audio playback; the live wallpaper engine maintains the frame rate output and image continuity of the wallpaper video. The audio output of the audio player instance and the wallpaper video rendering of the video playback component are executed in parallel without interference.

[0064] See Figure 5 In the graph, the horizontal axis represents the duration of voice playback in seconds, and the vertical axis represents the cumulative amount of voice audio data in KB. In the legend, the solid blue line represents the amount of voice audio data retrieved, and the dashed red line represents the amount of voice audio data played. The curves as a whole reflect the process in Implementation Example 5 of the dynamic wallpaper engine, where, after a voice content query command is initiated, the front-end service process retrieves and plays voice audio data from the server through the audio player instance.

[0065] The curve trend shows that the amount of audio data fetched increases linearly and rapidly from the start of playback (0 seconds), reaching a peak of 80KB at approximately 4 seconds and remaining constant. This indicates that the audio player instance stably fetches audio data from the server within the first 4 seconds, and the accumulated fetched data reaches the set cache threshold or file size. Subsequently, the data fetching volume remains stable, indicating the completion of network data fetching.

[0066] The playback volume of the audio data (represented by the red dotted line) is essentially zero during the initial 0-0.3 seconds of playback, indicating that playback has not yet begun and is likely in the player's data buffering phase. Subsequently, the playback volume begins to increase linearly, with the overall growth rate slightly lower than the data fetching volume, showing that playback is ongoing but the amount of playback data lags behind the amount of fetched data. This aligns with the mechanism in Example 5 where the audio player instance uses asynchronous network fetching and simultaneous playback and buffering. The playback volume reaches a cumulative 75KB after 5 seconds, which is still below the peak fetching volume, indicating a slight discrepancy between playback progress and fetching volume.

[0067] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application.

Claims

1. A method for zoned gesture and voice interaction on a live wallpaper, characterized in that, The method includes: In the runtime environment of the live wallpaper engine, touch events acting on the wallpaper display surface are received through the touch event callback interface, and the gesture recognition module built into the live wallpaper engine parses the touch coordinate information and double-tap gesture type from the touch events; Based on the vertical height of the visible display area of ​​the wallpaper display surface, the visible display area is divided into an upper interactive area and a lower interactive area; When the touch coordinate information falls into the upper interactive area and the gesture type is double-click, the live wallpaper engine sends an audio acquisition start command to the foreground service process that runs independently of the live wallpaper engine. When the touch coordinate information falls into the lower half of the interactive area and the gesture type is double-click, the dynamic wallpaper engine sends a voice content query command to the foreground service process.

2. The method for zoned gesture and voice interaction on a dynamic wallpaper according to claim 1, characterized in that, The system receives touch events applied to the wallpaper display surface through a touch event callback interface, obtains the original touch coordinates from the touch event by the dynamic wallpaper engine, maps the original touch coordinates to the normalized coordinate system of the visible display area, calculates the ratio of the vertical distance of the touch landing point relative to the top boundary of the visible display area to the total vertical height of the visible display area in the normalized coordinate system, and determines the interactive area to which the touch landing point belongs based on the relationship between the ratio and a preset partition threshold.

3. The method for zoned gesture and voice interaction on a dynamic wallpaper according to claim 1, characterized in that, The step of the live wallpaper engine sending an audio capture start command to a foreground service process that runs independently of the live wallpaper engine includes the live wallpaper engine passing the audio capture start command to the foreground service process through an inter-process communication interface, the foreground service process responding to the audio capture start command by creating an audio capture instance, the audio capture instance being configured with microphone hardware access permissions and audio encoding parameters, and starting to capture audio data frames from the microphone hardware.

4. The method for zoned gesture and voice interaction on a dynamic wallpaper according to claim 3, characterized in that, After the foreground service process creates an audio collector instance in response to the audio capture start command, it returns a confirmation message that audio capture has been started to the dynamic wallpaper engine. Upon receiving the confirmation message, the dynamic wallpaper engine pauses the decoding operation of the current wallpaper video player component and releases the audio hardware resources occupied by the player component, while maintaining the surface ownership of the wallpaper display surface belonging to the dynamic wallpaper engine.

5. The method for zoned gesture and voice interaction on a dynamic wallpaper according to claim 4, characterized in that, After releasing the audio hardware resources occupied by the player component, the dynamic wallpaper engine continues to render preset static placeholder images on the wallpaper display surface and pauses the periodic video resource query task initiated to the server until it receives a touch event transmitted by the touch event callback interface again and the gesture recognition module parses the double-tap gesture type from the touch event. At this time, the dynamic wallpaper engine sends an audio acquisition end command to the foreground service process.

6. The method for zoned gesture and voice interaction on a dynamic wallpaper according to claim 5, characterized in that, After the dynamic wallpaper engine sends an audio acquisition end command to the foreground service process, the foreground service process responds to the audio acquisition end command by stopping the acquisition operation of the audio acquisition instance, encapsulating the encoded audio data output by the audio acquisition instance into an audio file and writing it into the local cache, and uploading the audio file to the server through an independent network request module. After the upload is completed, the foreground service process sends a confirmation message to the dynamic wallpaper engine that the acquisition is complete and the upload is finished.

7. The method for zoned gesture and voice interaction on a dynamic wallpaper according to claim 6, characterized in that, After the foreground service process sends a confirmation message to the dynamic wallpaper engine indicating that the data collection and upload are complete, the dynamic wallpaper engine responds to the confirmation message by rebuilding the player component, rebinding the player component to the surface holding object of the wallpaper display surface, resuming video decoding and rendering operations from the last paused playback position of the player component, and simultaneously resuming the periodic video resource query task initiated to the server.

8. The method for zoned gesture and voice interaction on a dynamic wallpaper according to claim 1, characterized in that, The step of the live wallpaper engine sending a voice content query instruction to the foreground service process includes the live wallpaper engine transmitting the voice content query instruction to the foreground service process via an inter-process communication interface, the foreground service process responding to the voice content query instruction by calling a network request module to query the server for the latest voice content network address corresponding to the current pet identifier, and returning the latest voice content network address to the live wallpaper engine.

9. A method for zoned gesture and voice interaction on a dynamic wallpaper according to claim 8, characterized in that, After the foreground service process returns the latest voice content network address to the live wallpaper engine, the live wallpaper engine passes the latest voice content network address to the audio player instance running in the foreground service process. The audio player instance pulls the voice audio data from the latest voice content network address and outputs it through the speaker hardware. While the audio player instance is outputting the voice audio data, the live wallpaper engine maintains its own video playback component to continuously decode and render the wallpaper video, and the foreground service process does not request system audio focus.

10. A zoned gesture and voice interaction system for a live wallpaper, comprising a memory, a processor, and a computer program stored in the memory and running on the processor, characterized in that, When the processor executes the computer program, it implements the steps of the partition gesture and voice interaction method on a dynamic wallpaper as described in any one of claims 1 to 9.

Images