Control method and device

Abstract

The invention discloses a control method and device, and the method comprises the steps: obtaining a plurality of target interface images of a target application of electronic equipment in a running process, the target interface images comprising image frames with target objects; obtaining behavior data of the target object in the plurality of target interface images; and in the process of outputting the target interface image by the electronic equipment, controlling the target object to execute a behavior action corresponding to the behavior data in the visual perception area of the target user.

Description

Technical Field

[0001] The present application relates to the field of information processing technology, and more specifically to a control method and device. Background Art

[0002] Wallpapers for electronic devices can be divided into dynamic wallpapers and static wallpapers according to their presentation forms. Static wallpapers refer to wallpapers with static images as background images, while dynamic wallpapers refer to wallpapers with dynamic presentation effects. However, the presentation forms of current dynamic wallpapers are relatively simple, which cannot accurately provide a sense of participation for the current user, reducing the presentation effect and application accuracy of dynamic wallpapers. Summary of the invention

[0003] In view of this, this application provides the following technical solutions:

[0004] A control method, comprising:

[0005] Acquire a plurality of target interface images of a target application of the electronic device during operation, wherein the target interface images are image frames including target objects;

[0006] Obtaining behavior data of the target object in the plurality of target interface images; and,

[0007] During the process of the electronic device outputting the target interface image, the target object is controlled to perform a behavior action corresponding to the behavior data in the visual perception area of ​​the target user.

[0008] Optionally, it also includes:

[0009] obtaining association information between the plurality of target interface images, and integrating the plurality of target interface images into a target image set based on the association information; and

[0010] When the electronic device obtains the first trigger operation, multiple target interface images in the target image set are played in sequence according to the target sequence.

[0011] Optionally, the step of acquiring a plurality of target interface images of a target application of the electronic device during operation includes at least one of the following:

[0012] Monitoring the running process of the target application, and capturing the application interface presenting the target event as the target interface image;

[0013] Monitoring the behavior data of the target object in the application interface of the target application, and capturing the application interface where the target object performs the target behavior as the target interface image;

[0014] Monitoring the operation data of the target user acting on the electronic device, responding to the target operation of the target object, and intercepting the current application interface corresponding to the target operation as the target interface image;

[0015] An application interface of a target application at a target moment during its operation is captured as the target interface image.

[0016] Optionally, the obtaining of the behavior data of the target object in the plurality of target interface images includes:

[0017] Segmenting a first foreground image including only the target object from the target interface image;

[0018] The first foreground image is identified to obtain a first motion trajectory of the target object in the target image set.

[0019] Optionally, integrating the multiple target interface images into a target image set includes at least one of the following:

[0020] Directly stitching a plurality of target interface images captured during the operation of the target application into the target image set according to the image capture time;

[0021] Segmenting a foreground image including only the target object from the multiple target interface images, and generating the target image set based on the foreground image;

[0022] Segmenting a background image that does not include the target object from the multiple target interface images, and generating the target image set based on the background image;

[0023] A foreground image including the target object and a background image excluding the target object are segmented from the multiple target interface images, and the target image set is generated based on the foreground image and the background image.

[0024] Optionally, it also includes:

[0025] Generate a description text capable of representing the image content of the target interface image, so as to synchronously output the corresponding description text when the electronic device plays the target interface image.

[0026] Optionally, the generating of the description text capable of representing the image content of the target interface image includes at least one of the following:

[0027] Performing target generation processing on the target interface image using a target generation model to obtain a description text corresponding to the target interface image;

[0028] The target interface image and the annotation information for the target interface image are input into a target generation model to generate a description text corresponding to the target interface image.

[0029] Optionally, the controlling the target object to perform a behavior action corresponding to the behavior data in a visual perception area of ​​the target user includes:

[0030] The configuration information of the target object in the target interface image is obtained, and the motion parameters of the target object are determined based on the configuration information to control the target object to perform a behavior action corresponding to the first motion trajectory based on the motion parameters.

[0031] Optionally, it also includes:

[0032] In response to the interactive operation between the target user and the target object in the visual perception area, the target object is controlled to perform a behavioral action corresponding to the second motion trajectory, and / or the electronic device is controlled to switch from playing the current image frame to playing the target image frame.

[0033] A control device, comprising:

[0034] A first acquisition unit, configured to acquire a plurality of target interface images of a target application of the electronic device during operation, wherein the target interface images are image frames including target objects;

[0035] A second acquisition unit is used to obtain the behavior data of the target object in the multiple target interface images; and

[0036] A control unit is used to control the target object to perform a behavior action corresponding to the behavior data in the visual perception area of ​​the target user during the process of the electronic device outputting the target interface image.

[0037] A storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the control method as described in any one of the above items is implemented.

[0038] An electronic device, comprising:

[0039] A memory, used to store applications and data generated by the operation of the applications;

[0040] The processor is used to execute the application program to implement any of the control methods described above. BRIEF DESCRIPTION OF THE DRAWINGS

[0041] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are merely embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on the provided drawings without paying any creative work.

[0042] Figure 1 A flow chart of a control method provided in an embodiment of the present application;

[0043] Figure 2 A schematic diagram of an application scenario provided for an embodiment of the present application;

[0044] Figure 3 A schematic diagram of multi-screen linkage playback of dynamic wallpapers in a game application scenario provided by an embodiment of the present application;

[0045] Figure 4 A schematic diagram of the structure of a control device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0046] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0047] The terms "first" and "second" and the like in this application are used to distinguish different objects, rather than to describe a specific order. In addition, the terms "including" and "having" and any variations thereof are intended to cover non-exclusive inclusions. For example, a process, method, system, product or device comprising a series of steps or units is not limited to the listed steps or units, but includes steps or units that are not listed.

[0048] A control method provided in an embodiment of the present application can be applied to electronic devices such as mobile phones, tablet computers, wearable devices, vehicle-mounted devices, augmented reality (AR) / virtual reality (VR) devices, laptop computers, ultra-mobile personal computers (UMPCs), netbooks, personal digital assistants (PDAs), etc. The embodiment of the present application does not impose any restrictions on the specific types of electronic devices.

[0049] The control method of the embodiment of the present application is applied to the dynamic wallpaper display scene, wherein the dynamic wallpaper is a wallpaper with animation, and the application scope of the dynamic wallpaper includes the desktop, lock screen page or screen off page of the electronic device, etc., and can also be the display main interface of a specific application, etc. Based on the control method, the behavior of the target object in the dynamic wallpaper can be controlled, the user's sense of participation can be enhanced, and the personalized display effect of the dynamic wallpaper can be improved, thereby improving the user's visual experience.

[0050] See also Figure 1 , is a flow chart of a control method provided in an embodiment of the present application, the method may include the following steps:

[0051] S101, obtaining a plurality of target interface images of a target application of an electronic device during operation.

[0052] The target application of the electronic device may be an application currently displayed and running by the electronic device, such as an application currently displayed by the electronic device, that is, the front-end application of the electronic device. Correspondingly, the target application may also be an application specified by the user, such as an application frequently used by the user. Specifically, the types of target applications may include game applications, video playback applications, live broadcast applications, conference applications, chat applications, etc. There may be multiple interface images during the operation of the target application, and the interface image may be the main interface of the target application, or the image content presented when the target application is running. The target interface image may come from the same target application or from different target applications. The target interface image may be an image with specific elements of multiple interface images of the target application during operation. The target interface image is a component image of the dynamic wallpaper of the electronic device. Specifically, the target interface image includes an image frame with a target object, and the target object may be a key display object (including a specific element in the display content) in the display content of the target application. For example, when the target application is a game application, the target object may be a player character or a non-player character (such as a monster, a game NPC, etc.) in the game, and further may be a game character used by the target application of the current game application in the game. When the target application is a video application, the target object can be the main character in the video, such as the protagonist in the current video clip. When the target application is a video conferencing application, the target object can be the speaker in the video conference, etc. In the embodiment of the present application, multiple target interface images of the target application of the electronic device during operation can be obtained by taking screenshots of the application interface, or the corresponding target interface images can be obtained by connecting to the interface for displaying the image content of the target application, such as obtaining the image rendered by a graphics processor or a graphics card from the bottom layer through the API (Application Programming Interfac, application programming interface) of the target application for display by the target application. The target interface image can be a static image or a dynamic image converted from a video clip. The target interface image includes an image frame with a target object. In order to better highlight the characteristics of the target object and increase the user's experience effect, the image frame with the target object can be an image frame in which the target object meets a specific condition, and the specific condition can be the operation information of the target user operating the target application, and the condition related to the target application presenting the interactive information with the target object. For example, the image frame with the target object can be an image frame of the highlight moment of the target object. When the target application is a game application, the target interface image can be an image frame corresponding to a game character using a specific skill, or an image frame representing the moment when the game character of the current target user defeats other game characters; when the target application is a video application, the target interface image can be an image frame corresponding to a wonderful clip of the protagonist.By acquiring multiple target interface images of a target application of an electronic device during operation, relevant features of the target object can be presented in a dynamic wallpaper, thereby improving the user experience.

[0053] S102: Obtaining behavior data of a target object in a plurality of target interface images.

[0054] During the operation of the target application, there will be multiple target interface images. The behavior data of the target object in the corresponding target interface images may be the same or different. In order to reflect the unique characteristics and status of the target object in the content image of the target application, the behavior data of the target object in multiple target interface images can be obtained. The behavior data mainly reflects the action characteristics, attribute characteristics and some characteristics that distinguish the target object from other objects. For example, in a game application, the target object can be a game character, and the target interface image includes an image frame with the highlight moment of the game character. The behavior data corresponding to the game character can include the game character behavior data corresponding to the game character, which can specifically be the game character jumping, bypassing, flying, releasing big moves, moving positions, standing on the podium, jumping, crawling and other game character behavior data in the game scene. For another example, in a video application, the target object can be the protagonist in the video, and the target interface image includes an image frame with the highlight moment of the protagonist. Correspondingly, the behavior data of the protagonist in the target interface image can include various behavior data such as the expression behavior and action behavior of the protagonist, which can specifically be the data of the protagonist producing a specific performance behavior (such as the behavior data of the protagonist dancing). For another example, the target application is a video conferencing application, the target object may be a speaker or host in the video conference, and the corresponding behavior data of the speaker may be the action behavior data of the speaker's speech.

[0055] S103: When the electronic device outputs the target interface image, the target object is controlled to perform a behavior action corresponding to the behavior data in the visual perception area of ​​the target user.

[0056] By acquiring the behavior data of the target object in multiple target interface images, the subsequent actions of the target object when the electronic device outputs a dynamic wallpaper can be determined through the behavior data, so that the behavior actions of the target object can be consistent with the current application scenario, thereby improving the interactivity between the user and the dynamic wallpaper and personalizing the user's dynamic wallpaper.

[0057] Specifically, since the target interface image is a component image of the dynamic wallpaper of the electronic device, the process of the electronic device outputting the target interface can be the process of the electronic device playing the dynamic wallpaper, for example, it can be the process of the electronic device outputting the dynamic wallpaper when the screen is locked, or it can be the process of the electronic device outputting the dynamic wallpaper on the desktop. Specifically, in the process of the electronic device outputting the target interface image, the target object can be controlled to perform a behavior action corresponding to the behavior data of the target object in the visual perception area of ​​the target user, and the visual perception area includes the display area of ​​the electronic device corresponding to the display screen, and can also include the 3D display area of ​​the electronic device, such as the spatial three-dimensional projection area of ​​the electronic device, or the three-dimensional space area that can be perceived by the target user when the electronic device displays and outputs three-dimensional object data. The behavior action performed by the target object matches the behavior data in the multiple target interface images. Specifically, the behavior action can be a key action extracted from the behavior data, or an action generated by analyzing the behavior data. The current behavior action of the target object can be a behavior action corresponding to the behavior data that is unrelated to the dynamic wallpaper content currently played by the electronic device. For example, when the target application is a game application, the main interface of the game application, the introduction interface including a specific game character, and the highlight moment image including the specific game character can be used as the interface image of the dynamic wallpaper. When the image content corresponding to the current dynamic wallpaper output by the electronic device is the main interface of the game application, the behavior action of the game character corresponding to the target user can be presented in the image, such as presenting the jumping action of the game character, etc. Correspondingly, the current behavior action of the target object can also be a behavior action corresponding to the behavior data associated with the current output content of the dynamic wallpaper. If the image content corresponding to the current dynamic wallpaper is an image of a game character using special effects in a game application, the behavior action performed by the target object in the visual perception area of ​​the target user can be determined based on the action of the game character using special effects in the image content. For example, it can be the action of the game character releasing a big move using the special effect, etc.

[0058] In the embodiment of the present application, the behavior data of the target object in the target interface image can be determined according to the multiple target interface images of the target application of the electronic device during operation, and the target object is controlled to perform the behavior action corresponding to the behavior data in the visual perception area of ​​the target user during the process of the electronic device outputting the target interface image. In this way, the target user can participate more deeply in the dynamic change of the dynamic wallpaper corresponding to the target interface image, and the target object in the target application can also perform actions according to the behavior data of the target object in the change of the dynamic wallpaper, which increases the fun and improves the personalized playback effect of the dynamic wallpaper.

[0059] The control method of the embodiment of the present application is described below in combination with actual application scenarios.

[0060] In one embodiment of the present application, the control method also includes: obtaining association information between multiple target interface images, integrating the multiple target interface images into a target image set based on the association information; and, when the electronic device obtains a first trigger operation, playing the multiple target interface images in the target image set in sequence according to the target sequence.

[0061] During the operation of the target application, multiple target interface images can be obtained. For example, images can be captured or the main image interface can be extracted according to the image content output by the target application as the target interface image. The target interface image can be multiple images including the target object at different times, or a series of images describing the complete situation during the operation of the target application. In order to improve the effect of outputting the target interface image and the interactive program with the target user, the association information between the multiple target interface images can be determined first. The association information can be a temporal association relationship representing each target interface image, such as the temporal association of the image frames captured by the image content output by the target application corresponding to each target interface image. The association information can also be the spatial position association relationship of each target interface image, such as the association and continuity of the moving position of the target object included in the target interface image. The association information can also include the association of the source of the target interface image. Further, multiple target interface images can be integrated into a target image set based on the association information. Specifically, if the association information represents the temporal association relationship of the target interface image, each target interface image can be spliced ​​and integrated into a target image set according to the temporal sequence according to the temporal association relationship, that is, the target image set is determined as a dynamic wallpaper set. It is also possible to perform image segmentation processing on each target interface image according to the associated information to obtain a foreground image including the target object and the remaining background images, and then integrate the foreground image and multiple background images into image subsets respectively, and then associate the image subsets according to the associated information of the target interface image to form the target image set.

[0062] In the case where the electronic device obtains the first trigger operation, multiple target interface images in the target image set are played in sequence according to the target sequence. The first trigger operation represents the operation of triggering the electronic device to play the dynamic wallpaper. For example, the first trigger operation can be the operation of the target user manually triggering the dynamic wallpaper to be played, or it can be a trigger instruction that automatically triggers the electronic device to play the dynamic wallpaper according to the state of the electronic device. For example, when the electronic device enters an idle state or a dormant state, the electronic device can be automatically triggered to play the dynamic wallpaper. Specifically, it can be that when the operator of the electronic device is detected to be away from the electronic device, or the electronic device starts a multi-screen mode (wherein the multi-screen mode includes using multiple screens at the same time, that is, a mode in which multiple screens are all in a lit state or a use state), such as when a dual-screen device enables the dual-screen mode or a single-screen device is connected to an extended screen, an instruction to play the dynamic wallpaper can be issued on a display screen that currently does not display real-time data. Correspondingly, the target sequence based on which the multiple target interface images in the target image set are played can be determined according to the above-mentioned association relationship, such as a target sequence determined by sorting according to the generation time of the target interface image, or sorting each target interface image according to the spatial position change of the target object in the target interface image, etc. In this way, through the association relationship between each target interface image, when playing the dynamic wallpaper (i.e., the target image set), the target interface image can be played according to the association relationship, so that the image content of the dynamic wallpaper played can be more relevant and more in line with the actual application scenario, thereby improving the user experience and interactive connectivity. Of course, in other embodiments, if the screen of the electronic device is currently outputting the display content of other applications, it is also possible to control only the target object to be output in the visual perception area provided by the electronic device for the target user by floating or other methods that do not affect the output of the current display content.

[0063] On the basis of the above embodiment, the method of acquiring multiple target interface images of a target application of an electronic device during operation in the embodiment of the present application includes at least one of the following:

[0064] (1) Monitor the running process of the target application and capture the application interface presenting the target event as the target interface image.

[0065] The entire operation process of the target application can be monitored, or the operation process of a specific time period of the target application can be monitored. For example, the target application can be a game application, a video application, a live broadcast application, or a video conferencing application, etc. Specifically, the game operation process, the video playback process, the live broadcast process, or the video conferencing process can be monitored. The target event can be an event that can reflect the output of representative image content by the target application. For example, if the target application is a game application, the target event can be an event of a highlight moment of a game player killing a monster, defeating an opponent, or winning the current game stage; or an event of a game player entering a specific place or position; if the target application is a video application, the target event can be an event of a character releasing a big move, performing a classic action, or a specific action in a movie video; if the target application is a live broadcast application, the target event can be an action that can reflect the highlight moment of a character's wonderful moment, such as the host introducing a new product, performing a specific gesture or action. Thus, the application interface presenting the above target event is captured as the target interface image.

[0066] (2) Monitoring the behavior data of the target object in the application interface of the target application, and capturing the application interface where the target object performs the target behavior as the target interface image.

[0067] In this embodiment, the behavior data may characterize the main action behavior, position change behavior or expression change, explanation content change, level change, task execution status and other behavior data of the target object. For example, when the target application is a game application, the behavior data may be the score data of a specific game player, the completion status of a specific task, and the defeat status of a powerful enemy. Thus, the application interface of the target object implementing the target behavior can be intercepted as the target interface image. Specifically, the image frame of a game character or speaker, movie character, etc. implementing a specific action or releasing a specific function can be used as the target interface image, such as recording the highlight moment of a game player in a game application as the target interface image. This will make the interaction with the target user stronger and enhance the demand for personalized playback of dynamic wallpapers for the target user.

[0068] (3) Monitoring the operation data of the target user on the electronic device, responding to the target operation of the target object, and capturing the current application interface corresponding to the target operation as the target interface image.

[0069] The operation data of the target user acting on the electronic device may be an operation on the image output by the target application actively triggered by the target user. For example, the operation data may include a screenshot operation actively triggered by the target user. Specifically, when the target user believes that the current output content of the target application is suitable for generating wallpaper, a specific instruction may be given or a specific operation may be performed, so that the electronic device performs a screenshot operation to obtain the target interface image.

[0070] (4) Capture the application interface of the target application at the target moment during its operation as the target interface image.

[0071] Among them, the target moment can be a moment set specifically according to the operation of the target application. For example, in a game application, the target moment can be the moment of switching between game stages, or the moment of summarizing the results at the end of each game. In a video application, the target moment can be the moment when the characters in the video switch. Correspondingly, the target moment can also be a moment corresponding to a specific time interval, such as capturing the application interface of the current target application as the target interface image at corresponding time periods, which can better reflect the characteristics of the output screen of the target application.

[0072] Based on the aforementioned embodiments, in an embodiment of the present application, the behavior data of the target object in multiple target interface images is obtained, including: segmenting a first foreground image that only includes the target object from the target interface image; identifying the first foreground image to obtain a first motion trajectory of the target object in the target image set.

[0073] In this embodiment, the behavior data of the target object may be the motion trajectory of the target object in a plurality of target interface images. In order to accurately obtain the motion trajectory of the target object, a first foreground image including the target object may be segmented out in each target interface image, and the corresponding first motion trajectory may be obtained by analyzing the position information of the target object in each foreground image. Specifically, an image segmentation algorithm may be used to extract the foreground content including only the target object and remove background interference, thereby improving the accuracy of the motion trajectory of the target object.

[0074] In one implementation of the embodiment of the present application, a plurality of target interface images are integrated into a target image set, including at least one of the following:

[0075] (1) Multiple target interface images of the target application during operation are directly spliced ​​into a target image set according to the image capture time.

[0076] In this method, each target interface image is directly spliced ​​into a target image set according to the time sequence of the captured images, that is, a dynamic wallpaper is obtained, which can better reflect the coherence of the output content of the target application.

[0077] (2) Segment a foreground image including only the target object from the multiple target interface images, and generate a target image set based on the foreground image.

[0078] In this method, a target image set is generated only based on a foreground image including a target object, that is, a dynamic wallpaper is obtained, so that the object features of the target object can be more prominent in the dynamic wallpaper, thereby improving the personalized display of the dynamic wallpaper.

[0079] (3) Segment a background image that includes not only the target object from the multiple target interface images, and generate a target image set based on the background image. In this embodiment, the target image set is generated only based on the background image, that is, a dynamic wallpaper is generated.

[0080] (4) Segment a foreground image including a target object and a background image excluding a target object from a plurality of target interface images, and generate a target image set based on the foreground image and the background image.

[0081] In this implementation, a foreground image subset can be generated based on the foreground image, that is, a foreground dynamic wallpaper can be generated; a background image subset can be generated based on the background image, that is, a background dynamic wallpaper can be generated. Then, a target image set is generated based on the foreground image subset and the background image subset, that is, the two are merged into a dynamic wallpaper that is finally played by the electronic device.

[0082] Furthermore, in order to enable the target user to obtain the representative content of the dynamic wallpaper, a description text can also be generated so that the playback of the dynamic wallpaper presents a story effect. Correspondingly, the control method also includes: generating a description text that can represent the image content of the target interface image, so that the corresponding description text is output simultaneously when the electronic device plays the target interface image.

[0083] Specifically, the image content of the target interface image can be extracted, and a description text can be generated according to the image features included in the image content. For example, a language processing model can be used to generate guidance information for the language processing model based on the image content of the current target interface image, and the model parameters of the language processing model can be adjusted based on the guidance information. The image features can be processed based on the adjusted language processing model to obtain description information, and a complete story copy can be further generated based on the description information, that is, the final description text can be obtained, thereby showing the story of the target interface image.

[0084] In one implementation of the embodiment of the present application, generating a description text capable of representing the image content of the target interface image includes at least one of the following:

[0085] (1) The target generation model is used to perform target generation processing on the target interface image to obtain a description text corresponding to the target interface image.

[0086] (2) The target interface image and the annotation information for the target interface image are input into the target generation model to generate a description text of the target interface image.

[0087] In method (1), the target interface image can be directly processed using the target generation model to obtain the description text. Specifically, the image features representing the target interface image can be input into the target generation model, and then the target generation model analyzes and processes the image features to output the description text of the target interface image. For example, the target interface images are sequentially input into the image-to-text artificial intelligence processing model, and the AI ​​capabilities of the artificial intelligence processing model are used to identify the objects and behavior data in each interface image, and a description text that can represent the image content of the entire interface image is generated based on the behavior data of the target object; and the description text corresponding to each interface image can be polished and optimized according to the correlation between the previous and next frame interface images, so that the description text corresponding to the target interface image set has consistency and continuity, etc., so as to improve the reading experience of the target user.

[0088] In method (2), when the target generation model is used to generate the description text, when the target generation model processes the target interface image, it can also process it according to the annotation information of the target interface image. The annotation information can be basic text configured by the user for the target interface image, such as character information in the game interface or the configuration information of the game itself for the character, etc.

[0089] Among them, the target generation model can be a large language model, and by setting appropriate guidance information, the target generation model is guided to generate a complete story copy based on the basic text corresponding to the target application, and obtain a description text corresponding to the target interface image. For example, the target application is a game application, and the target interface image is a game plot image in the game application. The basic text of the target interface image can be generated first, such as a text describing the basic image content of the target interface image. Then, the basic text is input into the target generation model, and by setting appropriate guidance information (such as "Please generate a story copy based on the following game plot: [Basic text]"), the target generation model is guided to generate a complete story copy based on the basic text. The target generation model will polish and expand the basic text according to its pre-trained knowledge and language rules, add elements such as plot description and character emotions, so as to generate a story-rich copy, that is, to obtain a description text. For example, the basic text can be generated according to the information annotated by the target user for the game interface image, wherein the annotated content can be annotated according to the game character and the behavior category of the game character, for example, it can be annotated about the character level, equipment status, game map, skill name, battle result, etc. The specific annotation content can be annotated according to the actual situation of the current game. For example, the template of the annotation information may be "a character of level [character level], equipped with [equipment status], using [skill name] skill in [level or map name], [battle result], and [achievement mark], and the game progress is [game progress]", according to the template and the current game status, the annotation information obtained may be "a character of level 30, equipped with a rare sword, using the flame shock skill in the dark castle and winning and achieving the achievement of defeating 10 elite enemies in a row, and the game progress is 80%", then the basic text generated according to these annotation information may be "a character of level 30, equipped with a rare sword, using the flame shock skill in the dark castle, winning, and achieving the achievement of defeating 10 elite enemies in a row, and the game progress is 80%". The basic text may be further processed by the target generation model to obtain the final description text. In the process of generating the description text, the basic text may be polished and expanded, and elements such as plot description and character emotions may be added to generate a story-rich copy. For example, the description text may show in detail the complete game process of the game character corresponding to the target user starting from a novice, going through various adventure challenges and finally growing into a hero according to the above basic text. The corresponding storyline description of the descriptive text provided with each target interface image can deepen the user's understanding of the game storyline, thereby strengthening the user's memory of the game experience and emotional resonance.

[0090] In one implementation of the embodiments of the present application, controlling the target object to perform a behavioral action corresponding to the behavioral data in the visual perception area of ​​the target user includes: obtaining configuration information of the target object in the target interface image, and determining the motion parameters of the target object based on the configuration information to control the target object to perform a behavioral action corresponding to the first motion trajectory based on the operating parameters.

[0091] Among them, the configuration information of the target object in the target interface image can be configured according to the type of the target application and the type of the target object in the target application. Further, the configuration information can be the default configuration information about the target object automatically configured by the target application, or it can be the custom configuration information for the target object optimized by the target user according to the default configuration information. For example, in a game application, the configuration information of the target object in the target interface image can include the configuration information of the classification of game characters, such as the classification of characters including agility, strength, magic, and flying characters; the configuration information of the movement speed of the game character, and the speed parameters of the movement of different game characters can be controlled according to the movement speed configuration information. The acceleration parameter of the game character, based on which the acceleration of the movement of different characters can be controlled; the movement trajectory configuration information of the game character, based on which the horizontal movement, vertical movement, jumping movement mode, etc. of different game characters can be controlled; the head turning frequency configuration information of the game character, based on which the game character can walk how many pixels on the screen to turn his head once, etc.; the teleportation frequency configuration information of the game character, and the configuration information can be used to determine whether the character is a magic type, if there is a teleportation condition, and update the coordinates of the teleportation, etc. Then, the motion parameters of the target object can be determined according to these configuration information, so that the target object performs a behavior corresponding to the first motion trajectory based on the motion parameters. The first motion trajectory can be the moving trajectory of the target object in the image interface image. Taking the game application as an example, the target object performs a behavior corresponding to the motion trajectory during the game in the visual perception area. The behavior of the target object in the first motion trajectory is controlled by motion parameters. Still taking the above-mentioned game application as an example, the motion parameters determined by the above-mentioned configuration information can be used to control the behavior of the target object (such as a game character) in the dynamic wallpaper. When the dynamic wallpaper is played on the desktop of the electronic device, the game character can be controlled based on the motion parameters to avoid obstacles when encountering desktop icons, and avoid obstacles when encountering desktop boundaries. Further, when the mouse clicks on other screens, the game character can follow the mouse to move to other screens. Thereby improving the interactivity of the target object with the user in the dynamic wallpaper.

[0092] In one implementation of the embodiments of the present application, the control method also includes: in response to the interaction operation between the target user and the target object in the visual perception area, controlling the target object to perform a behavioral action corresponding to the second motion trajectory, and / or controlling the electronic device to change from playing the current image frame to playing the target image frame.

[0093] In the embodiment of the present application, the operation type of the interactive operation generated by the target user can be a voice interactive operation, a string interactive operation, such as a string corresponding to the keyboard input for indicating the relevant operation; it can also be a gesture operation, etc. The visual perception area can be the display desktop of the electronic device, or it can be a 3D display area. The target user can interact with the target object in the desktop or 3D display area, which can be based on gestures or guided operations. For example, the gesture is to re-plan the motion trajectory of the target object, that is, the second motion trajectory can be generated according to the interactive operation and the first motion trajectory, so as to control the target object to perform the behavior action with the second motion trajectory. Correspondingly, the interactive operation can also be an operation indicating the switching of the image content of the current dynamic wallpaper, that is, the dynamic wallpaper content corresponding to the currently displayed image frame can be switched to the dynamic wallpaper content corresponding to the target image frame based on the interactive operation with the target object. In this way, the motion trajectory, behavioral actions, and / or dynamic wallpaper content of the target object can be controlled according to the user's interactive operation, thereby enhancing the user's interactive experience with the dynamic wallpaper.

[0094] The following describes the control method of an embodiment of the present application by taking the target application as a game application and controlling an electronic device to play a dynamic wallpaper as an example.

[0095] See also Figure 2, is a schematic diagram of an application scenario provided by an embodiment of the present application. When an electronic device runs the game application, the user of the electronic device, i.e., the target user, corresponds to role A in the game application. The target user controls role A to perform tasks in the game, such as defeating monsters, or competing with other game characters. In this process, the game's application program interface (API) or image capture technology can be used to obtain game screen data. The game screen data is then analyzed to obtain multiple target interface images. Specifically, a deep learning algorithm can be used to perform a deep analysis of the acquired game screen data. The specific analyzed data content may include the scores of each game character in the current game, the completion of a specific task, and the defeat status of a powerful enemy (such as a super monster, etc.), and then the screens corresponding to the highlight moments of role A in the game screen are determined in turn, and these screens are used as target interface images. Specifically, after identifying the highlight moment of the game character A representing the target user, the corresponding game screen segments can be accurately captured and converted into high-quality animated images, which will constitute an important part of the dynamic wallpaper. The image frames before and after the highlight moment can also be processed by image segmentation to extract the game character corresponding to the game character A as the foreground wallpaper animated image, that is, the game character corresponding to the game character A is the target object. In this way, the movement trajectory of the game character corresponding to the foreground game character A in the desktop environment can be accurately controlled, thereby introducing a new dynamic element corresponding to the target application (that is, the game character corresponding to character A) into the dynamic wallpaper, so that the movement trajectory of the game character on the desktop is integrated with the overall visual effect of the dynamic wallpaper, thereby further enhancing the user's visual experience and sense of participation.

[0096] Further, after obtaining multiple highlight moment images of game character A as the target interface image, a comic strip-style screenshot arrangement based on game mileage can be constructed as a dynamic wallpaper for playback. The process includes arranging multiple game highlight moment screenshots of game character A related to each other in a certain logical order (such as the time sequence of the screenshots or the spatial position relationship of character A) to construct a dynamic wallpaper in the form of a comic strip. It is also possible to add text descriptions to each game highlight moment screenshot in a motion visual text association analysis processing mode. In this way, the coherence and readability of the entire story are enhanced, and the complete game process of the game protagonist starting from a novice, going through various adventure challenges, and finally growing into a hero is shown in detail. The corresponding storyline description equipped with each screenshot can deepen the user's understanding of the game storyline, thereby strengthening the user's memory and emotional resonance of the game experience, and enhancing the user's experience effect. Further, if the electronic device is in a multi-screen mode, that is, the electronic device may include a folding device with multiple display screens, or a device connected to an extended display screen, or a device capable of projecting at least one display area in space, etc. The above-constructed comic strip-style dynamic wallpaper can be played on multiple screens through a multi-screen linkage processing mode. During playback, the ratio of content between each screen can be changed synergistically based on the game storyline, thus presenting users with a coherent and immersive multi-screen dynamic wallpaper visual effect centered on the game content.

[0097] In the above application scenario, the screenshots of the highlight moments of the game character A corresponding to the target user at different times can be analyzed to extract key information, construct basic captions, generate captions for the highlight moment screenshot stories, and integrate them into the highlight moment screenshots to create a dynamic background map. In this process, in order to achieve better image presentation effects, different screenshots can be preprocessed. The preprocessing process includes: performing unified format conversion (such as conversion to RGB format) and size adjustment on the highlight moment screenshots of the target user at different times, and performing image enhancement operations, such as adjusting brightness and contrast, to improve image quality and reduce noise interference. In the process of generating descriptive text, the recognized text can also be post-processed through the OCR (optical character recognition) application, such as correcting erroneous characters, removing unnecessary symbols, etc., to ensure the accuracy of the subsequently generated descriptive text. In this process, a pre-trained convolutional neural network (CNN) model or a self-trained CNN model can also be used to extract features of image objects such as game characters, tasks, and equipment in the screenshots. For example, the appearance features of the character can be identified to determine the character's level, and the enemy's appearance can be identified to judge the enemy's strength, etc., so as to more accurately determine the highlight moments representing the target user and obtain the target interface image.

[0098] Furthermore, target detection algorithms (such as YOLO or Fast-RCNN) can also be used to locate key targets in the screenshot, such as the positions of characters, enemies, task-related elements, etc. This helps to analyze the subsequent information related to these targets, such as determining the equipment of the character, judging the results of the battle, and further determining the movement trajectory, highlight moments, etc. of the character. In order to accurately obtain the relevant features of the target object, image segmentation technology can also be used to segment the different elements in the screenshot, such as separating the character from the background, so as to more clearly analyze the equipment, status and other information of the character. In the process of generating the description text of each target interface image in the dynamic wallpaper, in addition to the image content features, it can also be processed according to the annotation information to improve the accuracy of the description text. Among them, the annotation information can be manually annotated for the screenshot information after the above-mentioned recognition processing, and annotated according to the major categories such as character level, equipment situation, skill use, level or map name, game progress, enemy strength, battle results, and achievement identification. For example, the specific value of the character level, the type and quality of the equipment, etc. are clearly marked. Based on the labeled data, generate basic copy according to the preset copy template, that is, "[Character level] character, equipped with [equipment status], using [skill name] skill in [level or map name], [battle result], and [achievement mark], the game progress is [game progress]". For example, if the character level is 30, equipped with a rare sword, uses the flame shock skill in the dark castle and wins, and achieves the achievement of defeating 10 elite enemies in a row, and the game progress is 80%, then generate "30-level character, equipped with a rare sword, uses the flame shock skill in the dark castle, wins, and achieves the achievement of defeating 10 elite enemies in a row, and the game progress is 80%". Input the basic copy into the large language model (such as the GPT series). By setting appropriate prompt words (such as "Please generate a story copy based on the following game plot: [Basic copy content]"), guide the large language model to generate a complete story copy based on the basic copy. The large language model will polish and expand the basic text based on its pre-trained knowledge and language rules, adding elements such as plot description and character emotions to generate a text with rich storytelling. Finally, a dynamic wallpaper is generated based on the text and multiple target interface images.

[0099] Furthermore, in order to enhance the interest of the target object in the dynamic wallpaper and the interactivity with the user, the behavior of the target object (i.e., the game character) can be controlled according to the characteristics of the target object in the target interface image, such as controlling the moving trajectory of the moving game character to automatically jump over, bypass, or fly over obstacles when encountering obstacles. Specifically, the motion parameters can be determined according to the configuration information for the game character, and the target object can be controlled to perform corresponding behavior according to the motion parameters.

[0100] The configuration information of the game characters in the game application includes but is not limited to:

[0101] Role classification: Agility, strength, magic, flying roles

[0102] Movement speed: Controls the speed parameters of different characters' movements

[0103] Acceleration: Controls the movement and parameters of different characters

[0104] Movement trajectory: control the horizontal movement, vertical movement, and jumping movement of different characters

[0105] Head turning frequency: How many pixels on the screen are required to turn the head once

[0106] Teleport frequency: Determine whether it is a magic type, if there are teleport conditions, teleport and update coordinates

[0107] In this way, the motion parameters of the game character presented in the dynamic wallpaper can be determined based on the above configuration parameters, so that it can avoid desktop icon obstacles and boundary obstacles, and follow the mouse to teleport to other screens when the mouse clicks on other screens.

[0108] See also Figure 3 , is a schematic diagram of multi-screen linkage playback of dynamic wallpaper in a game application scenario in an embodiment of the present application. Figure 3 In the scene shown, the visual perception area of ​​the target user presented by the electronic device is a plurality of spatial 3D display areas. When the target user is detected to be gazing at the display area 301, the background image presented by the dynamic wallpaper in the display area 301 is image 1, wherein the game character performs corresponding behavioral actions in the display area 301, such as rotation, etc. When the user is detected to be gazing at the display area 302, the game character will quickly move to the image 2 of the background image corresponding to the display area 302. In addition, image 1 and image 2 may include a description text of the game character, such as the text describing the initial growth of the game character in image 1, and the text describing the game character at the highest level may be displayed in image 2. In this way, comic book-style dynamic wallpapers can be played in multiple display areas to achieve coordinated changes in content between display areas.

[0109] The control method provided in the embodiment of the present application can enhance the user's sense of participation. For example, the user can control the movement of the target object, screen linkage, and dynamic behavior based on the characteristics of the game character through the mouse. The user can participate more deeply in the dynamic changes of the wallpaper. The game character can move on the wallpaper according to the character type to increase the fun. Based on the analysis of the highlight moment screenshots and the customization of wallpapers according to the player's game information, each user's wallpaper can reflect his or her unique experience and status in the game, making the wallpaper highly personalized. Functions such as screen linkage and automatic drawing bring coherent, rich and closely related visual effects to multi-screen displays, overcoming the problems of lack of coordination and game relevance of existing multi-screen wallpapers, and improving the overall visual experience.

[0110] In the embodiment of the present application, a control device is also provided. Figure 4 , the control device comprises:

[0111] A first acquisition unit 401 is used to acquire a plurality of target interface images of a target application of an electronic device during operation, wherein the target interface images are image frames including target objects;

[0112] A second acquisition unit 402 is used to obtain the behavior data of the target object in the multiple target interface images; and

[0113] The control unit 403 is used to control the target object to perform a behavior action corresponding to the behavior data in the visual perception area of ​​the target user during the process of the electronic device outputting the target interface image.

[0114] Optionally, the device further comprises:

[0115] an integration unit, configured to obtain association information between the plurality of target interface images, and integrate the plurality of target interface images into a target image set based on the association information; and

[0116] The playing unit is used to play a plurality of target interface images in the target image set in sequence according to a target sequence when the electronic device obtains a first trigger operation.

[0117] Optionally, the first acquiring unit includes at least one of the following:

[0118] A first interception subunit is used to monitor the running process of the target application and intercept the application interface presenting the target event as the target interface image;

[0119] A second interception subunit is used to monitor the behavior data of the target object in the application interface of the target application, and intercept the application interface where the target object performs the target behavior as the target interface image;

[0120] A third interception subunit is used to monitor the operation data of the target user acting on the electronic device, respond to the target operation of the target object, and intercept the current application interface corresponding to the target operation as the target interface image;

[0121] The fourth interception subunit is used to intercept the application interface of the target application at a target moment during the running process as the target interface image.

[0122] Optionally, the second acquiring unit includes:

[0123] A segmentation subunit, used to segment a first foreground image including only the target object from the target interface image;

[0124] The recognition subunit is used to recognize the first foreground image to obtain a first motion trajectory of the target object in the target image set.

[0125] Optionally, the integration unit includes at least one of the following:

[0126] A splicing subunit, used to directly splice a plurality of target interface images captured during the operation of the target application into the target image set according to the image capture time;

[0127] A first generating subunit is used to segment a foreground image including only the target object from the multiple target interface images, and generate the target image set based on the foreground image;

[0128] A second generating subunit is used to segment a background image that does not include the target object from the multiple target interface images, and generate the target image set based on the background image;

[0129] The third generating subunit is used to segment a foreground image including the target object and a background image excluding the target object from the multiple target interface images, and generate the target image set based on the foreground image and the background image.

[0130] Optionally, the device further comprises:

[0131] The text generation unit is used to generate a description text that can represent the image content of the target interface image, so as to synchronously output the corresponding description text when the electronic device plays the target interface image.

[0132] Optionally, the text generation unit includes at least one of the following:

[0133] A first processing subunit is used to perform target generation processing on the target interface image using a target generation model to obtain a description text corresponding to the target interface image;

[0134] The second processing subunit is used to input the target interface image and the annotation information for the target interface image into the target generation model to generate a description text corresponding to the target interface image.

[0135] Optionally, the control unit is configured to:

[0136] The configuration information of the target object in the target interface image is obtained, and the motion parameters of the target object are determined based on the configuration information to control the target object to perform a behavior action corresponding to the first motion trajectory based on the motion parameters.

[0137] Optionally, the device further comprises:

[0138] An interactive operation response unit is used to respond to the interactive operation between the target user and the target object in the visual perception area, control the target object to perform a behavioral action corresponding to the second motion trajectory, and / or control the electronic device to switch from playing the current image frame to playing the target image frame.

[0139] It should be noted that the specific implementation of each unit and sub-unit in this embodiment can refer to the corresponding content in the previous text and will not be described in detail here.

[0140] In another embodiment of the present application, a storage medium is provided, on which a computer program is stored. When the computer program is executed by a processor, the control method as described in any one of the above items is implemented.

[0141] In another embodiment of the present application, an electronic device is further provided, and the electronic device may include:

[0142] A memory, used to store applications and data generated by the operation of the applications;

[0143] A processor is used to execute the application program to implement any of the control methods described above.

[0144] It should be noted that the specific implementation of the processor in this embodiment can refer to the corresponding content in the previous text and will not be described in detail here.

[0145] In this specification, each embodiment is described in a progressive manner, and each embodiment focuses on the differences from other embodiments. The same or similar parts between the embodiments can be referred to each other. For the device disclosed in the embodiment, since it corresponds to the method disclosed in the embodiment, the description is relatively simple, and the relevant parts can be referred to the method part.

[0146] Professionals may further appreciate that the units and algorithm steps of each example described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, computer software, or a combination of the two. In order to clearly illustrate the interchangeability of hardware and software, the composition and steps of each example have been generally described in the above description according to function. Whether these functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. Professionals and technicians may use different methods to implement the described functions for each specific application, but such implementation should not be considered to be beyond the scope of this application.

[0147] The steps of the method or algorithm described in conjunction with the embodiments disclosed herein may be implemented directly using hardware, a software module executed by a processor, or a combination of the two. The software module may be placed in a random access memory (RAM), a memory, a read-only memory (ROM), an electrically programmable ROM, an electrically erasable programmable ROM, a register, a hard disk, a removable disk, a CD-ROM, or any other form of storage medium known in the art.

[0148] The above description of the disclosed embodiments enables those skilled in the art to implement or use the present application. Various modifications to these embodiments will be apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of the present application. Therefore, the present application will not be limited to the embodiments shown herein, but will conform to the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A control method, comprising: Acquire a plurality of target interface images of a target application of the electronic device during operation, wherein the target interface images are image frames including target objects; Obtaining behavior data of the target object in the multiple target interface images; as well as, During the process of the electronic device outputting the target interface image, the target object is controlled to perform a behavior action corresponding to the behavior data in the visual perception area of ​​the target user.

2. The method according to claim 1, further comprising: Obtaining association information between the multiple target interface images, and integrating the multiple target interface images into a target image set based on the association information; as well as, When the electronic device obtains the first trigger operation, multiple target interface images in the target image set are played in sequence according to the target sequence.

3. The method according to claim 1 or 2, wherein: The step of acquiring a plurality of target interface images of a target application of the electronic device during operation includes at least one of the following: Monitoring the running process of the target application, and capturing the application interface presenting the target event as the target interface image; Monitoring the behavior data of the target object in the application interface of the target application, and capturing the application interface where the target object performs the target behavior as the target interface image; Monitoring the operation data of the target user acting on the electronic device, responding to the target operation of the target object, and capturing the current application interface corresponding to the target operation as the target interface image; An application interface of a target application at a target moment during its operation is captured as the target interface image.

4. The method according to claim 1 or 2, wherein: The obtaining of the behavior data of the target object in the plurality of target interface images includes: Segmenting a first foreground image including only the target object from the target interface image; The first foreground image is identified to obtain a first motion trajectory of the target object in the target image set.

5. The method according to claim 2, wherein: Integrating the multiple target interface images into a target image set includes at least one of the following: Directly stitching a plurality of target interface images captured during the operation of the target application into the target image set according to the image capture time; Segmenting a foreground image including only the target object from the multiple target interface images, and generating the target image set based on the foreground image; Segmenting a background image that does not include the target object from the multiple target interface images, and generating the target image set based on the background image; A foreground image including the target object and a background image excluding the target object are segmented from the multiple target interface images, and the target image set is generated based on the foreground image and the background image.

6. The method according to claim 1 or 2, further comprising: Generate a description text that can represent the image content of the target interface image, so as to synchronously output the corresponding description text when the electronic device plays the target interface image.

7. The method according to claim 6, wherein: The generating of the description text capable of representing the image content of the target interface image includes at least one of the following: Performing target generation processing on the target interface image using a target generation model to obtain a description text corresponding to the target interface image; The target interface image and the annotation information for the target interface image are input into a target generation model to generate a description text corresponding to the target interface image.

8. The method according to claim 1, wherein: The controlling the target object to perform a behavior action corresponding to the behavior data in the visual perception area of ​​the target user includes: The configuration information of the target object in the target interface image is obtained, and the motion parameters of the target object are determined based on the configuration information to control the target object to perform a behavior action corresponding to the first motion trajectory based on the motion parameters.

9. The method according to claim 1, further comprising: In response to the interactive operation between the target user and the target object in the visual perception area, the target object is controlled to perform a behavioral action corresponding to the second motion trajectory, and / or the electronic device is controlled to switch from playing the current image frame to playing the target image frame.

10. A control device comprising: A first acquisition unit is used to acquire a plurality of target interface images of a target application of the electronic device during operation, wherein the target interface images are image frames including target objects; A second acquisition unit, used for acquiring behavior data of the target object in the plurality of target interface images; as well as, A control unit is used to control the target object to perform a behavior action corresponding to the behavior data in the visual perception area of ​​the target user during the process of the electronic device outputting the target interface image.

Images