Data processing method and device, equipment and storage medium

Abstract

The embodiment of the invention at least provides a data processing method and device, equipment and a storage medium. The data processing method comprises the steps of outputting at least one frame of target image in response to a target instruction; the target image is a video frame which is intercepted by the electronic equipment from a video output when the target application runs and has a target feature; outputting a target identifier while outputting at least one frame of target image; the target identifier is used for identifying a target feature in the frame of target image; and obtaining a target input for the target identifier, wherein the target input influences the number of the determined at least one frame of target image.

Description

Technical Field

[0001] The present application relates to but is not limited to the field of image processing technology, and in particular to a data processing method, device, equipment and storage medium. Background Art

[0002] When playing games, highlight moments have different definitions for different users. If the same definition is used, the highlight video output to each user for the same game screen will be the same, and it will be impossible to automatically output highlight videos that meet the needs of different users. Summary of the invention

[0003] In view of this, embodiments of the present application at least provide a data processing method, apparatus, device, and storage medium.

[0004] The technical solution of the embodiment of the present application is implemented as follows:

[0005] In a first aspect, an embodiment of the present application provides a data processing method, including: in response to a target instruction, outputting at least one frame of target image; the target image is a video frame with target features captured by the electronic device from a video output when a target application is running; while outputting at least one frame of target image, a target identifier is output; the target identifier is used to identify the target features in the frame of target image; and obtaining a target input for the target identifier, the target input affecting the number of at least one frame of target image determined.

[0006] In a second aspect, an embodiment of the present application provides a data processing device, comprising: a first output module, for outputting at least one frame of target image in response to a target instruction; the target image is a video frame with target features captured by the electronic device from a video output when a target application is running; a second output module, for outputting a target identifier while outputting at least one frame of target image; the target identifier is used to identify the target features in the frame of target image; an acquisition module, for obtaining a target input for the target identifier, the target input affecting the output of the target image that has not been output.

[0007] In a third aspect, an embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be executed on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0008] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium having a computer program stored thereon, which implements some or all of the steps in the above method when executed by a processor.

[0009] In a fifth aspect, an embodiment of the present application provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements some or all of the steps in the above method.

[0010] It should be understood that the above general description and the following detailed description are merely exemplary and explanatory, and are not intended to limit the technical solutions of the present application. BRIEF DESCRIPTION OF THE DRAWINGS

[0011] The drawings herein are incorporated into the specification and constitute a part of the specification. These drawings illustrate embodiments consistent with the present application and are used together with the specification to illustrate the technical solution of the present application.

[0012] Figure 1 A schematic diagram of an implementation flow of a data processing method provided in an embodiment of the present application;

[0013] Figure 2 A schematic diagram of an implementation flow of a data processing method provided in an embodiment of the present application;

[0014] Figure 3 A schematic diagram of an implementation flow of a data processing method provided in an embodiment of the present application;

[0015] Figure 4 A schematic diagram of an implementation flow of a data processing method provided in an embodiment of the present application;

[0016] Figure 5 A schematic diagram of an implementation flow of a data labeling method provided in an embodiment of the present application;

[0017] Figure 6 A schematic diagram of a highlight moment video after inference provided in an embodiment of the present application;

[0018] Figure 7 A schematic diagram of the identification type selection provided in the embodiment of the present application;

[0019] Figure 8 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application;

[0020] Fig. 9 A hardware entity schematic diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0021] In order to make the purpose, technical solutions and advantages of the present application clearer, the technical solutions of the present application are further elaborated in detail below in conjunction with the drawings and embodiments. The described embodiments should not be regarded as limiting the present application. All other embodiments obtained by ordinary technicians in the field without making creative work are within the scope of protection of the present application.

[0022] In the following description, reference is made to "some embodiments", which describe a subset of all possible embodiments, but it is understood that "some embodiments" may be the same subset or different subsets of all possible embodiments, and may be combined with each other without conflict. The terms "first / second / third" involved are merely used to distinguish similar objects and do not represent a specific order for the objects. It is understood that "first / second / third" may be interchanged in a specific order or sequence where permitted, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein.

[0023] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing this application and are not intended to limit this application.

[0024] When playing games, highlight moments have different definitions for different users. If the same definition is used, the highlight video output to each user for the same game screen will be the same, and it will be impossible to automatically output highlight videos that meet the needs of different users.

[0025] The present application embodiment provides a data processing method, which can be executed by a processor of an electronic device. The electronic device may refer to a server, a laptop, a tablet computer, a desktop computer, a smart TV, a set-top box, a mobile device (such as a mobile phone, a portable video player, a personal digital assistant, a dedicated messaging device, a portable gaming device), or other device with data processing capabilities.

[0026] Figure 1 A schematic diagram of the implementation flow of a data processing method provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the method includes the following steps S101 to S103, combining Figure 1 The steps shown are explained.

[0027] Step S101: In response to a target instruction, output at least one frame of target image.

[0028] The target image is a video frame with target features captured by the electronic device from a video output when the target application is running.

[0029] In some embodiments, the target instruction may be an instruction input by a user based on a mouse, keyboard, voice control, gesture, etc. to play a video output when the target application is running.

[0030] In some embodiments, the target application may be a game application, an office application, etc.; it is understandable that the target instruction may be an instruction for playing a game video during the running process of the game application.

[0031] In some embodiments, it may also be a target instruction for a video of a target application imported by a user; illustratively, a user imports a pre-recorded game application video into an electronic device, and a play instruction for the game application video is issued.

[0032] In some embodiments, the target feature represents the highlight operation of the user in the process of operating the target application; for example, if the target application is a game application, the target feature can be the highlight operation generated by the user during the game, such as the user defeating the enemy, the user causing multiple kills, etc.

[0033] In some embodiments, it can be understood that in response to a playback instruction for a video output when a target application is running, the output video is intercepted based on preset interception rules to obtain at least one frame of a target image carrying target features; wherein the preset interception rules may be intercepting the output video at a preset time point, intercepting according to a preset number of frames, etc.

[0034] For example, the target application is a game application, and the target feature is a user winning. In response to a play instruction input by a user through a mouse, keyboard, voice control, gesture, etc. for a video output when the target application is running, the output video is intercepted based on a preset interception rule to obtain at least one frame of a target image with the user winning.

[0035] In other embodiments, in response to a target instruction, at least one frame of a target image is output; it is understandable that in response to a playback instruction for a video output when a target application is running, each frame of the video is recognized based on an initial recognition model to obtain at least one frame of a target image carrying target features.

[0036] In some embodiments, the initial recognition model can be obtained by training based on the basic rules of the game; or it can be obtained by training based on game videos of other users; wherein the initial recognition model obtained by training based on other users can be obtained based on the following methods:

[0037] Acquire at least one training sample; each training sample includes at least one game video and a labeled video corresponding to each game video; input at least one game video in each training sample and a labeled video corresponding to each game video into the initial recognition model to be trained to obtain a predicted image; the predicted image includes a video frame with highlight moments in each game video; adjust the model parameters of the initial recognition model to be trained based on the loss value between the predicted image and the labeled video until the convergence condition is met, and output the trained initial recognition model.

[0038] In some embodiments, the video outputted when the target application is running is recognized based on the trained initial recognition model to obtain at least one frame of target image including target features.

[0039] For example, the target application is a game application, the target features are winning and causing multiple kills, and two frames of target images are output as an example to illustrate the embodiment of the present application. The game video is recognized based on the trained initial recognition model to obtain two frames of target images, wherein the target feature in the first frame of the target image is that the user has caused multiple kills, and the target feature in the second frame of the target image is that the user has won.

[0040] Step S102: Outputting a target identification while outputting at least one frame of target image.

[0041] The target identifier is used to identify the target feature in the target image of the frame.

[0042] In some embodiments, while outputting at least one frame of the target image through the above-mentioned preset interception rule or initial recognition model, a target identification for the target feature in the target image is output.

[0043] In some embodiments, there is also indication information between the target identifier and the target feature, which is used to indicate the association relationship between the target identifier and the target feature; illustratively, the indication information may be an arrow connecting the target identifier and the target feature.

[0044] In some embodiments, the target identification is generated for the target feature in each frame of the target image, and is used to identify the position of the target feature in the target image and provide the user with a selection mark of whether the target feature is accurate.

[0045] Step S103: Obtain target input for the target identifier.

[0046] The target input affects the number of at least one frame of target images determined.

[0047] In some embodiments, the target input may be a selection operation, a click operation, etc., by a user on a target identifier through a mouse, keyboard, voice control, gesture, etc.

[0048] In some embodiments, the target input is an input for target identification by a user based on target features in a target image; illustratively, the target input is a selection operation for target identification based on whether the target features are accurate.

[0049] In some embodiments, the target input is used to affect the number of target images that are not output; it is understandable that the user's selection input based on the target feature that characterizes the target feature of the target identifier meets the user's browsing needs, or the selection input based on the target feature that characterizes the target feature of the target identifier does not meet the user's browsing needs, adjusts the preset interception rules for the video output when the target application is running, thereby affecting the number of at least one frame of target images output; wherein, based on the user's selection input based on the target feature that characterizes the target feature of the target identifier meets the user's browsing needs, the preset interception rules for the video output when the target application is running are adjusted, thereby increasing the number of at least one frame of target images output; based on the user's selection input based on the target feature that characterizes the target feature of the target identifier does not meet the user's browsing needs, the preset interception rules for the video output when the target application is running are adjusted, thereby reducing the number of at least one frame of target images output.

[0050] Exemplarily, the embodiment of the present application is described by taking the target instruction as a play instruction, the target application as a game application, the target feature as a game highlight moment, and the target identification as a correct identification and an incorrect identification as an example. In response to the user's play instruction for the game video when the game application is used through a mouse, keyboard, voice control, gesture, etc., the game video is intercepted based on a preset interception rule to obtain at least one frame of a target image with a user winning, a user causing a kill, etc.; at the same time, a confirmation identification or an incorrect identification for the user winning, the user causing a kill, etc. in each target image is output; based on the user's selection input based on the correct identification based on the winning or the user causing the kill in the target image, the number of at least one frame obtained through the preset interception rule increases; based on the user's selection input based on the incorrect identification based on the winning or the user causing the kill in the target image, the number of at least one frame obtained through the preset interception rule decreases.

[0051] In other embodiments, the user's selection input for target identification based on whether the target features in the output target image are accurate is used as a training sample to train the above-mentioned initial recognition model, thereby affecting the number of at least one frame of target images output by the initial recognition model.

[0052] Exemplarily, the embodiment of the present application is described by taking the target instruction as a play instruction, the target application as a game application, the target feature as a game highlight moment, and the target identifier as a correct identifier and an incorrect identifier as an example. In response to the play instruction of the game video when the game application is used, at least one frame of the target image including the game highlight moment and the correct identifier and the incorrect identifier for each target image is obtained based on the initial recognition model; the user's target input for the correct identifier and the incorrect identifier based on the highlight moment is obtained; wherein, if the target input is the user selecting the correct identifier, it represents that the game highlight moment is accurate, that is, it represents that the target image corresponding to the game highlight moment is accurate; if the target input is the user selecting the incorrect identifier, it represents that the game highlight moment is inaccurate, that is, it represents that the target image corresponding to the game highlight moment is inaccurate; the accurate target image and the corresponding target input or the inaccurate target image and the corresponding target input are used as samples to train the above-mentioned initial recognition model, thereby affecting the number of target images output by the initial recognition model.

[0053] In an embodiment of the present application, in response to a video output when a target application is played, the video is intercepted by using preset interception rules to obtain at least one frame including a user's highlight moment and a corresponding target identifier; so that the user can determine whether the highlight moment in the target image meets the user's browsing needs based on the target identifier; based on the user's target input for the target identifier based on whether the highlight moment in the target image meets the user's browsing needs, the number of at least one frame of the target image output based on the preset interception rules is affected.

[0054] In some embodiments, the above method further comprises:

[0055] When determining at least one frame of target images, if the target input for the target identifier satisfies a first condition, the target input increases the number of images with target features in the determined at least one frame of target images.

[0056] The first condition indicates that the target feature in at least one frame of the target image is a feature that meets the browsing needs of the target user.

[0057] In some embodiments, the characteristics of the target user's browsing needs may be the user's highlight operations or erroneous operations when using or operating the target application; for example, taking a game application as an example; the characteristics of the target user's browsing needs may be the user's victory or multi-killing; taking an office application as an example, the characteristics of the target user's browsing needs may be user operation errors, etc.

[0058] In some embodiments, if the target identification represents a correct identification of a target feature that meets the target user's browsing requirements and an incorrect identification of a feature that does not meet the target user's browsing requirements, the first condition for target input is that the user selects the correct identification.

[0059] In some embodiments, it can be understood that if the target features in the output target image meet the browsing needs of the target user and the target input of the user for the target identification meets the first condition; the target image including the target features that meet the browsing needs of the target user and the target input that meets the first condition are used as samples to train the above-mentioned initial recognition model to obtain the trained initial recognition model, and the number of target images output by the trained initial recognition model for the video output when the target application is running is greater than or equal to the number of target images output by the initial recognition model before training.

[0060] Exemplarily, the embodiment of the present application is described by taking the target application as a game application, the target feature as a game highlight moment, and the target identifier as a correct identifier and an incorrect identifier as examples. If the target input is a correct identifier selected by the user, it indicates that the game highlight moment meets the browsing needs of the target user, that is, it indicates that the target image corresponding to the game highlight moment meets the browsing needs of the target user; the target image corresponding to the game highlight moment that meets the browsing needs of the target user and the corresponding target input that meets the browsing needs of the target user are used as samples to train the above-mentioned initial recognition model to obtain a trained initial recognition model, and the number of target images output by the trained initial recognition model for the target game video is greater than or equal to the number of target images output by the initial recognition model before training.

[0061] When determining at least one frame of target images, if the target input for the target identifier does not satisfy the first condition, the target input reduces the number of images with target features in the determined at least one frame of target images.

[0062] In some embodiments, it can be understood that if the target features in the output target image do not meet the target user's browsing requirements and thus the user's target input for the target identification does not meet the first condition; the target image including the target features that do not meet the target user's browsing requirements and the target input that does not meet the first condition are used as samples to train the above-mentioned initial recognition model to obtain the trained initial recognition model, and the trained initial recognition model outputs target images for the video output when the target application is running, which is less than the number of target images output by the initial recognition model before training.

[0063] Exemplarily, the embodiment of the present application is described by taking the target application as a game application, the target feature as a game highlight moment, and the target identifier as a correct identifier and an incorrect identifier as examples. If the target input is a wrong identifier selected by the user, it indicates that the game highlight moment does not meet the browsing needs of the target user, that is, it indicates that the target image corresponding to the game highlight moment does not meet the browsing needs of the target user; the target image corresponding to the game highlight moment that does not meet the browsing needs of the target user and the corresponding target input that does not meet the browsing needs of the target user are used as samples to train the above-mentioned initial recognition model to obtain a trained initial recognition model, and the number of target images output by the trained initial recognition model for the target game video is less than the number of target images output by the initial recognition model before training.

[0064] In an embodiment of the present application, the above-mentioned initial recognition model is trained based on whether the target input corresponding to the target feature of the user meets the browsing needs of the target user as a sample. If the target input corresponding to the target feature meets the browsing needs of the target user, the number of output target images of the initial recognition model after training increases; if the target input corresponding to the target feature does not meet the browsing needs of the target user, the number of output target images of the initial recognition model after training decreases.

[0065] In some embodiments, the target identifier includes a first identifier and a second identifier.

[0066] If the target input indicates that the target user has selected the first identifier, it is determined that the target input satisfies the first condition.

[0067] In some embodiments, the first condition represents that the target feature in at least one frame of the target image is a feature that meets the browsing needs of the target user.

[0068] In some embodiments, if the user's target input is to select the first identifier, it is characterized that the target feature in the target image meets the target user's browsing needs, thereby determining that the target input meets the first condition.

[0069] Exemplarily, the first mark is a confirmation mark, and the target feature is that the user wins; when the user selects the confirmation mark in the target image with a mouse, it indicates that the user considers the victory to be a highlight moment.

[0070] If the target input indicates that the target user has selected the second identifier, it is determined that the target input does not satisfy the first condition.

[0071] In some embodiments, if the user's target input is to select the second identifier, the target feature portion in the target image satisfies the target user's browsing requirements, thereby determining that the target input does not satisfy the first condition.

[0072] Exemplarily, the second identifier is an error identifier, and the target feature is that the user caused the kill; the user selects the error identifier in the target image with the mouse, which indicates that the kill was not a highlight moment that the user considers.

[0073] In the embodiment of the present application, through the user's selection of the first identifier and the second identifier based on the target feature, it is determined whether the target input meets the first condition to determine whether the target feature meets the user's browsing needs.

[0074] Figure 2 A schematic diagram of an implementation flow of a data processing method provided in an embodiment of the present application, which method can be executed by a processor of an electronic device. Figure 1 Steps S101 and S102 in the above example can be updated to step S201, combining Figure 2 The steps shown are explained.

[0075] Step S201: In response to a target instruction, a first model in the electronic device outputs at least one frame of a target image with a target identifier.

[0076] The target identifier is part of the pixels in at least one frame of the target image, and the first model is used to identify the target image from the input image and add the target identifier to the target image.

[0077] In some embodiments, the first model is a trained image recognition model, which may include a time series model (such as an autoregressive integrated moving average model, a generalized autoregressive conditional heteroskedasticity model, etc.), a machine learning model (such as a vector machine model, a random forest model, etc.), a deep learning model (convolutional neural network model, a recurrent neural network model, a self-attention mechanism neural network model), etc.

[0078] In some embodiments, the target identifier may be pixel points corresponding to text, graphics, etc. in the target image; illustratively, the target identifier includes text of a correct identifier and an incorrect identifier, and the target identifier may be pixel points corresponding to the correct identifier and the incorrect identifier in the target image.

[0079] In some embodiments, the input image may be a video or image output by the electronic device when the target application is running; the first model recognizes target features in the video or image output when the target application is running to obtain a target image, and adds a target identifier to the target features in the target image.

[0080] Exemplarily, in response to playing a video or image output by an electronic device when a game application is running, a first model identifies the video or image to obtain a target image having a highlight moment, and adds correctly identified and incorrectly identified pixel points to the highlight moment in the target image.

[0081] In some embodiments, after obtaining the target input for the target identifier, the method further includes:

[0082] The first model is adjusted based on the target input, and different target inputs result in different numbers of target images with target features output by the first model.

[0083] In some embodiments, the target input can characterize whether the target features in the target image meet the user's browsing needs or whether the target features in the target image do not meet the user's browsing needs; based on the target input characterizing that the target features in the target image meet the user's browsing needs, the first model is adjusted to increase the number of target images with target features output by the first model; based on the target input characterizing that the target features in the target image do not meet the user's browsing needs, the first model is adjusted to increase the number of target images with target features output by the first model.

[0084] In an embodiment of the present application, a video output when a target application is running is identified based on an image recognition model to obtain a target image having target features and a corresponding target identifier; based on the user's target input for the target identifier, the first model is adjusted to adjust the number of target images having target features output by the first model.

[0085] Figure 3 The present invention provides a method for processing a quantity according to an embodiment of the present invention. The method can be executed by a processor of an electronic device. The method includes steps S301 and S302. Figure 3 The steps shown are explained.

[0086] Step S301: Generate a first video based on the output of the first model.

[0087] In some embodiments, the output of the first model includes at least one frame of target image carrying target features and target identification.

[0088] In some embodiments, it can be understood that at least one frame of target image is synthesized based on video synthesis software to obtain a first video; wherein each video frame carried in the first video carries the target feature.

[0089] Step S302: output the first video so that a target identifier is outputted while outputting at least one frame of the target image.

[0090] In some embodiments, it is understandable that while showing the first video to the user, the video frame corresponding to the target image in the first video and the target identifier corresponding to the target feature in the target image are shown to the user; the user inputs the target identifier based on the target feature to determine whether the target feature in the target image output by the first model meets the user's browsing needs.

[0091] In some embodiments, in the process of displaying the first video to the user, in response to the user manually dragging the video bar of the first video or the first model automatically dragging the video bar of the first video to display a frame of target image, a target identification for the target features in the frame of target image is generated to provide the user with an input to the first model based on the target identification to determine whether the target features in the target image meet the user's browsing needs.

[0092] Exemplarily, the first video is obtained by intercepting a game video by the first model. When the first model generates the first video and displays it to the user, in response to the user manually dragging the video bar of the first video to display the target image including the kill caused by the user, a correct mark and an incorrect mark for the kill caused by the user are generated. If the user selects the correct mark, the highlight moment representing the kill caused by the user in the target image of the frame meets the user's browsing needs, that is, the user believes that the kill is a highlight moment.

[0093] In some embodiments, based on the user's target input for target identification, the first model is adjusted to adjust the number of target images having target features, thereby adjusting the number of target images in the first video.

[0094] Exemplarily, the output of the first model includes at least one frame of target image of the user achieving victory and causing a kill, as well as correct identification and incorrect identification for achieving victory and causing a kill; based on video synthesis software, the at least one frame of target image is synthesized into a first video; wherein, in the process of browsing the first video, the user can choose between correct identification and incorrect identification, and if the user's current video frame has an incorrect identification, it indicates that achieving victory or causing a kill in the current video frame does not meet the user's browsing needs, that is, the user does not consider achieving victory or causing a kill to be a highlight moment; if the user's current video frame has a correct identification, it indicates that achieving victory or causing a kill in the current video frame meets the user's browsing needs, that is, the user considers achieving victory or causing a kill to be a highlight moment.

[0095] In an embodiment of the present application, while synthesizing at least one frame of target image with target features output by the first model to obtain the first video, a target identifier for the target feature is output to determine whether the target feature meets the user's browsing needs based on the user's target input for the target identifier, thereby improving the user experience.

[0096] Figure 4 The following is a schematic diagram of a data processing method provided in an embodiment of the present application. The method can be executed by a processor of an electronic device. The method includes steps S401 and S402. Figure 4 The steps shown are explained.

[0097] Step S401: determine multiple frames of target images based on the output of the second model, and generate a second video based on the multiple frames of target images.

[0098] The second model is used to identify the target feature type from the input image.

[0099] In some embodiments, the second model is used to identify the types of target features in the video output when the target application is running, that is, to determine which are highlight moment images.

[0100] In some embodiments, the output of the second model includes a category of each target feature in the video output by the target application when it is running.

[0101] Exemplarily, the second model identifies the game video output when the game application is running to obtain the category of the highlight moment, such as the category of the highlight moment when the user wins, and the category of the highlight moment when the user causes a kill.

[0102] In some embodiments, each frame of the target image in the second video also includes a category of highlight moments; and the second video is synthesized using multiple frames of the target image based on video synthesis software.

[0103] Step S402: adding a target identifier to the image frame of the second video so that the target identifier is outputted while the second video is outputted.

[0104] In some embodiments, it is understandable that while showing the second video to the user, the video frame corresponding to the target image in the second video and the target identifier corresponding to the target feature in the target image are shown to the user; the user inputs the target identifier based on the target feature and the category of the target feature to determine whether the target feature in the target image output by the first model meets the user's browsing needs and whether the category of the target feature is accurate.

[0105] Exemplarily, the output of the second model includes highlight moments categorized as user victory and kills, determining multiple frames of target images with highlight moments of user victory and kills based on the highlight moments categorized as user victory and kills, and having video synthesis software generate a second video as well as correct and incorrect identification of highlight moments in the target image.

[0106] In some embodiments, after obtaining the target input for the target identifier, the method further includes:

[0107] The second model is adjusted based on the target input, and different target inputs result in different numbers of target images determined based on the output of the second model.

[0108] In some embodiments, based on a user's target input for target identification, the number of target images having target features determined based on the output of the second model is adjusted.

[0109] Exemplarily, if the user selects the correct identification based on the highlight moment of achieving victory or causing a kill in the target image, the number of highlight moment categories output by the second model is adjusted to increase, thereby increasing the number of target images determined based on the highlight moment categories output by the second model; if the user selects the wrong identification based on the highlight moment of achieving victory or causing a kill in the target image, the number of highlight moment categories output by the second model is adjusted to decrease, thereby decreasing the number of target images determined based on the highlight moment categories output by the second model.

[0110] In an embodiment of the present application, multiple frames of target images are determined based on the target feature category output by the second model; based on the user's target input for the target identification, the output of the second model is adjusted to adjust the number of target images with the target features.

[0111] In some embodiments, the above-mentioned target input-based adjustment model also includes the following implementation methods:

[0112] If the target input satisfies the first condition, the target feature position and the target feature type in the corresponding target image are determined, and the model is adjusted based on the target image, the target feature position and the target feature type, so that the number of target images determined based on the model increases.

[0113] In some embodiments, the first condition represents that the target feature in at least one frame of the target image meets the browsing requirements of the target user.

[0114] In some embodiments, it can be understood that if the target identifier includes a correct identifier and an incorrect identifier, the target input meeting the first condition indicates that the user has selected the correct identifier.

[0115] In some embodiments, when the target input satisfies the first condition, the positions of target features in the target image and the categories of each target feature in the target image are determined; the first model and the second model are trained based on the target image, the positions of the target features in the target image, and the categories of the target features to adjust the parameters of the first model and the second model so that the number of categories of target features output by the trained second model increases; and the number of target images output by the trained first model increases.

[0116] Exemplarily, if the user selects the correct identification based on the highlight moment in the current video frame, it indicates that the user recognizes the highlight moment; determine the type of the corresponding highlight moment and the position in the corresponding video frame; based on the video frame, the type of the corresponding highlight moment and the position in the corresponding video frame; train the above-mentioned first model and the second model to modify the model parameters of the first model and the second model so that the number of highlight moment videos output by the model increases.

[0117] If the target input does not satisfy the first condition, the model is adjusted based on the corresponding target image so that the number of target images determined based on the model is reduced.

[0118] In some embodiments, it can be understood that if the target identifier includes a correct identifier and an incorrect identifier, the target input not satisfying the first condition indicates that the user has selected an incorrect identifier.

[0119] In some embodiments, when the target input does not meet the first condition, the first model is trained based on the target images corresponding to the target features that do not meet the user's browsing requirements to adjust the parameters of the first model so that the number of target images that do not meet the user's browsing requirements output by the trained first model is reduced.

[0120] Exemplarily, if the user selects an incorrect label based on a highlight moment in the current video frame, it indicates that the user does not recognize the highlight moment; based on the video frame corresponding to the highlight moment that the user does not recognize, the parameters of the first model are modified so that the number of video frames of the highlight moment output by the first model is reduced.

[0121] In an embodiment of the present application, based on whether the user's input for the target identification satisfies the first condition, the first model and the second model are adjusted according to the target image, the category of the target feature, and the position of the target feature in the target image, thereby adjusting the number of target images determined to have the target feature.

[0122] An exemplary application of a data processing method provided in an embodiment of the present application in a practical scenario is described below.

[0123] Users have the need for model personalization, but they may lack relevant knowledge of AI and cannot train personal models. In addition, we may have missed recognition and misidentification in AI-generated game videos. In our video recognition of game highlights, there will be images with low recognition rates that are not recognized as highlights. This will miss some highlights. Recognition situations may manifest as three categories: mislabeling, false negatives, and false positives. Mislabeling means that an error has occurred when labeling an image, false negatives mean that an image that should have been judged as a positive sample is mistakenly judged as a negative sample, and false positives mean that an image that should have been judged as a negative sample is mistakenly judged as a positive sample.

[0124] In related technologies, data labeling is performed in laboratories or companies by data labelers with relevant knowledge background and experience in data labeling. It is necessary to determine what to label and how to label it. For example, the position of the label in the image, the size of the label and the size of the image, etc. Based on this, the AI ​​model is trained.

[0125] In an embodiment of the present application, a labeling prompt is given based on an existing model, so that the user can complete the labeling based on the prompt. And the user can use the built-in training system of this solution to complete the training without the need for relevant knowledge of training. This solution proposes a model training method based on user labeling to train the user's personal model. Thus, a training channel for the user's personal model is established, allowing the user to complete the three stages of data labeling, training and evaluation of the personal model through graphical simple operations without the relevant knowledge background of training. Among them, in the data labeling stage, the user does not have relevant labeling background knowledge and cannot provide more data labeling samples to improve the recognition accuracy. This solution will automatically generate labeled data based on the existing model, allowing the user to select the labeled data according to the prompt. The embodiment of the present application uses video import and reasoning to provide a labeling form so that the user can easily complete data labeling.

[0126] Figure 5 The following is a schematic diagram of a data labeling method provided in an embodiment of the present application. The method can be executed by a processor of an electronic device. The method includes steps S501 to S505. Figure 5 The steps shown are explained.

[0127] Step S501: video capture.

[0128] In some embodiments, the user turns on the video capture function and the highlight moment video generation function. The software (corresponding to the first model and the second model in the above embodiment) will automatically capture the game video and infer the highlight moment video while the user is playing the game.

[0129] Step S502: Add imported video.

[0130] In some embodiments, the software will import videos captured by the user using the software by default. It also supports the video import function and the reasoning function based on the imported video.

[0131] Step S503: infer the current video to obtain the highlight moment video of the current video.

[0132] In some embodiments, the software identifies video frames with highlight moments in an imported video and generates a highlight moment video based on the video frames with highlight moments. Figure 6A schematic diagram of a highlight moment video after inference provided in an embodiment of the present application, wherein 601 is an identified highlight moment that causes a kill for the user.

[0133] Step S504: drag the video slider.

[0134] In some embodiments, the user may manually drag the video bar or drag the video bar based on a preset speed to show the user the highlight moment in the highlight moment video. Figure 6 , where 602 is the dragging position, that is, the highlight moment video frame currently being displayed to the user.

[0135] Step S505: Select identification type.

[0136] In some embodiments, while generating the highlight moment video as described above, an identifier for the highlight moment is generated, including a correct identifier and an incorrect identifier. If the user selects the correct identifier based on the highlight moment in the current video frame, it indicates that the user recognizes the highlight moment; and the video frame corresponding to the highlight moment is stored in an image folder of the electronic device, and the type of the highlight moment and the position in the corresponding video frame are stored in a tag folder of the electronic device; if the user selects an incorrect identifier based on the highlight moment in the current video frame, it indicates that the user does not recognize the highlight moment; and the video frame corresponding to the highlight moment is stored in the image folder of the electronic device, and the type of the highlight moment corresponding to the incorrect identifier and the position in the corresponding video frame are not stored.

[0137] For example, Figure 7 Schematic diagram of the selection of identification types provided in the embodiment of the present application; wherein 701 is the highlight moment and the position of the highlight moment in the video frame, 702 is the identification of the highlight moment, including correct identification and incorrect identification, according to Figure 7 It can be seen that if the user selects the correct identification, the type of the highlight moment and the position of the highlight moment in the video frame are stored in the label folder of the electronic device; the video frame corresponding to the highlight moment is stored in the image folder of the electronic device.

[0138] In some embodiments, during the recognition process according to the software (corresponding to the first model and the second model in the above embodiments), if a video that should be recognized as a highlight moment is not accurately recognized, the threshold for the software to recognize the highlight moment will be adjusted; for example, when scoring an image, an image must score 80 points to be determined as a highlight moment. If, as a result, a video that should be recognized as a highlight moment is not accurately recognized, the recognition score will be reduced to 75 points. If the image scores 75 points, it is determined that a highlight moment exists to generate more annotation information.

[0139] In some embodiments, the type of the highlight moment and the position in the corresponding video frame are stored in a label folder of the electronic device; if the user selects an incorrect label based on the highlight moment in the current video frame, it indicates that the user does not recognize the highlight moment; and the video frame corresponding to the highlight moment is stored in an image folder of the electronic device, and the type of the highlight moment corresponding to the incorrect label and the position in the corresponding video frame are not stored; based on the data in the label folder and the image folder, the above-mentioned software (corresponding to the above-mentioned first model and second model) is trained, and based on the data marked by the user in the label folder and the image folder, the parameters of the above-mentioned software are modified to adjust the number of highlight moment videos output by the software; wherein, based on the user selecting the correctly marked data, the number of highlight moment videos output by the software is increased; based on the user selecting the incorrectly marked data, the number of highlight moment videos output by the software is reduced.

[0140] In some embodiments, in order to prevent the model running in the software from overfitting, the model running in the software is uploaded to the server for evaluation using existing data, and the evaluation results are provided to the user. If the evaluation results are not ideal, the user is allowed to decide whether to apply the model running in the software.

[0141] In an embodiment of the present application, the user's highlight moment video and corresponding identification are output through the existing image recognition model, and the existing image recognition model is trained based on the correct identification and incorrect identification selected by the user to obtain an image recognition model that meets the user, thereby making the user's personal highlight moment video more accurate.

[0142] Based on the foregoing embodiments, an embodiment of the present application provides a data processing device, which includes the units included and the modules included in the units, which can be implemented by a processor in an electronic device; of course, it can also be implemented by a specific logic circuit; in the implementation process, the processor can be a central processing unit (CPU), a microprocessor (MPU), a digital signal processor (DSP) or a field programmable gate array (FPGA), etc.

[0143] Figure 8 A schematic diagram of the structure of a data processing device provided in an embodiment of the present application is shown in FIG. Figure 8As shown, the data processing device 800 includes: a first output module 801, a second output module 802, and an acquisition module 803, wherein: the first output module 801 is used to output at least one frame of target image in response to a target instruction; the target image is a video frame with target features captured by the electronic device from a video output when a target application is running; the second output module 802 is used to output a target identifier while outputting at least one frame of target image; the target identifier is used to identify the target features in the frame of target image; the acquisition module 803 is used to obtain a target input for the target identifier, and the target input affects the output of the target image that has not been output.

[0144] In some embodiments, the data processing device 800 also includes a determination module (not shown in the figure), which is used to determine that when at least one frame of target image is determined, if the target input for the target identifier satisfies a first condition, the target input increases the number of images with target features in the at least one frame of target image determined; the first condition characterizes that the target features in at least one frame of target image are features that meet the browsing needs of the target user; when determining at least one frame of target image, if the target input for the target identifier does not satisfy the first condition, the target input reduces the number of images with target features in the at least one frame of target image determined.

[0145] In some embodiments, the target identifier includes a first identifier and a second identifier; the determination module is further used to determine that the target input satisfies the first condition if the target input indicates that the target user has selected the first identifier; and to determine that the target input does not satisfy the first condition if the target input indicates that the target user has selected the second identifier.

[0146] In some embodiments, the first output module 801 and the second output module 802 are also used to respond to a target instruction, and the first model in the electronic device outputs at least one frame of target image with a target identifier; the target identifier is a portion of pixels in at least one frame of target image, and the first model is used to identify the target image from the input image and add the target identifier to the target image; after obtaining the target input for the target identifier, it also includes: adjusting the first model based on the target input, and different target inputs make the first model output different numbers of target images with target features.

[0147] In some embodiments, the second output module 802 is further used to generate a first video based on the output of the first model; output the first video so that a target identifier is output while outputting at least one frame of the target image.

[0148] In some embodiments, the second output module 802 is also used to determine multiple frames of target images based on the output of the second model, and generate a second video based on the multiple frames of target images; the second model is used to identify target feature categories from the input image; add a target identifier to the image frame of the second video, so that the target identifier is output while the second video is output; after obtaining the target input for the target identifier, it also includes: adjusting the second model based on the target input, and different target inputs result in different numbers of target images determined based on the output of the second model.

[0149] In some embodiments, the determination module is also used to determine the target feature position and target feature type in the corresponding target image if the target input satisfies a first condition, and adjust the model based on the target image, target feature position and target feature type so that the number of target images determined based on the model increases; if the target input does not satisfy the first condition, adjust the model based on the corresponding target image so that the number of target images determined based on the model decreases.

[0150] The description of the above device embodiment is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. In some embodiments, the functions or modules included in the device provided in the embodiment of the present application can be used to execute the method described in the above method embodiment. For technical details not disclosed in the device embodiment of the present application, please refer to the description of the method embodiment of the present application for understanding.

[0151] It should be noted that in the embodiment of the present application, if the above method is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium, including a number of instructions to enable an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a U disk, a mobile hard disk, a read-only memory (ROM), a disk or an optical disk. In this way, the embodiment of the present application is not limited to any specific hardware, software or firmware, or any combination of hardware, software, and firmware.

[0152] An embodiment of the present application provides an electronic device, including a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, some or all of the steps in the above method are implemented.

[0153] The embodiment of the present application provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by a processor, some or all of the steps in the above method are implemented. The computer-readable storage medium can be transient or non-transient.

[0154] An embodiment of the present application provides a computer program, including a computer-readable code. When the computer-readable code is run in an electronic device, a processor in the electronic device executes some or all of the steps for implementing the above method.

[0155] The embodiment of the present application provides a computer program product, which includes a non-transitory computer-readable storage medium storing a computer program, and when the computer program is read and executed by a computer, some or all of the steps in the above method are implemented. The computer program product can be implemented specifically by hardware, software or a combination thereof. In some embodiments, the computer program product is specifically embodied as a computer storage medium, and in other embodiments, the computer program product is specifically embodied as a software product, such as a software development kit (SDK) and the like.

[0156] It should be noted here that the description of the various embodiments above tends to emphasize the differences between the various embodiments, and the same or similar aspects can be referenced to each other. The description of the above device, storage medium, computer program and computer program product embodiments is similar to the description of the above method embodiment, and has similar beneficial effects as the method embodiment. For technical details not disclosed in the embodiments of the device, storage medium, computer program and computer program product of this application, please refer to the description of the method embodiment of this application for understanding.

[0157] Fig. 9 A hardware entity diagram of an electronic device provided in an embodiment of the present application, such as Fig. 9 As shown, the hardware entity of the electronic device 900 includes: a processor 901 and a memory 902, wherein the memory 902 stores a computer program that can be run on the processor 901, and the processor 901 implements the steps in the method of any of the above embodiments when executing the program.

[0158] The memory 902 stores computer programs that can be run on the processor. The memory 902 is configured to store instructions and applications executable by the processor 901. It can also cache data to be processed or processed by the processor 901 and various modules in the electronic device 900 (for example, image data, audio data, voice communication data, and video communication data). This can be achieved through flash memory (FLASH) or random access memory (Random Access Memory, RAM).

[0159] When the processor 901 executes the program, the steps of any of the above methods are implemented. The processor 901 generally controls the overall operation of the electronic device 900.

[0160] An embodiment of the present application provides a computer storage medium, which stores one or more programs. The one or more programs can be executed by one or more processors to implement the steps of the method of any of the above embodiments.

[0161] It should be noted here that the description of the above storage medium and device embodiments is similar to the description of the above method embodiments, and has similar beneficial effects as the method embodiments. For technical details not disclosed in the storage medium and device embodiments of this application, please refer to the description of the method embodiments of this application for understanding.

[0162] The processor may be at least one of an Application Specific Integrated Circuit (ASIC), a Digital Signal Processor (DSP), a Digital Signal Processing Device (DSPD), a Programmable Logic Device (PLD), a Field Programmable Gate Array (FPGA), a Central Processing Unit (CPU), a controller, a microcontroller, and a microprocessor. It is understandable that the electronic device that implements the functions of the processor may also be other, and the embodiments of the present application are not specifically limited.

[0163] The above-mentioned computer storage medium / memory can be a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), a magnetic random access memory (FRAM), a flash memory (Flash Memory), a magnetic surface memory, an optical disk, or a compact disc read-only memory (CD-ROM) and the like; it can also be various terminals including one or any combination of the above-mentioned memories, such as mobile phones, computers, tablet devices, personal digital assistants, etc.

[0164] It should be understood that "one embodiment" or "an embodiment" mentioned throughout the specification means that specific features, structures or characteristics related to the embodiment are included in at least one embodiment of the present application. Therefore, "in one embodiment" or "in an embodiment" appearing throughout the specification does not necessarily refer to the same embodiment. In addition, these specific features, structures or characteristics can be combined in one or more embodiments in any suitable manner. It should be understood that in various embodiments of the present application, the size of the serial number of each step / process mentioned above does not mean the order of execution, and the execution order of each step / process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiment of the present application. The serial numbers of the embodiments of the present application mentioned above are for description only and do not represent the advantages and disadvantages of the embodiments.

[0165] It should be noted that, in this article, the terms "include", "comprises" or any other variations thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, an element defined by the sentence "comprises a ..." does not exclude the existence of other identical elements in the process, method, article or device including the element.

[0166] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. The device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as: multiple units or components can be combined, or can be integrated into another system, or some features can be ignored or not executed. In addition, the coupling, direct coupling, or communication connection between the components shown or discussed can be through some interfaces, and the indirect coupling or communication connection of the devices or units can be electrical, mechanical or other forms.

[0167] The units described above as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units; they may be located in one place or distributed on multiple network units; some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0168] In addition, all functional units in the embodiments of the present application can be integrated into one processing unit, or each unit can be a separate unit, or two or more units can be integrated into one unit; the above integrated unit can be implemented in the form of hardware or in the form of hardware plus software functional units. A person of ordinary skill in the art can understand that all or part of the steps of the above method embodiments can be completed by hardware related to program instructions, and the above program can be stored in a computer-readable storage medium. When the program is executed, the steps of the above method embodiments are executed; and the above storage medium includes: mobile storage devices, read-only memory (ROM), disks or optical disks, etc. Various media that can store program codes.

[0169] Alternatively, if the above-mentioned integrated unit of the present application is implemented in the form of a software function module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application can essentially or in other words, the part that contributes to the relevant technology can be embodied in the form of a software product, which is stored in a storage medium and includes a number of instructions for an electronic device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the methods described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as mobile storage devices, ROMs, magnetic disks, or optical disks.

[0170] The above is only an implementation method of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of changes or substitutions within the technical scope disclosed in the present application, which should be included in the protection scope of the present application.

Claims

1. A data processing method, comprising: In response to the target instruction, output at least one frame of target image; The target image is a video frame with target features captured by the electronic device from a video output when the target application is running; Outputting at least one frame of target image and outputting the target identification at the same time; The target identifier is used to identify the target feature in the target image of the frame; A target input for the target identifier is obtained, where the target input affects the number of at least one frame of the determined target image.

2. The method according to claim 1, further comprising: When determining at least one frame of target image, if the target input for the target identifier satisfies a first condition, the target input increases the number of images with target features in the determined at least one frame of target image; the first condition indicates that the target feature in the at least one frame of target image is a feature that satisfies the browsing needs of the target user; When determining at least one frame of target images, if the target input for the target identifier does not satisfy the first condition, the target input reduces the number of images with target features in the determined at least one frame of target images.

3. The method according to claim 2, wherein the target identifier includes a first identifier and a second identifier, and further comprising: If the target input indicates that the target user has selected the first identifier, determining that the target input satisfies the first condition; If the target input indicates that the target user has selected the second identifier, it is determined that the target input does not satisfy the first condition.

4. The method according to claim 1, wherein in response to the target instruction, outputting at least one frame of the target image, and outputting the target identification while outputting at least one frame of the target image, comprises: In response to the target instruction, the first model in the electronic device outputs at least one frame of target image with a target identifier; The target identifier is a portion of pixels in at least one frame of a target image, and the first model is used to identify the target image from the input image and add the target identifier to the target image; After obtaining the target input for the target identifier, the method further includes: The first model is adjusted based on the target input, and different target inputs result in different numbers of target images with target features output by the first model.

5. The method according to claim 4, further comprising: generating a first video based on an output of the first model; The first video is outputted so that the target identifier is outputted while at least one frame of the target image is outputted.

6. The method according to claim 1, before outputting at least one frame of target image in response to the target instruction, further comprising: Determine a plurality of target image frames based on the output of the second model, and generate a second video based on the plurality of target image frames; The second model is used to identify the target feature category from the input image; Adding a target identifier to an image frame of the second video so that the target identifier is outputted while the second video is outputted; After obtaining the target input for the target identifier, the method further includes: The second model is adjusted based on the target input, and different target inputs result in different numbers of target images determined based on the output of the second model.

7. The method according to any one of claims 4 to 6, adjusting the model based on the target input, comprising: If the target input satisfies the first condition, determining the target feature position and the target feature type in the corresponding target image, and adjusting the model based on the target image, the target feature position and the target feature type, so that the number of target images determined based on the model increases; If the target input does not satisfy the first condition, the model is adjusted based on the corresponding target image so that the number of target images determined based on the model is reduced.

8. A data processing device, comprising: A first output module, configured to output at least one frame of target image in response to a target instruction; The target image is a video frame with target features captured by the electronic device from a video output when the target application is running; A second output module, configured to output a target identification while outputting at least one frame of the target image; The target identifier is used to identify the target feature in the target image of the frame; The acquisition module is used to obtain a target input for the target identifier, wherein the target input affects the output of the target image that has not been output.

9. A computer device comprising a memory and a processor, wherein the memory stores a computer program that can be run on the processor, and when the processor executes the program, the following steps are implemented: In response to the target instruction, at least one frame of target image is output; the target image is a video frame with target features captured by the electronic device from a video output when the target application is running; Outputting at least one frame of target image and outputting the target identification at the same time; The target identifier is used to identify the target feature in the target image of the frame; A target input for the target identifier is obtained, where the target input affects the number of at least one frame of the determined target image.

10. A computer-readable storage medium having a computer program stored thereon, wherein the computer program implements the steps of the method according to claims 1 to 7 when executed by a processor.

Images