Data processing method and computer device

Abstract

The embodiment of the present application provides a data processing method and a computer device, which relates to the target detection technology in artificial intelligence technology and can be applied in the field of video saliency detection; including: obtaining data structure features corresponding to multimedia data, generating an incentive image corresponding to the data structure features; the incentive image is used to highlight the predicted area where the salient object in the multimedia data is located; obtaining salient content features corresponding to the multimedia data, enhancing the salient content features according to the incentive image, and obtaining incentive content features; generating an object area image corresponding to the incentive content features; the object area image is used to detect salient objects in the multimedia data. By adopting the embodiment of the present application, the accuracy of saliency detection of multimedia data can be improved.

Description

Technical Field

[0001] The present application relates to the field of Internet technology, and in particular to a data processing method and a computer device. Background Art

[0002] Visual Saliency Detection refers to simulating human visual characteristics through intelligent algorithms to extract salient areas (i.e. areas of human interest) in images. The rapid development of Internet technology has brought about the spread of large amounts of data, and how to quickly extract salient areas from large amounts of video data has become a hot topic.

[0003] In the prior art, a video can be divided into multiple video frames, and features are extracted for each video frame to obtain feature information corresponding to each video frame, and saliency detection is performed on the video based on the feature information. However, due to the motion of salient objects in the video, insignificant objects in the video may become salient in some individual video frames, resulting in a lack of global information in the feature information extracted from the video frames, which in turn leads to a low accuracy of the saliency detection results of the video. Summary of the invention

[0004] The embodiments of the present application provide a data processing method and a computer device, which can improve the accuracy of saliency detection of multimedia data.

[0005] On the one hand, an embodiment of the present application provides a data processing method, including:

[0006] Acquire data structure features corresponding to multimedia data, and generate an excitation image corresponding to the data structure features; the excitation image is used to highlight the predicted area where the salient object in the multimedia data is located;

[0007] Obtaining salient content features corresponding to the multimedia data, and enhancing the salient content features according to the stimulus image to obtain stimulus content features;

[0008] An object region image corresponding to the stimulus content feature is generated; the object region image is used to detect salient objects in multimedia data.

[0009] On the one hand, an embodiment of the present application provides a data processing method, including:

[0010] Obtaining sample data, obtaining sample data structure features corresponding to the sample data in an initial excitation prediction component of an initial data detection model, and generating a sample excitation image corresponding to the sample data structure features; the sample excitation image is used to highlight the prediction area where the sample object in the sample data is located;

[0011] In the initial saliency prediction component of the initial data detection model, a sample content feature corresponding to the sample data is obtained, and the sample content feature is enhanced according to the sample stimulus image to obtain the sample stimulus content feature;

[0012] Generate a sample region image corresponding to the sample data in the initial saliency prediction component;

[0013] According to the sample excitation image, the sample region image and the sample labels corresponding to the sample data, the network parameters in the initial data detection model are corrected, and the initial data detection model including the corrected network parameters is determined as the data detection model; the data detection model is used to detect salient objects contained in the multimedia data.

[0014] An embodiment of the present application provides a data processing device, including:

[0015] The excitation image generation module is used to obtain the data structure features corresponding to the multimedia data and generate an excitation image corresponding to the data structure features; the excitation image is used to highlight the predicted area where the salient object in the multimedia data is located;

[0016] A first excitation module is used to obtain a significant content feature corresponding to the multimedia data, and enhance the significant content feature according to the excitation image to obtain an excitation content feature;

[0017] The saliency map generation module is used to generate an object region image corresponding to the excitation content feature; the object region image is used to detect salient objects in multimedia data.

[0018] Wherein, the data structure feature includes a spatial feature, and the excitation image includes a spatial excitation image;

[0019] The stimulus image generation module includes:

[0020] A data input unit, used for acquiring multimedia data and inputting the multimedia data into a spatial excitation prediction component in a data detection model;

[0021] A spatial feature extraction unit, configured to extract features from multimedia data according to a spatial encoder in a spatial excitation prediction component, and obtain spatial features corresponding to the multimedia data;

[0022] The spatial feature decoding unit is used to decode the spatial features according to the spatial decoder in the spatial excitation prediction component to generate a spatial excitation image corresponding to the multimedia data.

[0023] Among them, the spatial encoder includes residual blocks and hole convolution layers;

[0024] The spatial feature extraction unit includes:

[0025] A first convolution processing subunit, configured to perform convolution processing on the multimedia data according to the residual block in the spatial excitation prediction component to obtain residual convolution features corresponding to the multimedia data;

[0026] The second convolution processing subunit is used to obtain the dilation rate corresponding to the dilated convolution layer, and perform convolution processing on the residual convolution feature according to the dilation rate to obtain the dilated convolution feature corresponding to the residual convolution feature;

[0027] The convolution feature concatenation subunit is used to determine the residual convolution features and the hole convolution features as the spatial features corresponding to the multimedia data.

[0028] The multimedia data includes video data, the data structure feature includes a time feature, and the excitation image includes a time excitation image;

[0029] The stimulus image generation module includes:

[0030] A video frame acquisition unit is used to acquire video data and acquire adjacent video frames T in the video data. n-1 and video frame T n ; n is a positive integer less than or equal to the number of video frames into which the video data is divided;

[0031] Optical flow image acquisition unit, used to acquire video frame T n-1 and video frame T n The optical flow image is used to characterize the pixel motion of salient objects in video data.

[0032] A temporal feature extraction unit is used to input the optical flow image into the temporal excitation prediction component in the data detection model, extract features of the optical flow image according to the temporal encoder in the temporal excitation prediction component, and obtain temporal features corresponding to the optical flow image;

[0033] The temporal feature decoding unit is used to decode the temporal feature according to the temporal decoder in the temporal excitation prediction component to generate a temporal excitation image corresponding to the optical flow image.

[0034] The multimedia data includes video data, the data structure features include spatial features and temporal features, and the excitation images include spatial excitation images and temporal excitation images;

[0035] The stimulus image generation module includes:

[0036] A spatial excitation image generation unit, used for inputting video data into a spatial excitation prediction component in a data detection model, obtaining spatial features corresponding to the video data in the spatial excitation prediction component, and generating a spatial excitation image corresponding to the spatial features;

[0037] A motion feature acquisition unit is used to acquire an optical flow image corresponding to the video data, input the optical flow image into a temporal excitation prediction component in a data detection model, and acquire object motion features corresponding to the optical flow image according to a temporal encoder in the temporal excitation prediction component; the optical flow image is used to characterize pixel motion of a salient object in the video data;

[0038] A temporal excitation unit, used to enhance the motion features of the object according to the spatial excitation image, and obtain the temporal features corresponding to the optical flow image;

[0039] The temporal excitation image generation unit is used to generate a temporal excitation image corresponding to the optical flow image according to the temporal decoder in the temporal excitation prediction component.

[0040] The time incentive unit includes:

[0041] A point multiplication operation subunit, used for performing a point multiplication operation on the spatial excitation image and the object motion feature to obtain a transition feature corresponding to the optical flow image;

[0042] The excitation subunit is used to obtain the excitation rate corresponding to the object motion feature, and generate the time feature corresponding to the optical flow image according to the excitation rate, transition feature and object motion feature; the excitation rate is used to balance the relationship between the transition feature and the object motion feature.

[0043] The multimedia data includes video frames T in the video data. n , n is a positive integer less than or equal to the number of video frames into which the video data is divided;

[0044] The first incentive module includes:

[0045] The saliency extraction unit is used to extract the video frame T n The saliency prediction component in the data detection model is inputted, and the video frame T is decoded according to the saliency encoder in the saliency prediction component. n Perform feature extraction and obtain video frame T n The corresponding content convolution features;

[0046] The hidden state acquisition unit is used to obtain the video frame T in the video data according to the convolutional recurrent neural network in the saliency prediction component. n-1 The corresponding implicit state features;

[0047] The time information adding unit is used to generate a video frame T according to the implicit state features and content convolution features. n Corresponding salient content features.

[0048] The first incentive module includes:

[0049] A first feature enhancement unit, configured to enhance the salient content features according to the spatial excitation image to obtain the spatial excitation features corresponding to the video data;

[0050] A second feature enhancement unit is used to enhance the salient content features according to the temporal excitation image to obtain a temporal excitation feature corresponding to the video data;

[0051] The excitation feature concatenation unit is used to determine the spatial excitation feature and the temporal excitation feature as the excitation content feature corresponding to the video data.

[0052] Among them, the saliency map generation module includes:

[0053] An online stimulus image generation unit, configured to decode stimulus content features according to a saliency decoder in a saliency prediction component, and generate an online stimulus image corresponding to the stimulus content features;

[0054] The online excitation unit is used to enhance the salient content features according to the online excitation image, obtain the optimized excitation content features corresponding to the multimedia data, and generate the object area image corresponding to the optimized excitation content features.

[0055] An embodiment of the present application provides a data processing device, including:

[0056] The sample excitation image generation module is used to obtain sample data, obtain sample data structure features corresponding to the sample data in the initial excitation prediction component of the initial data detection model, and generate a sample excitation image corresponding to the sample data structure features; the sample excitation image is used to highlight the prediction area where the sample object in the sample data is located;

[0057] A second excitation module is used to obtain sample content features corresponding to the sample data in the initial saliency prediction component of the initial data detection model, and to enhance the sample content features according to the sample excitation image to obtain sample excitation content features;

[0058] A sample image generation module, used to generate a sample region image corresponding to the sample data in the initial saliency prediction component;

[0059] The network parameter correction module is used to correct the network parameters in the initial data detection model according to the sample excitation image, the sample area image and the sample labels corresponding to the sample data, and determine the initial data detection model including the corrected network parameters as the data detection model; the data detection model is used to detect the salient objects contained in the multimedia data.

[0060] Wherein, the initial excitation prediction component includes an initial spatial excitation prediction component and an initial temporal excitation prediction component;

[0061] The sample stimulus image generation module includes:

[0062] A sample space excitation image acquisition unit, used to input the sample data into the initial space excitation prediction component in the initial data detection model, and obtain a sample space excitation image corresponding to the sample data in the initial space excitation prediction component;

[0063] A sample motion feature acquisition unit, used to acquire a sample optical flow image corresponding to the sample data, input the sample optical flow image into an initial time excitation prediction component, and acquire a sample motion feature corresponding to the sample optical flow image in the initial time excitation prediction component;

[0064] The sample time excitation image acquisition unit is used to enhance the sample motion feature according to the sample space excitation image, obtain the sample time feature corresponding to the sample data, and generate the sample time excitation image corresponding to the sample data according to the sample time feature.

[0065] Among them, the network parameter correction module includes:

[0066] A first loss function determination unit, used to obtain a sample label corresponding to the sample data, and determine a first loss function corresponding to the initial spatial excitation prediction component according to an error between the sample spatial excitation image and the sample label;

[0067] A second loss function determination unit, used to determine a second loss function corresponding to the initial time excitation prediction component according to an error between the sample time excitation image and the sample label;

[0068] A third loss function determination unit, used to determine a third loss function corresponding to the initial saliency prediction component according to an error between the sample region image and the sample label;

[0069] A parameter correction unit is used to correct the network parameters in the initial data detection model according to the first loss function, the second loss function and the third loss function, and determine the initial data detection model including the corrected network parameters as the data detection model.

[0070] Wherein, the first loss function determining unit includes:

[0071] A cross entropy loss generation subunit is used to obtain the category probability distribution corresponding to the sample space stimulus image according to the sample label, and generate a cross entropy loss function according to the category probability distribution;

[0072] The structural similarity loss subunit is used to divide the sample space stimulus image into K stimulus image blocks, divide the sample label into K label blocks, and generate a structural similarity loss function according to the structural similarity between the K stimulus image blocks and the K label blocks; K is a positive integer;

[0073] The intersection-over-union loss subunit is used to obtain the union and intersection between the sample space stimulus image and the sample label, and generate the intersection-over-union loss function according to the union and the intersection;

[0074] The loss function adding subunit is used to determine the cross entropy loss function, the structural similarity loss function and the intersection-over-union loss function as the first loss function corresponding to the initial spatial excitation prediction component.

[0075] The device further comprises:

[0076] A training times acquisition module is used to obtain the training times of the initial data detection model;

[0077] A training times judgment module, used for, when the training times are less than or equal to a first threshold, enhancing the sample content features according to the sample labels to obtain the sample incentive content features;

[0078] The training times judgment module is further used to generate a candidate stimulus image according to the course learning rate, the sample label and the sample stimulus rate when the training times are greater than the first threshold and less than or equal to the second threshold, and to enhance the sample content features according to the candidate stimulus image to obtain the sample stimulus content features;

[0079] The training times judgment module is also used to execute the step of enhancing the sample content features according to the sample excitation image to obtain the sample excitation content features when the training times are less than the second threshold.

[0080] On one hand, an embodiment of the present application provides a computer device, including a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method in one aspect of the embodiment of the present application.

[0081] On one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. The computer program includes program instructions. When the program instructions are executed by a processor, the steps of the method in one aspect of the embodiment of the present application are executed.

[0082] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program includes computer instructions, the computer instructions are stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the method provided in various optional modes of the above-mentioned first aspect.

[0083] The embodiment of the present application can obtain the data structure feature corresponding to the multimedia data, generate the excitation image corresponding to the data structure feature, and the excitation image can be used to highlight the predicted area where the salient object in the multimedia data is located, and then obtain the salient content feature corresponding to the multimedia data, enhance the salient content feature according to the excitation image to obtain the excitation content feature, and generate the object area image corresponding to the excitation content feature, and the object area image can be used to detect the salient object in the multimedia data. It can be seen that the data structure feature and the salient content feature can be obtained from the multimedia data, and the excitation image corresponding to the data structure feature can be generated. The salient content feature is enhanced by the excitation image, so that the object area image finally generated can accurately capture the salient object in the multimedia data, and the accuracy of the saliency detection of the multimedia data can be improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0084] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0085] Figure 1 It is a structural diagram of a network architecture provided by an embodiment of the present application;

[0086] Figure 2 is a schematic diagram of a saliency detection scenario provided in an embodiment of the present application;

[0087] Figure 3 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0088] Figure 4 It is a planar schematic diagram of a dilated convolution process provided in an embodiment of the present application;

[0089] Figure 5 It is a structural diagram of an excitation module provided in an embodiment of the present application;

[0090] Figure 6 It is a schematic diagram of the framework structure of a data detection model provided in an embodiment of the present application;

[0091] Figure 7 This is a schematic diagram of an application scenario of video saliency detection provided by an embodiment of the present application;

[0092] Figure 8 It is a flowchart of a data processing method provided in an embodiment of the present application;

[0093] Fig. 9 It is a training diagram of a data detection model provided in an embodiment of the present application;

[0094] Fig.10 is a schematic diagram of experimental results of video saliency detection provided by an embodiment of the present application;

[0095] Fig.11 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0096] Fig.12 is a structural schematic diagram of a data processing device provided in an embodiment of the present application;

[0097] Fig.13 is a structural schematic diagram of a computer device provided in an embodiment of the present application;

[0098] Fig.14 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0099] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.

[0100] The embodiments of the present application relate to artificial intelligence (AI) technology. Artificial intelligence is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. In other words, artificial intelligence is a comprehensive technology in computer science that attempts to understand the essence of intelligence and produce a new intelligent machine that can respond in a similar way to human intelligence. Artificial intelligence is to study the design principles and implementation methods of various intelligent machines so that machines have the functions of perception, reasoning and decision-making.

[0101] Artificial intelligence technology is a comprehensive discipline that covers a wide range of fields, including both hardware-level and software-level technologies. Basic artificial intelligence technologies generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning. The embodiments of this application specifically relate to computer vision technology (CV) under artificial intelligence technology.

[0102] Computer vision is a science that studies how to make machines "see". To put it more specifically, it refers to machine vision that uses cameras and computers to replace human eyes to identify, track and measure targets, and further performs graphic processing to make computer processing into images that are more suitable for human eye observation or transmission to instruments for detection. As a scientific discipline, computer vision studies related theories and technologies, and attempts to establish an artificial intelligence system that can obtain information from images or multi-dimensional data. Computer vision technology generally includes image processing, image recognition, image semantic understanding, image retrieval, OCR, video processing, video semantic understanding, video content / behavior recognition, three-dimensional object reconstruction, 3D technology, virtual reality, augmented reality, simultaneous positioning and mapping, and other technologies, as well as common biometric recognition technologies such as face recognition and fingerprint recognition. The embodiments of the present application specifically relate to video saliency detection in computer vision technology.

[0103] See also Figure 1 , Figure 1 Schematic diagram of a network architecture provided by an embodiment of the present application. Figure 1 As shown, the network architecture may include a server 10d and a user terminal cluster, and the user terminal cluster may include one or more user terminals, and the number of user terminals is not limited here. Figure 1 As shown, the user terminal cluster may specifically include user terminal 10a, user terminal 10b and user terminal 10c, etc. Among them, server 10d may be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, as well as big data and artificial intelligence platforms. User terminal 10a, user terminal 10b and user terminal 10c, etc. may include: smart phones, tablet computers, laptop computers, PDAs, mobile Internet devices (mobile internet devices, MIDs), wearable devices (such as smart watches, smart bracelets, etc.) and smart TVs, etc., which have video / image playback functions. As Figure 1 As shown, the user terminal 10a, the user terminal 10b and the user terminal 10c can respectively establish a network connection with the server 10d, so that each user terminal can exchange data with the server 10d through the network connection.

[0104] In data processing scenarios, Figure 1 Taking the user terminal 10a shown as an example, the user terminal 10a can obtain multimedia data (for example, video data, or image data), which can be video data or image data downloaded from the Internet, or photos or videos taken in real time by a camera in the user terminal 10a. The user terminal 10a can extract data structure features (for example, time features, or spatial features) from the multimedia data, and can generate an incentive image by decoding the data structure features. The incentive image can be used to highlight the predicted area where the salient object in the multimedia data is located, that is, the incentive image can play a guiding role in the saliency detection of the multimedia data to more accurately determine the shape and position of the salient object in the multimedia data. The user terminal 10a can also extract salient content features from the multimedia data, enhance the salient content features according to the incentive image obtained above, obtain incentive content features, and then generate an object area image after decoding the incentive content features, and the object area image can be used to detect the salient object in the multimedia data. The embodiment of the present application can enhance the features of the extracted salient content features through the excitation image of the multimedia data. The generated excitation content features can better express the global information contained in the multimedia data. The object area image generated based on the excitation content features can improve the accuracy of saliency detection in the multimedia data.

[0105] Please also see Figure 2 , Figure 2 is a schematic diagram of a saliency detection scenario provided by an embodiment of the present application. Figure 1 Taking the user terminal 10a in the corresponding embodiment as an example, the saliency detection process of image data is described in detail. Figure 2 As shown, the user terminal 10a can obtain tourist photos taken by a camera device (for example, a SLR camera, a mobile phone camera, etc.), and use the tourist photos as the image to be detected 20a.

[0106] The user terminal 10a can obtain a data detection model, which can be used to generate an object region image 20i corresponding to the image to be detected 20a, and the object region image 20i can be used to determine the shape and position of the salient object contained in the image to be detected 20a. The data detection model can include an excitation prediction component 20b and a saliency detection component 20e. The excitation prediction component 20b can be used to predict a data structure feature 20c with rich structural information (for example, spatial information) and generate an excitation image 20d corresponding to the data structure feature 20c; the saliency detection component 20e can be used to predict a salient content feature 20f with salient object information and generate an object region image 20i corresponding to the salient content feature 20f. The excitation prediction component 20b can be obtained by training based on a dilated residual convolutional neural network and semi-course learning, and the saliency detection component 20e can be obtained by training based on a dilated residual convolutional neural network.

[0107] Among them, the specific process of generating the excitation image 20d may include: the user terminal 10a may first input the image to be detected 20a into the encoder 1 of the excitation prediction component 20b, that is, use multiple residual blocks to perform convolution operations and pooling operations on the image to be detected 20a to obtain the depth features corresponding to the image to be detected 20a. In order to deal with the uncertainty of the size of significant objects, dilated convolutions (also called expanded convolutions or dilated convolutions) can be introduced in the encoder 1 to extract multi-level feature maps, that is, by setting different dilation rates, the depth features can be processed by dilated convolutions to obtain dilated convolution feature maps with different scales. The user terminal 10a can use the depth feature map and the dilated convolution feature maps of different scales as the data structure features 20c associated with the image to be detected 20a. The user terminal 10a can decode the data structure feature 20c according to the decoder 1 in the excitation prediction component 20b to generate an excitation image 20d corresponding to the data structure feature 20c. The excitation image 20d can include spatial structure information in the image to be detected 20a, and has a guiding role in predicting salient objects in the image to be detected 20a.

[0108] Further, the specific process of generating the object area image 20i may include: the user terminal 10a may input the image to be detected 20a into the encoder 2 of the saliency prediction component 20e, that is, the feature extraction of the image to be detected 20a may be performed through the residual block and the hole convolution, and the salient content feature 20f corresponding to the image to be detected 20a may be obtained. It should be noted that the encoder 1 in the excitation prediction component 20b has the same structure as the encoder 2 in the saliency prediction component 20e, that is, the number of residual blocks, the number of hole convolution stages, the size of the convolution kernel used in each residual block, etc. contained in the encoder 1 are the same as those of the encoder 2; but the network parameters used in the components are different, such as the parameters of the convolution kernel, the network parameters in the residual block, etc. are all different. Of course, the decoder 1 in the excitation prediction component 20b and the decoder 2 in the saliency prediction component 20e also have the same structure, but the parameters used are different.

[0109] After the user terminal 10a extracts the significant content feature 20f from the image to be detected 20a, the significant content feature 20f can be input into the excitation module 20g. At the same time, the excitation image 20d generated by the excitation prediction component 20b can also be input into the excitation module 20g. The user terminal 10a can perform a dot multiplication operation on the significant content feature 20f and the excitation image 20d, that is, the significant content feature 20f can be represented as a feature matrix (such as feature matrix A), and the excitation image 20d can be represented as a feature matrix (such as feature matrix B). By correspondingly multiplying the elements in the feature matrix A and the feature matrix B, a new feature matrix is ​​obtained. The user terminal 10a can obtain the excitation rate β, and add the product between the new feature matrix and the excitation rate β to the product between the significant content feature 20f and (1-β) to obtain the excitation content feature 20h. The user terminal 10a can decode the excitation content feature 20h according to the decoder 2 in the saliency prediction component 20e to generate the object area image 20i corresponding to the excitation content feature 20h.

[0110] After acquiring the object area image 20i, the user terminal 10a can determine the shape and position of the significant object contained in the image to be detected 20a according to the object area image 20i, and perform subsequent processing on the image to be detected 20a. For example, after the user terminal 10a detects the significant object in the image to be detected 20a, the image optimization processing (for example, increasing contrast, increasing transparency, adding filters, etc.) can be performed on the area where the significant object is located, or the detected significant object is identified to obtain the identification result for the significant object, etc. The user terminal 10a can also determine the foreground area and background area in the image to be detected 20a according to the object area image 20i, and the background area can be virtualized. Among them, the area where the significant object is located in the image to be detected 20a can be called the foreground area of ​​the image to be detected 20a, such as the white area in the object area image 20i, and the remaining areas in the image to be detected 20a except the area where the significant object is located can all be called background areas, such as the black area in the object area image 20i.

[0111] See also Figure 3 , Figure 3 1 is a flowchart of a data processing method provided in an embodiment of the present application. It is understandable that the data processing method can be executed by a computer device, which can be a user terminal, or a server, or a system composed of a user terminal and a server, or a computer program application (including program code), which is not specifically limited here. Figure 3 As shown, the data processing method may include the following steps:

[0112] Step S101, obtaining data structure features corresponding to multimedia data, and generating an excitation image corresponding to the data structure features; the excitation image is used to highlight the predicted area where the salient object in the multimedia data is located.

[0113] Specifically, computer equipment (such as the above Figure 1 The user terminal 10a in the corresponding embodiment can obtain multimedia data (such as the above Figure 2 The multimedia data is input into the trained data detection model, and the multimedia data is subjected to saliency detection according to the data detection model to generate an object region image (such as the above) for detecting salient objects in the multimedia data. Figure 2The object area image 20i in the corresponding embodiment). The multimedia data in the embodiment of the present application may be video data or image data; the multimedia data may refer to video / image data downloaded by the computer device from the Internet, or video / image data obtained in a multimedia application (such as a video playback application), or video / image data taken by a camera device (for example, photos, shot videos, surveillance videos, etc.). The present application does not limit the source of the multimedia data.

[0114] It should be noted that saliency detection can be understood as detecting the most prominent and attractive area or object in multimedia data, and the most prominent and attractive area or object in multimedia data can be called a salient object. The data detection model can refer to a network model for performing saliency detection on multimedia data. The data detection model can include at least two branches, each of which can be regarded as a network component. For example, the data detection model can include a spatial excitation prediction component (such as the above Figure 2 The corresponding embodiment of the incentive prediction component 20b) and the significance prediction component (as described above Figure 2 The saliency prediction component 20f) in the corresponding embodiment may have two branches, or the data detection model may include two branches, namely, a temporal excitation prediction component and a saliency prediction component, or the data detection model may include three branches, namely, a spatial excitation prediction component, a temporal excitation prediction component and a saliency prediction component. Among them, the spatial excitation prediction component may be used to extract spatial information in multimedia data and generate a spatial excitation image; the temporal excitation prediction component may be used to extract temporal information in multimedia data (the multimedia data at this time is video data) and generate a temporal excitation image; the saliency prediction component may be used to extract salient content in multimedia data and generate an object region image for a salient object, that is, it may be used to predict saliency. The spatial information and temporal information of multimedia data may both be referred to as data structure features corresponding to multimedia data, and the spatial excitation image and the temporal excitation image may both be referred to as excitation images corresponding to multimedia data, which may be used to highlight the predicted region where the salient object in the multimedia data is located.

[0115] It is understandable that when the computer device inputs the multimedia data into the data detection model, it actually inputs the multimedia data into different branches of the data detection model. For the spatial excitation prediction component of the data detection model, the data structure features at this time include spatial features, and the excitation image includes a spatial excitation image. After the computer device inputs the multimedia data into the spatial excitation prediction component, it can be based on the spatial encoder in the spatial excitation prediction component (such as the above Figure 2The encoder 1 in the corresponding embodiment extracts features from the multimedia data to obtain spatial features (also referred to as spatial information) corresponding to the multimedia data, and then the spatial decoder in the spatial excitation prediction component (such as the above) can be used to extract features from the multimedia data. Figure 2 The decoder 1) in the corresponding embodiment decodes the spatial features to generate a spatially excited image corresponding to the multimedia data. The spatial encoder can be essentially understood as a process of downsampling multimedia data for feature extraction, and the spatial decoder can essentially be understood as a process of upsampling spatial features to generate an image.

[0116] Among them, the spatial encoder can be a dilated residual convolutional network, which can include multiple residual blocks and dilated convolutional layers (the dilated residual convolutional network can be a network composed of a ResNet network + a dilated convolutional layer, and the ResNet network is a model structure in a convolutional neural network). The residual block can refer to directly skipping one or more convolutional layers of the input information and transmitting it to the subsequent network layer, which is conducive to protecting the integrity of the multimedia information. For example, if a residual block includes two convolutional layers and the input information is A, then the computer device can input information A into the residual block, and then perform a convolution operation on information A through the first convolutional layer, and output information B in the first convolutional layer. At this time, information B can be used as input information of the second convolutional layer, and a convolution operation is performed on information B through the second convolutional layer, and information C can be output in the second convolutional layer. At this time, information A can be directly skipped over the two convolutional layers in the residual block and input into the next residual block together with information C. This process can also be called a residual connection.

[0117] The computer device can perform convolution processing on the multimedia data according to the residual blocks contained in the spatial excitation prediction component to obtain residual convolution features corresponding to the multimedia data, that is, multiple connected residual blocks in the void residual convolution network are used to perform convolution processing on the multimedia data in turn to extract deep features. The deep features at this time can be called residual convolution features. Each residual block can be composed of one or more convolution layers, normalization, and activation functions. The convolution processing process of multimedia data is described in detail below, taking any convolution layer in multiple residual blocks as an example. Each convolution layer can correspond to one or more convolution kernels (kernels, which can also be called filters or receptive fields). Convolution processing can refer to matrix multiplication operations between the convolution kernel and the input matrix corresponding to the multimedia data. The number of rows of the output image features after the convolution operation is H. out and the number of columns W out It is determined by the size of the input matrix, the size of the convolution kernel, the stride, and the padding, that is, H out =(H in -H kerner+2*padding) / stride+1,W out =(W in -W kerner +2*padding) / stride+1. H in , H kerner Respectively represent the number of rows of output image features and the number of rows of convolution kernels; W in , W kerner They represent the number of columns of the input matrix and the number of columns of the convolution kernel respectively.

[0118] Optionally, a pooling layer can be embedded after the convolution layer. The pooling layer can be an average pooling layer or a maximum pooling layer. The average pooling operation method is to calculate an average value to represent the row (or column) in each row (or column) of the image features output by the previous convolution layer of the pooling layer. The maximum pooling operation is to select a maximum value to represent the row (or column) in each row (or column) of the image features output by the previous convolution layer of the pooling layer.

[0119] Assuming that the residual block includes two convolutional layers, after two convolution operations, the output after the two convolutions can be added to the input before the convolution operation, and the output result corresponding to the residual block can be obtained after the activation function. The calculation formula of the output result can be expressed as:

[0120]

[0121] Among them, R out It can be expressed as the output result of the residual block, x can be expressed as the input of the residual block, W1 and W2 represent the weights in the first and second convolution operations respectively, and σ can be expressed as an activation function. Each residual block in the dilated residual convolution network can perform the above operations and finally output the residual convolution features of multimedia data.

[0122] It should be noted that the number of multimedia data can be n (n is a positive integer), and the residual convolution feature corresponding to the nth multimedia data can be expressed as X n , residual convolution feature X n It can be expressed as the output result of the last layer of the ResNet network for the nth multimedia data. The residual convolution feature X n The scale can be expressed as w×h×c, where the first dimension w represents the difference convolution feature X n The width of the second dimension h represents the difference convolution feature X n The height of the third dimension c represents the difference convolution feature X n The number of channels.

[0123] Since the salient objects in different multimedia data have different shapes and sizes, in order to adapt the data detection model to salient objects of different sizes to extract more representative features, a dilated convolution can be introduced in the spatial encoder, that is, the dilated residual convolution network can include a dilated convolution layer, and the residual convolution features are processed by dilated convolution to extract multi-level feature maps with different scales. Further, the computer device can obtain the dilated rate corresponding to the dilated convolution layer, and convolve the residual convolution features according to the dilated rate to obtain the dilated convolution features corresponding to the residual convolution features, and then the residual convolution features and the dilated convolution features can be determined as the spatial features corresponding to the multimedia data. Among them, the dilated convolution can be to inject holes into the standard convolution feature map to increase the receptive field; compared with the standard convolution process, the dilated convolution process introduces a dilated rate, which can be used to represent the number of intervals of the convolution kernel, and the dilated rate in the standard convolution process is 1.

[0124] The computer device can extract multi-level features with different scales by continuously improving the void ratio, which can be expressed as M is a positive integer representing the maximum dilation rate, or can be expressed as the number of dilated convolution layers, and k is a positive integer less than or equal to M. When k is equal to 1, the dilation rate is 2, that is, the number of intervals of the convolution kernel in the dilated convolution process is 1; when k is equal to 2, the dilation rate is 4, that is, the number of intervals of the convolution kernel in the dilated convolution process is 3; the dilated convolution feature associated with the nth multimedia data extracted by the kth dilated convolution layer can be expressed as The dilated convolution feature The scale can be expressed as w×h×c′, where c′ represents the hole convolution feature. The number of channels.

[0125] The computer device can convolute the residual feature X n The dilated convolution features output by the M dilated convolution layers constitute a feature set (this feature set can be called spatial feature ), the spatial characteristics Ding is expressed as:

[0126]

[0127] in, The scale can be expressed as w×h×(c+M×c′), and the spatial characteristics and Not only retains the original residual convolution feature X n , and has a larger receptive field, which is conducive to obtaining multi-scale information. Computer equipment can The input is sent to the spatial decoder of the spatial excitation prediction component, and the spatial features are processed by the spatial decoder. Decoding can generate a spatially motivated image corresponding to the multimedia data

[0128] Please also see Figure 4 , Figure 4 Schematic diagram of a dilated convolution process provided by an embodiment of the present application. Figure 4 As shown, assuming the residual convolution feature X n The specific size is 17×17×1 (the first dimension is used to identify the feature width, the second dimension is used to identify the feature height, and the third dimension is used to identify the number of channels), that is, the residual convolution feature X n It can be a 17×17 matrix with 1 channel, and the size of the convolution kernel is 3×3×1 (the first dimension is used to identify the width of the convolution kernel, the second dimension is used to identify the height of the convolution kernel, and the third dimension is used for the number of convolution kernel channels), that is, the convolution kernel can be a 3×3 matrix with 1 channel, and the convolution kernel can have 9 parameters.

[0129] When the dilation rate is 1, it means that the dilated convolution processing at this time is the standard convolution processing, and the receptive field of the convolution kernel is 3×3=9. As shown in the feature matrix 30a, there are 3×3 adjacent elements in the feature matrix 30a that undergo convolution operations with the convolution kernel, and the remaining elements can be ignored, that is, the weights of the 9 dots (i.e., 9 elements) in the feature matrix 30a are not 0, and the weights of the remaining elements are 0.

[0130] When the dilation rate is 2, it means that in the dilation convolution processing at this time, the receptive field of the convolution kernel is 7×7=49, as shown in the feature matrix 30b. In each receptive field of size 7×7 of the feature matrix 30b, there are still only 9 elements that can be convolved with the convolution kernel, and the remaining elements can be ignored, that is, the weights of the 9 dots (i.e., 9 elements) in the feature matrix 30b are not 0, and the weights of the remaining elements are 0, and there is 1 element between two adjacent dots.

[0131] When the dilation rate is 2, it means that in the dilated convolution processing at this time, the receptive field of the convolution kernel is 15×15=225, as shown in the feature matrix 30c. In each receptive field of size 15×15 of the feature matrix 30c, there are still only 9 elements that can be convolved with the convolution kernel, and the remaining elements can be ignored, that is, the weights of the 9 dots (i.e., 9 elements) in the feature matrix 30c are not 0, and the weights of the remaining elements are 0, and there are 3 elements between two adjacent dots.

[0132] from Figure 4It can be seen that no matter how the dilation rate changes, the number of convolution kernel parameters remains unchanged, while the size of the receptive field increases exponentially with the increase of the dilation rate. The dilation rate can be used to represent the expansion size of the receptive field in the convolution process. By introducing dilated convolution in the spatial encoder, multi-level information in multimedia data can be obtained, thereby enriching the extracted spatial information.

[0133] Optionally, if the data detection model includes a temporal excitation prediction component but does not include a spatial excitation prediction component, the multimedia data at this time may be video data (when the multimedia data is image data, there is no need to use a temporal excitation prediction component), the data structure feature may be a temporal feature, and the excitation image generated according to the temporal excitation prediction component may be a temporal excitation image, that is, the computer device may acquire the video data and perform frame processing on the video data, dividing the video data into a series of video frames {T1, T2, ..., T n}, that is, dividing the video data into n video frames.

[0134] The computer device can obtain adjacent video frames T in n video frames. n-1 and video frame T n , n is a positive integer less than or equal to the number of video frames divided by the video data, and the default video frame number is T n is the current video frame in the video data. The computer device can obtain the video frame T n-1 To video frame T n The optical flow image is input into the time excitation prediction component in the data detection model, and the time encoder in the time excitation prediction component is used to extract the features of the optical flow image, so as to obtain the time features corresponding to the optical flow image (the time features at this time may include the time information in the video image, which is the output result of the time encoder), and then the time features may be decoded according to the time decoder in the time excitation prediction component to generate the time excitation image corresponding to the optical flow image. Among them, the optical flow may refer to the instantaneous speed of the pixel motion of the spatial moving object on the observation imaging plane, that is, the optical flow image may be used to characterize the pixel motion of the salient object in the video data.

[0135] It should be noted that the network structure of the temporal excitation prediction component is the same as that of the spatial excitation prediction component, that is, the network structure of the temporal encoder is the same as that of the spatial encoder, and the network structure of the temporal decoder is also the same as that of the spatial decoder, but the parameters in the temporal excitation prediction component are different from those in the spatial excitation prediction component. In other words, the temporal encoder may include a dilated residual convolutional network, and the optical flow image is convolved with the residual block in the dilated residual convolutional network to obtain the deep residual convolutional features corresponding to the optical flow image, and then the dilated convolutional layer in the dilated residual convolutional network may be used to obtain the dilated convolutional features corresponding to the deep residual convolutional features, and the deep residual convolutional features and dilated convolutional features in the temporal encoder are used as the temporal features corresponding to the optical flow image. The temporal feature extraction process here can refer to the aforementioned description of the spatial feature extraction process, which will not be repeated here.

[0136] Optionally, if the data detection model includes both a temporal excitation prediction component and a spatial excitation prediction component, the multimedia data at this time may be video data, the data structure features may include temporal features and spatial features, the excitation image may include a spatial excitation image and a temporal excitation image, and the temporal excitation prediction component at this time may include a spatial excitation module. The computer device may obtain an optical flow image corresponding to the video data (i.e., a video frame T in the video data). n-1 To video frame T n The optical flow image is input into the temporal excitation prediction component in the data detection model, and the object motion features corresponding to the optical flow image can be obtained according to the temporal encoder in the temporal excitation prediction component (the object motion features at this time can refer to the output results of the temporal encoder). The computer device can perform spatial excitation on the object motion features, that is, enhance the object motion features according to the spatial excitation image to obtain the temporal features corresponding to the optical flow image, and then decode the temporal features according to the temporal decoder in the temporal excitation prediction component to generate a temporal excitation image corresponding to the optical flow image.

[0137] The temporal features after feature enhancement based on the spatial excitation image can be combined with the spatial information contained in the spatial excitation image, and can also be understood as the temporal features after spatial excitation. The computer device can perform a dot multiplication operation on the spatial excitation image and the object motion feature to obtain the transition feature corresponding to the optical flow image; obtain the excitation rate corresponding to the object motion feature, and generate the temporal features corresponding to the optical flow image according to the excitation rate, the transition feature and the object motion feature. The excitation rate is used to balance the relationship between the transition feature and the object motion feature. Among them, the excitation module can essentially be understood as a feature fusion process, that is, the object motion feature and the spatial excitation image are feature-fused, which can enhance the guiding role of the spatial feature. The specific calculation formula of the temporal feature can be expressed as:

[0138]

[0139] in, It can be represented as a video frame T n The temporal features after spatial excitation processing, ⊙, can be expressed as a matrix dot multiplication operation, β s→t It can be expressed as the time incentive rate in the time incentive prediction component, T is represented by the spatial excitation prediction component for the video frame T n The spatially motivated image, It can be represented as a video frame T n-1 To video frame T n The object motion characteristics corresponding to the optical flow image.

[0140] Computer equipment can transfer time characteristics Input the time decoder in the time excitation prediction component, and the time decoder Decode and generate video frame T n Corresponding time-stimulated image

[0141] Please also see Figure 5 , Figure 5 Schematic diagram of the structure of an excitation module provided in an embodiment of the present application. Figure 5 As shown, the computer device can use the time encoder in the time excitation prediction component to generate the video frame T n-1 To video frame T n The optical flow image is subjected to feature extraction to obtain the object motion feature 40a corresponding to the optical flow image, and the object motion feature 40a is subjected to a dot multiplication operation with the spatial excitation image 40b generated by the spatial excitation prediction component to obtain the transition feature 40c of the optical flow image. The computer device can obtain the excitation rate β in the temporal excitation prediction component s→t , the incentive rate β s→t and the product of transition characteristic 40c, and parameter (1-β s→t ) and the product of the object motion feature 40a are added to obtain the time feature 40d corresponding to the optical flow image.

[0142] Step S102, obtaining the salient content features corresponding to the multimedia data, and enhancing the salient content features according to the incentive image to obtain incentive content features.

[0143] Specifically, the computer device can input multimedia data into the saliency prediction component in the data detection model, perform feature extraction on the multimedia data according to the saliency encoder in the saliency prediction component, and obtain the saliency content features corresponding to the multimedia data (the saliency content features at this time can refer to the output results of the saliency encoder). Among them, the saliency prediction component can include a saliency encoder and a saliency decoder, the network structure of the saliency encoder is the same as the network structure of the spatial encoder, but the network parameters of the saliency encoder and the spatial encoder are different; similarly, the network structure of the saliency decoder is the same as the network structure of the spatial decoder, but the network parameters of the saliency decoder and the spatial decoder are different. In other words, the spatial encoder, the temporal encoder and the saliency encoder in the data detection model all have the same network structure, but the network parameters of the spatial encoder, the temporal encoder and the saliency encoder are all different; the spatial decoder, the temporal decoder and the saliency decoder also have the same network structure, but the network parameters of the spatial decoder, the temporal decoder and the saliency decoder are all different. The extraction process of the above-mentioned saliency features can refer to the description of the spatial feature extraction process in the above-mentioned step S101, which will not be repeated here.

[0144] Optionally, when the multimedia data is video data, in order to take into account the time information between adjacent video frames, a convolutional recurrent neural network (Convolutional Long Short-Term Memory, ConvLSTM) can be introduced in the saliency prediction component to inject the time information into the content features extracted from the saliency encoder. Among them, the ConvLSTM network can refer to replacing the full connection layer of the long short-term memory network (Long Short-Term Memory, LSTM) with a convolution layer, and performing a convolution operation, which can not only obtain the temporal relationship between n video frames in the video data, but also extract features similar to the convolution layer, extract the content features of the video frame, that is, the time content features corresponding to the video frame can be obtained through the ConvLSTM network. Optionally, the ConvLSTM in the saliency prediction component can be replaced by an attention and Transformer mechanism.

[0145] After the ConvLSTM network is introduced into the saliency prediction component, the computer device can process the video frame T n The saliency prediction component in the data detection model is inputted, and the saliency encoder in the saliency prediction component is used to detect the video frame T n By performing feature extraction, we can get the video frame T n The corresponding content convolution feature (the content convolution feature is the output result of the saliency encoder at this time). The computer device can obtain the video frame T in the video data according to the convolutional recurrent neural network in the saliency prediction component.n-1 The corresponding hidden state features can then generate the video frame T according to the hidden state features and content convolution features. n Corresponding salient content features. It should be noted that if the ConvLSTM network is not introduced into the saliency prediction component, the output result of the saliency encoder can be determined as the salient content feature; if the ConvLSTM network is introduced into the saliency prediction component, the output result of the saliency encoder can be determined as the content convolution feature, and the content convolution feature injected with time information can be determined as the salient content feature.

[0146] The computer device can forward encode all video frames in the video data, and first initialize the hidden state characteristics and By encoding the video frame T1 according to the saliency encoder in the saliency prediction component, the content convolution feature corresponding to the video frame T1 can be obtained. Then, the content convolution features can be convolved through the ConvLSTM network. and latent state features Perform forward coding to obtain the implicit state characteristics and corresponding to the video frame T1 Subsequently, the computer device may encode the video frame T2 according to the saliency encoder in the saliency prediction component to obtain the content convolution feature corresponding to the video frame T2. Then, the content convolution features can be convolved through the ConvLSTM network. and latent state features By forward encoding, we can get the implicit state features corresponding to the video frame T2 Similarly, the video frame T can be obtained through the ConvLSTM network n The corresponding hidden state features

[0147] The forward encoding and reverse encoding processes of the ConvLSTM network are the same except for the order of the input video frames (forward encoding can refer to encoding the video frames in order from front to back, and reverse encoding can refer to encoding the video frames in order from back to front). This ensures that the timing information of the video frames will not be lost along with the video frame sequence.

[0148] Among them, the saliency encoder extracts the video frame T n The associated content convolutional features can be expressed as The video frame T can be obtained through the forward encoding of the ConvLSTM network n-1 The corresponding positive hidden state feature is The video frame T can be obtained through the reverse encoding of the ConvLSTM network n-1 The corresponding reverse hidden state features (The forward implicit state feature is and reverse hidden state features It can be called video frame T n-1 The corresponding hidden state features). For the video frame T n , the positive hidden state feature and reverse hidden state features They can be expressed as:

[0149]

[0150]

[0151] The computer device calculates the video frame T according to the above formula (4) and formula (5): n The corresponding positive hidden state is and reverse hidden state features And the positive implicit state and and reverse hidden state features Determined as video frame T n Corresponding salient content features.

[0152] Optionally, if the data detection model includes a spatial excitation prediction component and a saliency prediction component, the computer device can perform feature enhancement on the salient content feature according to the spatial excitation image to obtain the spatial excitation feature corresponding to the salient content feature. The spatial excitation feature at this time can be used as the video frame T n Corresponding incentive content features. The incentive content features may include positive implicit state features after spatial incentives And the reverse implicit state characteristics after spatial excitation Its calculation method can be expressed as:

[0153]

[0154]

[0155] Among them, in formula (6) Can represent the positive implicit state characteristics after spatial excitation can be represented as the spatially excited image generated in the spatially excited prediction component, It can be expressed as the forward excitation rate in the spatial excitation module of the saliency prediction component; It can represent the reverse implicit state characteristics after spatial excitation and Ding is represented as the inverse excitation rate in the spatial excitation module of the saliency prediction component.

[0156] Optionally, if the data detection model includes a temporal excitation prediction component and a saliency prediction component, the computer device can perform feature enhancement on the salient content feature according to the temporal excitation image to obtain a temporal excitation feature corresponding to the salient content feature. The temporal excitation feature at this time can be used as the temporal excitation feature of the video frame T n Corresponding incentive content features. The incentive content features may include positive implicit state features after time incentives And the reverse implicit state characteristics after time stimulation Its calculation method can be expressed as:

[0157]

[0158]

[0159] Among them, in formula (8) Can represent the positive implicit state characteristics after time stimulation can be represented as the temporal excitation image generated in the temporal excitation prediction component, It can be expressed as the forward excitation rate in the time excitation module of the saliency prediction component; It can represent the reverse implicit state characteristics after time excitation It can be expressed as the reverse excitation rate in the temporal excitation module of the saliency prediction component.

[0160] Optionally, if the data detection model includes a temporal excitation prediction component, a spatial excitation prediction component, and a saliency prediction component, the computer device may perform feature enhancement on the salient content feature according to the spatial excitation image to obtain the spatial excitation feature corresponding to the salient content feature; and at the same time, perform feature enhancement on the salient content feature according to the temporal excitation image to obtain the temporal excitation feature corresponding to the salient content feature, and then the spatial excitation feature and the temporal excitation feature may be used as the video frame T n Corresponding incentive content features. The incentive content features at this time may include the positive implicit state features after spatial incentives Positive latent state characteristics after time stimulation Reverse hidden state characteristics after spatial excitation And the reverse implicit state characteristics after time stimulation Its calculation method can be expressed as:

[0161]

[0162]

[0163] Where cat represents the feature connection operation, formula (10) can be expressed as the result of adding formula (6) and formula (8), and formula (11) can be expressed as the result of adding formula (7) and formula (9), that is, Can represent the positive implicit state characteristics after spatial excitation and the positive implicit state characteristics after time stimulation The sum of It can represent the reverse implicit state characteristics after spatial excitation and the reverse hidden state characteristics after time excitation The computer equipment can and Connect and get the video frame T n Corresponding incentive content characteristics.

[0164] It should be noted that the computer device is based on the spatial excitation image and time-stimulated images When enhancing the features of salient content, we first need to and time-stimulated images Downsampling is performed so that the downsampled spatial excitation image The size of is the same as the size of the salient content feature, the downsampled temporal stimulus image The size is the same as the size of the salient content feature.

[0165] Step S103, generating an object region image corresponding to the excitation content feature; the object region image is used to detect salient objects in multimedia data.

[0166] Specifically, the computer device can input the excitation content features into the saliency decoder in the saliency prediction component, decode the excitation content features according to the saliency decoder, and generate an object area image corresponding to the multimedia data, and the object area image can be used to detect salient objects in the multimedia data.

[0167] Optionally, the data detection model may also include an online excitation mechanism, that is, the computer device may decode the excitation content feature according to the saliency decoder in the saliency prediction component, generate an online excitation image corresponding to the excitation content feature, and then perform feature enhancement on the saliency content feature according to the online excitation image to obtain an optimized excitation content feature corresponding to the multimedia data, and then input the optimized excitation content feature into the saliency decoder to generate an object region image corresponding to the optimized excitation content feature. In the actual application of the data detection model, the output image of the saliency decoder may be used as an online excitation image, and the online excitation image may be used to repeatedly replace the spatial excitation image and the temporal excitation image, and the saliency content feature may be subjected to online excitation processing, so that the saliency feature after online excitation may be obtained, and then the final object region image may be generated by the saliency decoder, and the use of online excitation may provide better guidance for the saliency detection of multimedia data.

[0168] Please also see Figure 6 , Figure 6 Schematic diagram of the framework structure of a data detection model provided in the embodiment of the present application. Figure 6 As shown, the data detection model may include a spatial excitation prediction component 50a, a temporal excitation prediction component 50d, and a saliency prediction component 50h. The spatial excitation prediction component 50a may include a spatial encoder and a spatial decoder, the temporal excitation prediction component 50d may include a temporal encoder and a temporal decoder, and the saliency prediction component 50h may include a saliency encoder and a saliency decoder; the network structures of the spatial encoder, the temporal encoder, and the saliency encoder are the same, but the network parameters of the three are different, and the network structures of the spatial decoder, the temporal decoder, and the saliency decoder are also the same, but the network parameters of the three are different.

[0169] For the current video frame T in the video data n , the computer device can convert the video frame T n Input to the spatial encoder, in the video frame T n The spatial feature 50b can be extracted, and the spatial feature 50b can be input into a spatial decoder, and a spatial excitation image 50c can be generated according to the spatial decoder.

[0170] The computer device can obtain the video frame T n-1 To video frame T n The optical flow image 50e is input into the temporal encoder, and the video frame T is extracted from the optical flow image 50e. n According to the excitation module 1 in the time excitation prediction component 50d, the object motion feature can be spatially excited based on the spatial excitation image 50c to obtain the video frame T nThe corresponding time feature 50f; the time feature 50f can then be input into a time decoder, and a time excitation image 50g can be generated according to the time decoder.

[0171] Then, the computer device can process the video frame T n Input to the saliency encoder, in video frame T n The content convolution feature 50i is extracted from the saliency prediction component 50h; the ConvLSTM network can be introduced into the saliency prediction component 50h, and the video frame T can be n-1 The corresponding hidden state feature 50k and content convolution feature 50i are bidirectionally encoded to obtain the video frame T n The computer device can perform spatial excitation on the salient content feature 50j based on the spatial excitation image 50c through the excitation module 2 in the saliency prediction component 50h to obtain the salient content feature 50j after spatial excitation; perform temporal excitation on the salient content feature 50j based on the temporal excitation image 50g through the excitation module 3 in the saliency prediction component 50h to obtain the salient content feature 50j after temporal excitation; the computer device can use the salient content feature 50j after spatial excitation and the salient content feature 50j after temporal excitation as the video frame T n The corresponding excitation content feature 50m can then be input into a saliency decoder, and an online excitation image 50n can be generated based on the saliency decoder.

[0172] The computer device can replace the spatial excitation image 50c and the temporal excitation image 50g with the online excitation image 50n, perform excitation processing on the salient content feature 50j, and obtain an optimized online excitation image. The computer device can also repeat the above process and finally output an object area image for the salient object.

[0173] Optionally, after the computer device generates the object area image, the key area where the salient object is located can be extracted from the multimedia data. When the multimedia data is video data, the key area can be used as a preprocessing step for the video understanding task, that is, in the video understanding task, the extracted key area can be analyzed to improve the accuracy of video understanding. Optionally, the key area can also be used as a preprocessing step for the target tracking task, that is, in the target tracking task, the key area can be used as the area where the target (i.e., the salient object) is located. By identifying the key area, the detection accuracy in target tracking can be improved.

[0174] Optionally, the computer device can also deploy the data detection model on the mobile terminal side, which can be used to distinguish the foreground and background of the beauty function in the mobile phone camera. After performing saliency detection on the image or video, the key area containing the salient object can be extracted. The key area can be prepared for subsequent background blur and portrait processing. Please refer to Figure 7 , Figure 7 is a schematic diagram of an application scenario of video saliency detection provided by an embodiment of the present application. The data detection model can be deployed on the mobile terminal side, such as Figure 7 As shown, the data detection model can be deployed in the user terminal 60a. When the user of the user terminal 60b (such as user A) selects the camera application icon 60b in the user terminal 60a, the user terminal 60b can respond to the trigger operation of the camera application icon 60b, open the camera application in the user terminal 60a, and user A can take photos 60c in real time through the camera application.

[0175] User A can process the captured photo 60c through the function controls in the camera application, such as the camera application can include controls such as "trim", "rotate", "filter", "mosaic" and "background blur". User A can perform a trigger operation on the "background blur" control, and the user terminal 60a can respond to the trigger operation on the "background blur" control, and can use the data detection model to perform saliency detection on the photo 60c to obtain the object area image 60d corresponding to the photo 60c.

[0176] The user terminal 60a can extract the key area containing the significant object (i.e., the foreground area in the photo 60c, the butterfly in the white area) from the photo 60d according to the object area image 60d; the user terminal 60a can perform blur processing on the background area except the white area in the photo 60d to obtain the photo 60e after the background blur processing. After obtaining the photo 60e, the user terminal 60a can display the photo 60e in the camera application and display a storage prompt box 60f in the camera application. The storage prompt box 60f can include two controls of "yes" and "no". When the user A is satisfied with the photo 60e, the "yes" control in the storage prompt box 60f can be triggered to store the photo 60e in the local of the user terminal 60a; when the user A is not satisfied with the photo 60e, the "no" control in the storage prompt box 60f can be triggered to cancel the photo 60e and perform background blur processing on the photo 60c again until the background blur photo that the user A is satisfied with is obtained.

[0177] In an embodiment of the present application, spatial features and temporal features can be obtained from multimedia data (such as video data or image data), a spatial excitation image corresponding to the multimedia data is generated according to the spatial features, and a temporal excitation image corresponding to the multimedia data is generated according to the temporal features. The excitation mechanism is used to enhance the features of the salient content in the multimedia data, so that the generated object area image can not only accurately capture the salient objects in the multimedia data, but also accurately detect the boundaries of the salient objects, thereby improving the accuracy of saliency detection of the multimedia data; in the saliency detection application, the object area image can be used as an online excitation image, and an optimized object area image is generated by the online excitation image, thereby further improving the accuracy of saliency detection of the multimedia data; based on the object area image, key areas in the multimedia data can be extracted, and a more accurate processing area can be provided for subsequent processing tasks of the multimedia data (for example, video content understanding, image modification, target tracking, etc.), which is beneficial to the success rate of subsequent processing tasks.

[0178] See also Figure 8 , Figure 8 1 is a flowchart of a data processing method provided in an embodiment of the present application. It is understandable that the data processing method can be executed by a computer device, which can be a user terminal, or a server, or a system composed of a user terminal and a server, or a computer program application (including program code), which is not specifically limited here. Figure 8 As shown, the data processing method may include the following steps:

[0179] Step S201, obtain sample data, obtain sample data structure features corresponding to the sample data in the initial excitation prediction component of the initial data detection model, and generate a sample excitation image corresponding to the sample data structure features; the sample excitation image is used to highlight the prediction area where the sample object in the sample data is located.

[0180] Specifically, it is understood that the data detection model needs to be trained before it can be used. Fig. 9 , the training process of the data detection model is explained in detail. Fig. 9 is a training diagram of a data detection model provided in an embodiment of the present application, such as Fig. 9As shown, the data detection model may include an initial spatial excitation prediction component 70a, an initial temporal excitation prediction component 70d, and an initial saliency prediction component 70h. In other words, the training process of the data detection model may refer to the training process of the three branches of the initial spatial excitation prediction component 70a, the initial temporal excitation prediction component 70d, and the initial saliency prediction component 70h. It should be noted that in the embodiment of the present application, the data detection model that has not been trained can be referred to as the initial data detection model, the initial spatial excitation prediction component 70a can be understood as an incompletely trained spatial excitation prediction component, the initial temporal excitation prediction component 70d can be understood as an incompletely trained temporal excitation prediction component, and the initial saliency prediction component 70h can be understood as an incompletely trained saliency prediction component.

[0181] The computer device may first obtain sample data, which may carry annotated sample labels, and the sample labels may be used to mark the actual shape and position of the sample object in the sample data. The sample data in this application may refer to a series of annotated video frames (these video frames may be obtained by framing different video data), and the sample object in the sample data may refer to a notable item or notable area in the video data.

[0182] The computer device can input the sample data into the initial data detection model, that is, first input the sample data into the initial spatial excitation prediction component 70a in the initial data detection model, encode the sample data according to the spatial encoder in the initial spatial excitation prediction component 70a, obtain the sample spatial feature 70b corresponding to the sample data, input the sample spatial feature 70b into the spatial decoder for decoding, and generate the sample spatial excitation image corresponding to the sample data. Subsequently, the computer device can obtain the sample optical flow image 70e corresponding to the sample data, input the sample optical flow image 70e into the initial temporal excitation prediction component 70d, encode the sample optical flow image 70e according to the temporal encoder in the initial temporal excitation prediction component 70d, and obtain the sample motion feature corresponding to the sample optical flow image 70e, and then enhance the sample motion feature according to the sample spatial excitation image (i.e., excitation processing) through the excitation module in the initial temporal excitation prediction component 70d, and obtain the sample temporal feature 70f corresponding to the sample data, and input the sample temporal feature 70f into the temporal decoder for decoding, and generate the sample temporal excitation image corresponding to the sample data. The generation process of the sample spatial excitation image and the sample temporal excitation image can refer to the above. Figure 3 The description of generating the spatial excitation image and the temporal excitation image in step S101 of the corresponding embodiment will not be repeated here.

[0183] Step S202, in the initial saliency prediction component of the initial data detection model, sample content features corresponding to the sample data are obtained, and the sample content features are enhanced according to the sample stimulus image to obtain sample stimulus content features.

[0184] Specifically, the computer device can input the sample data into the initial saliency prediction component 70h, encode the sample data according to the saliency encoder, and obtain the sample convolution content feature 70i corresponding to the sample data, and then introduce the ConvLSTM network into the initial saliency prediction component 70h, and bidirectionally encode the sample convolution content feature 70i and the implicit state feature 70k of the previous sample data through the ConvLSTM network to obtain the sample content feature 70j corresponding to the sample data. The computer device can perform spatial excitation (i.e., feature enhancement) on the sample content feature 70j according to the sample spatial excitation image to obtain the sample content feature 70j after spatial excitation, and perform temporal excitation (i.e., feature enhancement) on the sample content feature 70j according to the sample temporal excitation image to obtain the sample content feature 70j after temporal excitation, and then determine the sample content feature 70j after spatial excitation and the sample content feature 70j after temporal excitation as the sample excitation content feature 70m. Among them, the specific extraction process of the sample excitation content feature 70m can be referred to the above Figure 3 The description of the process of extracting the features of the incentive content in step S102 of the corresponding embodiment will not be repeated here.

[0185] Step S203: generating a sample region image corresponding to the sample data in the initial saliency prediction component.

[0186] Specifically, the computer device may input the sample excitation content feature 70m into the saliency decoder, decode the sample excitation content feature 70m according to the saliency decoder, and generate a sample region image corresponding to the sample data.

[0187] Step S204, based on the sample excitation image, the sample region image and the sample labels corresponding to the sample data, the network parameters in the initial data detection model are corrected, and the initial data detection model including the corrected network parameters is determined as the data detection model; the data detection model is used to detect salient objects contained in the multimedia data.

[0188] Specifically, the computer device can correct the network parameters in the initial data detection model according to the sample space excitation image, the sample time excitation image, the sample area image and the sample label. After the training is completed, the network parameters can be saved, and the initial data detection model containing the corrected network parameters can be determined as the data detection model. The trained data detection model can be used to detect the salient objects contained in the video data. Among them, the network parameters in the initial data detection model may include but are not limited to: encoder parameters, decoder parameters, excitation rate, and ConvLSTM network parameters in the three branches of the initial space excitation prediction component 70a, the initial time excitation prediction component 70d, and the initial saliency prediction component 70h.

[0189] In other words, the training process of the initial data detection model may include: determining the first loss function corresponding to the initial spatial excitation prediction component 70a according to the sample spatial excitation image and the sample label, that is, adjusting the network parameters in the initial spatial excitation prediction component 70a by minimizing the first loss function; determining the second loss function corresponding to the initial time excitation prediction component according to the sample time excitation image and the sample label, that is, adjusting the network parameters in the initial time excitation prediction component 70d by minimizing the second loss function; determining the third loss function corresponding to the initial saliency prediction component 70h according to the sample area image 70n and the sample label, that is, adjusting the network parameters in the initial saliency prediction component 70h by minimizing the third loss function.

[0190] In an embodiment of the present application, cross entropy loss, SSIM (structural similarity) loss, and IoU (intersection over union) loss can be used to measure the quality of the sample area image at the pixel level, block level, and object level, respectively. That is, the network parameters in the initial data detection model can be adjusted according to the cross entropy loss, SSIM loss, and IoU loss.

[0191] Among them, the computer device can obtain the category probability distribution corresponding to the sample space excitation image according to the sample label, and generate a cross entropy loss function according to the category probability distribution; then the sample space excitation image can be divided into K (K is a positive integer) excitation image blocks, and the sample label can be divided into K label blocks, and then the structural similarity loss function can be generated according to the structural similarity between the K excitation image blocks and the K label blocks; then the union and intersection between the sample space excitation image and the sample label can be obtained, and the intersection-over-union loss function can be generated according to the union and intersection; the computer device can determine the cross entropy loss function, the structural similarity loss function and the intersection-over-union loss function as the first loss function corresponding to the initial space excitation prediction component 70a. The first loss function l1(S s , GT) can be expressed as:

[0192]

[0193]

[0194]

[0195]

[0196] in, represents the cross entropy loss function corresponding to the initial spatial excitation prediction component 70a, represents the structural similarity loss function corresponding to the initial spatial excitation prediction component 70a, represents the cross entropy loss function corresponding to the initial spatial excitation prediction component 70a; Ss represents the sample spatial excitation image, and GT represents the sample label.

[0197] For formula (14), the computer device can divide the sample space stimulus image and the sample label. For example, the sample space stimulus image can be divided into K stimulus image blocks, each of which can include N×N elements, and the sample label can be divided into K label blocks, each of which can include N×N elements. For the stimulus image block X={x i |i=1,...,N 2} and label block Y = {y i |i=1,...,N 2}, μ in formula (14) x , μ y can be expressed as the mean and variance of the stimulus image block X, σ x , σ y can be expressed as the mean and variance of the label block Y, σ xy represents covariance, c1 and c2 are parameters, and the values ​​of c1 and c2 can be set to: c1 = 0.01 in the embodiment of the present application 2 , c2=0.03 2 , c1 and c2 can be used to maintain the stability of the loss function.

[0198] Similarly, the second loss function l2(S t , GT) can be expressed as:

[0199]

[0200] in, represents the cross entropy loss function corresponding to the initial time incentive prediction component 70d, represents the structural similarity loss function corresponding to the initial time excitation prediction component 70d, represents the cross entropy loss function corresponding to the initial time incentive prediction component 70d; S t represents the sample time excitation image.

[0201] Similarly, the second loss function l3(S v , GT) can be expressed as:

[0202]

[0203] in, represents the cross entropy loss function corresponding to the initial saliency prediction component 70h, represents the structural similarity loss function corresponding to the initial saliency prediction component 70h, represents the cross entropy loss function corresponding to the initial saliency prediction component 70h; S v Represents the sample region image.

[0204] Furthermore, the first loss function l1(S s , GT), the second loss function l2(S t , GT) and the second loss function l3(S v , GT) is determined as the total target loss function of the initial data detection model, and the total target loss function L can be expressed as:

[0205] L=l1(S s ,GT)+l2(S t ,GT)+l3(S v , GT) (18)

[0206] By minimizing the total objective loss function in formula (18), the network parameters of the initial data detection model can be continuously adjusted until the training is completed, thereby obtaining a trained data detection model.

[0207] Optionally, during the training process of the initial data detection model, a semi-curriculum learning strategy may be used in the initial spatial excitation prediction component 70a and the initial temporal excitation prediction component 70d to update the sample excitation image (including the sample spatial excitation image and the sample temporal excitation image). Fig. 9 As shown, the spatial excitation image set 70c can be represented as the sample spatial excitation image learning process in the initial spatial excitation prediction component 70a, and the temporal excitation image set 70g can be represented as the sample temporal excitation image learning process in the initial temporal excitation prediction component 70d.

[0208] The following uses the learning process of the sample space excitation image in the initial space excitation prediction component 70a as an example to describe the half-course learning in a similar way. Half-course learning can mean first using simple tasks for training (the sample space excitation image can be directly updated according to the sample label), and then gradually continuing training with more difficult tasks (the sample space excitation image generated in the initial space excitation prediction component 70a can be used instead of the sample label to update the sample space excitation image), which can better optimize the initial space excitation prediction component 70a. The updating process of the sample space excitation image can be expressed as

[0209] E s =α⊙GT+(1-α)⊙S s (19)

[0210] Among them, α can be expressed as the course learning rate, which can automatically decrease from the initial 1 to 0 during the training process. This operation will gradually transfer the spatial excitation from the label to the learnable sample space excitation image; E s It can be represented as an updated sample space excitation image, namely, a candidate space excitation image.

[0211] Optionally, the half-course learning can be divided into three course stages during the training process. As the number of training rounds changes, the sample space stimulus image E in formula (19) s Can be updated to:

[0212]

[0213] epoch can be expressed as the number of times all sample data are fully trained once. When epoch≤2 (which can be regarded as the first threshold), the sample label GT can be used as the spatial excitation image to perform spatial excitation on the initial temporal excitation prediction component 70d and the initial saliency prediction component 70h. When 2<epoch≤10 (which can be regarded as the second threshold), α⊙GT+(1-α)⊙S can be used. s As the spatial excitation image, the initial temporal excitation prediction component 70d and the initial saliency prediction component 70h are spatially excited; when epoch>10, the output result S of the spatial decoder can be s The initial temporal excitation prediction component 70d and the initial saliency prediction component 70h are spatially excited. In the specific implementation process of the embodiment of the present application, the epoch can be set to 15 times.

[0214] Stage 1: Due to the imbalance of foreground and background pixels in video saliency detection, sample labels are used as spatial excitation images to strengthen the network's focus on salient areas, providing a shortcut for optimization in the early stages of training.

[0215] Stage 2: Data detection models often rely on perfect truth maps (sample labels). Once the truth maps are removed, the performance of the data detection model will decline. Therefore, in the embodiment of the present application, the output results of the initial spatial excitation prediction component 70a (controlled by the course learning rate α) can be gradually used to replace the sample labels. In this stage 2, the output results of the initial spatial excitation prediction component 70a are used to introduce disturbances to prevent the data detection model from being too dependent on the sample labels.

[0216] Phase 3: When α decays to zero, the data detection model is only stimulated by the output result of the initial spatial excitation prediction component 70a, which can avoid the data detection model from being overly dependent on the sample label.

[0217] After the initial data detection model is trained based on the above process, in order to verify the performance of the data detection model, the trained data detection model can be verified using an experimental data set. In the embodiment of the present application, the most commonly used video saliency detection data set can be used to verify the trained data detection model. The video saliency detection data set may include but is not limited to: Freiburg-Berkeley motion segmentation data set (FBMS), video saliency detection data set (ViSal), densely annotated video segmentation data set (DAVIS) and densely annotated video salient object detection data set (DAVSOD). Among them, the FBMS data set may include 59 videos, including 720 video frames carrying sample labels, 29 videos for the training of the initial data detection model, and the remaining videos can be used to test the trained data detection model. The DAVIS data set is a high-quality, high-resolution data set, which includes densely annotated data sets of 480p and 1080p resolutions, with a total of 50 video sequences and 3455 densely annotated video frames at the pixel level. 2079 video frames from 30 videos are used to train the initial data detection model, and 1376 video frames from 20 videos are used to test the trained data detection model. The ViSal dataset may refer to a dataset specifically designed for video saliency detection, which includes 17 videos and 193 manually annotated video frames. The DAVSOD dataset may refer to the most challenging video saliency detection dataset with pixel-level annotations and eye-fixation labels.

[0218] In an embodiment of the present application, a computer device may compare the data detection model in the present application with an existing saliency detection method for images and a saliency detection method for videos. Among them, the existing image-based saliency detection methods may include but are not limited to: DSS (Deeply supervised salient object Detection with short connection, a static image saliency detection model), BMPM (bi-directional message passing model for salient object detection, bidirectional message passing saliency detection model), BASNet (Boundary-Aware Salient object Detection, a saliency detection algorithm focusing on edges).Existing video-based saliency detection methods may include but are not limited to: SIVM (Segment salient objects from images and videos, a saliency detection method for image and video segmentation salient objects), MSTM (real time salient object detection with minimal spanning tree, a real-time saliency detection method based on minimum spanning tree), SFLR (video saliency detection via spatial-temporal fusion and low-rank coherence diffusion, a video saliency detection method based on space-time fusion and low-rank coherence diffusion), SCOM (spatiotemporal constrained optimization for salient object detection, a saliency detection method using spatiotemporal constraint optimization), SCNN (weakly supervised salient object detection with spatiotemporal cascade neural networks, used for weakly supervised saliency detection to obtain spatiotemporal cascade neural networks), FCNS (video salient object detection via fully convolutional networks, based on fully connected convolutional network saliency detection), FGRNE (Flow guided recurrent neural encoder for video salient object detection, optical flow guided recurrent neural encoder for saliency detection), PDBM (pyramid neural network for salient object detection), etc. DilatedDeeper ConvLSTM for Video Salient Object Detection, a saliency detection method based on pyramid dilation and convolutional recurrent neural network), SSAV (saliency-shift aware video salient object detection, a video saliency detection method for saliency transfer), and MGAN (motion guided attention model for video salient object detection).

[0219] Among them, the saliency detection method proposed in this application can be experimented separately according to whether online excitation is used. When the data detection model adopts dual excitation of spatial excitation and temporal excitation, the saliency detection method at this time can be called method 1; when the data detection model adopts triple excitation of spatial excitation, temporal excitation and online excitation, the saliency detection method at this time can be called method 2. The experimental results of method 1, method 2, existing saliency detection methods for images, and saliency detection methods for videos provided in the embodiments of this application on the FBMS dataset and ViSal dataset are shown in Table 1 below:

[0220] Table 1

[0221]

[0222] Among them, the experimental results of method 1, method 2, existing saliency detection methods for images, and saliency detection methods for videos provided in the embodiments of the present application in the DAVIS dataset and the DAVSOD dataset are shown in Table 2:

[0223] Table 2

[0224]

[0225] It should be noted that in the specific experimental process, the mean square error (MSE) and F-measure (F-measure, the weighted harmonic mean of recall and precision under non-negative weight β, can be expressed as max F β ) and S-measure (a method that can measure the detection results of salient objects and backgrounds at the same time) are used as evaluation indicators to measure the effectiveness of each saliency detection method. The purpose of training the data detection model is to make the mean square error as small as possible while making the F-measure and S-measure as large as possible. Optionally, in the experiment, other indicators (for example, mean F β and E metric, mean F β Compared with max F β , focusing more on boundary information, the E metric can combine local pixel values ​​and image-level averages) to measure the effects of various saliency detection methods. This application does not limit the evaluation indicators of the method.

[0226] As can be seen from Tables 1 and 2 above, Method 1 and Method 2 provided in the embodiments of the present application have the best statistical results in the experimental results of the saliency detection methods. When online excitation is used in the experiment (such as Method 2), it can be found that the accuracy of the saliency image finally obtained is improved. The experimental results show that even without further training, accurate online excitation images can provide more accurate guidance for the data detection model.

[0227] Optionally, through the above Tables 1 and 2, the data set also has a great impact on the performance of the data detection model. For video data, attention is often focused on moving objects. Some moving salient objects are not obvious in a single video frame, and the movement of salient objects will distract the attention of the data detection model. For example, for a simple data set such as the ViSal data set with a prominent target object, both the image-based saliency detection method and the video-based saliency detection method have very good performance. However, in some complex data sets, such as the DAVSOD data set and the FBMS data set, the salient objects in the time domain are not significant in the spatial domain, that is, the moving objects are not obvious in some single video frames (for example, the display area of ​​the moving objects in some single video frames accounts for a small proportion of the entire video frame, such as only 10%), and the statistical results of each saliency detection method in the DAVSOD data set and the FBMS data set are worse than the statistical results in the ViSal data set. Since both the method 1 and the method 2 proposed in the embodiment of the present application can include spatial excitation and temporal excitation, specific information can be strengthened from the perspective of space and time, and compared with the existing saliency detection methods, it has better performance.

[0228] Please also see Fig.10 , Fig.10 FIG. 1 is a schematic diagram of experimental results of a video saliency detection provided by an embodiment of the present application. Fig.10 The a-th column in the figure represents the input original image of each saliency detection method in the experiment, the b-th column represents the label image corresponding to the original image, the c-th column represents the saliency image output by method 1 (without using online excitation), the d-th column to the j-th column represent the saliency images output by the existing saliency detection methods, the d-th column specifically represents the saliency image output by the MGAN method, the e-th column specifically represents the saliency image output by the SSAV method, the f-th column specifically represents the saliency image output by the PDBM method, the g-th column specifically represents the saliency image output by the FCNS method, the h-th column specifically represents the saliency image output by the BASNet method, the i-th column specifically represents the saliency image output by the BMPM method, and the j-th column specifically represents the saliency image output by the DSS method.

[0229] like Fig.10As shown in the figure, in the salient images shown in columns h to j, the foreground and background cannot be distinguished due to the lack of temporal information; in the salient images shown in columns d to g, the boundaries of the predicted salient objects are unclear and the internal regions are blurred due to the contradiction between temporal features and spatial features. In comparison, the salient images corresponding to method 1 show clear boundaries and internal salient regions, which can improve the authenticity of the internal salient regions (compared with the labeled images).

[0230] In an embodiment of the present application, two types of incentive strategies, spatial incentive and temporal incentive, can be set to train the data detection model, which can enhance the accuracy of saliency detection of video data. During the training process, half-course learning (changing the label learning rate) is used for model training, that is, at the beginning of training, features can be selectively enhanced by using labels, and then the task difficulty is gradually increased. Learnable incentive images are used instead of labels for incentives, which can improve the training efficiency of the data detection model and accelerate the convergence speed.

[0231] See also Fig.11 , Fig.11 Schematic diagram of a data processing device provided in an embodiment of the present application. Fig.11 As shown, the data processing device 1 may include: an incentive image generation module 11, a first incentive module 12, and a saliency map generation module 13;

[0232] The excitation image generation module 11 is used to obtain the data structure features corresponding to the multimedia data and generate an excitation image corresponding to the data structure features; the excitation image is used to highlight the predicted area where the salient object in the multimedia data is located;

[0233] A first excitation module 12 is used to obtain a significant content feature corresponding to the multimedia data, and enhance the significant content feature according to the excitation image to obtain an excitation content feature;

[0234] The saliency map generation module 13 is used to generate an object region image corresponding to the excitation content feature; the object region image is used to detect salient objects in multimedia data.

[0235] The specific functional implementation of the incentive image generation module 11, the first incentive module 12, and the saliency map generation module 13 can be found in the above Figure 3 Steps S101 to S103 in the corresponding embodiment will not be described in detail here.

[0236] Please also see Fig.11 , the data structure feature includes a spatial feature, and the excitation image includes a spatial excitation image;

[0237] The motivation image generation module 11 may include: a data input unit 1101, a spatial feature extraction unit 1102, and a spatial feature decoding unit 1103;

[0238] A data input unit 1101 is used to obtain multimedia data and input the multimedia data into a spatial excitation prediction component in a data detection model;

[0239] The spatial feature extraction unit 1102 is used to extract features from multimedia data according to the spatial encoder in the spatial excitation prediction component to obtain spatial features corresponding to the multimedia data;

[0240] The spatial feature decoding unit 1103 is used to decode the spatial features according to the spatial decoder in the spatial excitation prediction component to generate a spatial excitation image corresponding to the multimedia data.

[0241] The specific functional implementation of the data input unit 1101, the spatial feature extraction unit 1102, and the spatial feature decoding unit 1103 can be found in the above Figure 3 The step S101 in the corresponding embodiment will not be described in detail here.

[0242] Please also see Fig.11 ,The spatial encoder includes residual blocks and dilated convolutional layers;

[0243] The spatial feature extraction unit 1102 may include: a first convolution processing subunit 11021, a second convolution processing subunit 11022, and a convolution feature stitching subunit 11023;

[0244] The first convolution processing subunit 11021 is used to perform convolution processing on the multimedia data according to the residual block in the spatial excitation prediction component to obtain residual convolution features corresponding to the multimedia data;

[0245] The second convolution processing subunit 11022 is used to obtain a dilation rate corresponding to the dilated convolution layer, and perform convolution processing on the residual convolution feature according to the dilation rate to obtain a dilated convolution feature corresponding to the residual convolution feature;

[0246] The convolution feature concatenation subunit 11023 is used to determine the residual convolution features and the hole convolution features as the spatial features corresponding to the multimedia data.

[0247] The specific functional implementation of the first convolution processing subunit 11021, the second convolution processing subunit 11022, and the convolution feature splicing subunit 11023 can be found in the above Figure 3 The step S101 in the corresponding embodiment will not be described in detail here.

[0248] Please also see Fig.11, the multimedia data includes video data, the data structure feature includes a time feature, and the excitation image includes a time excitation image;

[0249] The incentive image generation module 11 may include: a video frame acquisition unit 1104, an optical flow image acquisition unit 1105, a time feature extraction unit 1106, and a time feature decoding unit 1107;

[0250] The video frame acquisition unit 1104 is used to acquire video data and acquire adjacent video frames T in the video data. n-1 and video frame T n ; n is a positive integer less than or equal to the number of video frames into which the video data is divided;

[0251] The optical flow image acquisition unit 1105 is used to acquire the video frame T n-1 and video frame T n The optical flow image is used to characterize the pixel motion of salient objects in video data;

[0252] The time feature extraction unit 1106 is used to input the optical flow image into the time excitation prediction component in the data detection model, and extract features of the optical flow image according to the time encoder in the time excitation prediction component to obtain the time features corresponding to the optical flow image;

[0253] The temporal feature decoding unit 1107 is used to decode the temporal feature according to the temporal decoder in the temporal excitation prediction component to generate a temporal excitation image corresponding to the optical flow image.

[0254] The specific functional implementation of the video frame acquisition unit 1104, the optical flow image acquisition unit 1105, the time feature extraction unit 1106, and the time feature decoding unit 1107 can be found in the above Figure 3 The step S101 in the corresponding embodiment will not be described in detail here.

[0255] Please also see Fig.11 , multimedia data includes video data, data structure features include spatial features and temporal features, and excitation images include spatial excitation images and temporal excitation images;

[0256] The excitation image generation module 11 may include: a spatial excitation image generation unit 1108, a motion feature acquisition unit 1109, a temporal excitation unit 1110, and a temporal excitation image generation unit 1111;

[0257] The spatial excitation image generation unit 1108 is used to input the video data into the spatial excitation prediction component in the data detection model, obtain the spatial features corresponding to the video data in the spatial excitation prediction component, and generate a spatial excitation image corresponding to the spatial features;

[0258] The motion feature acquisition unit 1109 is used to acquire an optical flow image corresponding to the video data, input the optical flow image into the temporal excitation prediction component in the data detection model, and acquire the object motion feature corresponding to the optical flow image according to the temporal encoder in the temporal excitation prediction component; the optical flow image is used to characterize the pixel motion of the salient object in the video data;

[0259] The time excitation unit 1110 is used to enhance the object motion feature according to the spatial excitation image to obtain the time feature corresponding to the optical flow image;

[0260] The temporal excitation image generation unit 1111 is used to generate a temporal excitation image corresponding to the optical flow image according to the temporal decoder in the temporal excitation prediction component.

[0261] The specific functional implementation of the spatial excitation image generation unit 1108, the motion feature acquisition unit 1109, the time excitation unit 1110, and the time excitation image generation unit 1111 can be referred to above. Figure 3 The step S101 in the corresponding embodiment will not be described in detail here.

[0262] Among them, when the data input unit 1101, the spatial feature extraction unit 1102, and the spatial feature decoding unit 1103 are performing corresponding operations, the video frame acquisition unit 1104, the optical flow image acquisition unit 1105, the temporal feature extraction unit 1106, the temporal feature decoding unit 1107, the spatial excitation image generation unit 1108, the motion feature acquisition unit 1109, the temporal excitation unit 1110, and the temporal excitation image generation unit 1111 are all suspended from performing operations; when the video frame acquisition unit 1104, the optical flow image acquisition unit 1105, the temporal feature extraction unit 1106, and the temporal feature decoding unit 1107 are performing corresponding operations, the data input unit 1101, the spatial feature extraction unit 1106, and the temporal feature decoding unit 1107 are all suspended from performing operations. Unit 1102, spatial feature decoding unit 1103, spatial excitation image generating unit 1108, motion feature acquiring unit 1109, temporal excitation unit 1110, and temporal excitation image generating unit 1111 all suspend the execution of operations; when the spatial excitation image generating unit 1108, the motion feature acquiring unit 1109, the temporal excitation unit 1110, and the temporal excitation image generating unit 1111 are performing corresponding operations, the data input unit 1101, the spatial feature extraction unit 1102, the spatial feature decoding unit 1103, the video frame acquiring unit 1104, the optical flow image acquiring unit 1105, the temporal feature extraction unit 1106, and the temporal feature decoding unit 1107 all suspend the execution of operations.

[0263] Please also see Fig.11 , the time excitation unit 1110 may include: a point multiplication operation subunit 11101, an excitation subunit 11102;

[0264] The dot multiplication operation subunit 11101 is used to perform a dot multiplication operation on the spatial excitation image and the object motion feature to obtain a transition feature corresponding to the optical flow image;

[0265] The excitation subunit 11102 is used to obtain the excitation rate corresponding to the object motion feature, and generate the time feature corresponding to the optical flow image according to the excitation rate, transition feature and object motion feature; the excitation rate is used to balance the relationship between the transition feature and the object motion feature.

[0266] The specific functional implementation of the dot product operation subunit 11101 and the excitation subunit 11102 can be found in the above Figure 3 The step S101 in the corresponding embodiment will not be described in detail here.

[0267] Please also see Fig.11 , the multimedia data includes video frames T in the video data n , n is a positive integer less than or equal to the number of video frames into which the video data is divided;

[0268] The first excitation module 12 may include: a saliency extraction unit 121, a hidden state acquisition unit 122, a time information addition unit 123, a first feature enhancement unit 124, a second feature enhancement unit 125, and an excitation feature splicing unit 126;

[0269] The saliency extraction unit 121 is used to extract the video frame T n The saliency prediction component in the data detection model is inputted, and the video frame T is decoded according to the saliency encoder in the saliency prediction component. n Perform feature extraction and obtain video frame T n The corresponding content convolution features;

[0270] The hidden state acquisition unit 122 is used to acquire the video frame T in the video data according to the convolutional recurrent neural network in the saliency prediction component. n-1 The corresponding implicit state features;

[0271] The time information adding unit 123 is used to generate a video frame T according to the implicit state feature and the content convolution feature. n Corresponding salient content features.

[0272] A first feature enhancement unit 124, configured to enhance the salient content features according to the spatial excitation image to obtain spatial excitation features corresponding to the video data;

[0273] A second feature enhancement unit 125 is used to enhance the salient content features according to the temporal excitation image to obtain a temporal excitation feature corresponding to the video data;

[0274] The excitation feature concatenation unit 126 is used to determine the spatial excitation feature and the temporal excitation feature as the excitation content feature corresponding to the video data.

[0275] The specific functional implementation of the saliency extraction unit 121, the implicit state acquisition unit 122, the time information adding unit 123, the first feature enhancement unit 124, the second feature enhancement unit 125, and the excitation feature splicing unit 126 can be referred to above. Figure 3 Step S102 in the corresponding embodiment will not be described in detail here.

[0276] Please also see Fig.11 , the saliency map generation module 13 may include: an online excitation image generation unit 131, an online excitation unit 132;

[0277] The online excitation image generation unit 131 is used to decode the excitation content feature according to the saliency decoder in the saliency prediction component, and generate an online excitation image corresponding to the excitation content feature;

[0278] The online excitation unit 132 is used to enhance the salient content features according to the online excitation image, obtain the optimized excitation content features corresponding to the multimedia data, and generate the object area image corresponding to the optimized excitation content features.

[0279] The specific functional implementation of the online excitation image generation unit 131 and the online excitation unit 132 can be found in the above Figure 3 Step S103 in the corresponding embodiment will not be described in detail here.

[0280] In an embodiment of the present application, spatial features and temporal features can be obtained from multimedia data (such as video data or image data), a spatial excitation image corresponding to the multimedia data is generated according to the spatial features, and a temporal excitation image corresponding to the multimedia data is generated according to the temporal features. The excitation mechanism is used to enhance the features of the salient content in the multimedia data, so that the generated object area image can not only accurately capture the salient objects in the multimedia data, but also accurately detect the boundaries of the salient objects, thereby improving the accuracy of saliency detection of the multimedia data; in the saliency detection application, the object area image can be used as an online excitation image, and an optimized object area image is generated by the online excitation image, thereby further improving the accuracy of saliency detection of the multimedia data; based on the object area image, key areas in the multimedia data can be extracted, and a more accurate processing area can be provided for subsequent processing tasks of the multimedia data (for example, video content understanding, image modification, target tracking, etc.), which is beneficial to the success rate of subsequent processing tasks.

[0281] See also Fig.12 , Fig.12 Schematic diagram of a data processing device provided in an embodiment of the present application. Fig.12 As shown, the data processing device 2 may include: a sample excitation image generation module 21, a second excitation module 22, a sample image generation module 23, and a network parameter correction module 24;

[0282] The sample excitation image generation module 21 is used to obtain sample data, obtain sample data structure features corresponding to the sample data in the initial excitation prediction component of the initial data detection model, and generate a sample excitation image corresponding to the sample data structure features; the sample excitation image is used to highlight the prediction area where the sample object in the sample data is located;

[0283] The second excitation module 22 is used to obtain the sample content features corresponding to the sample data in the initial saliency prediction component of the initial data detection model, and enhance the sample content features according to the sample excitation image to obtain the sample excitation content features;

[0284] A sample image generation module 23, used to generate a sample region image corresponding to the sample data in the initial saliency prediction component;

[0285] The network parameter correction module 24 is used to correct the network parameters in the initial data detection model according to the sample excitation image, the sample area image and the sample labels corresponding to the sample data, and determine the initial data detection model including the corrected network parameters as the data detection model; the data detection model is used to detect the salient objects contained in the multimedia data.

[0286] The specific functional implementation of the sample excitation image generation module 21, the second excitation module 22, the sample image generation module 23, and the network parameter correction module 24 can be found in the above Figure 8 Steps S201 to S204 in the corresponding embodiment will not be described in detail here.

[0287] Please also see Fig.12 ,The initial excitation prediction component includes an initial spatial excitation prediction component and an initial temporal excitation prediction component;

[0288] The sample excitation image generation module 21 may include: a sample space excitation image acquisition unit 211, a sample motion feature acquisition unit 212, and a sample time excitation image acquisition unit 213;

[0289] The sample space excitation image acquisition unit 211 is used to input the sample data into the initial space excitation prediction component in the initial data detection model, and obtain the sample space excitation image corresponding to the sample data in the initial space excitation prediction component;

[0290] The sample motion feature acquisition unit 212 is used to acquire a sample optical flow image corresponding to the sample data, input the sample optical flow image to the initial time excitation prediction component, and acquire the sample motion feature corresponding to the sample optical flow image in the initial time excitation prediction component;

[0291] The sample time excitation image acquisition unit 213 is used to enhance the sample motion feature according to the sample space excitation image, obtain the sample time feature corresponding to the sample data, and generate the sample time excitation image corresponding to the sample data according to the sample time feature.

[0292] The specific functional implementation of the sample space excitation image acquisition unit 211, the sample motion feature acquisition unit 212, and the sample time excitation image acquisition unit 213 can be found in the above Figure 8 The step S201 in the corresponding embodiment will not be described in detail here.

[0293] Please also see Fig.12 , the network parameter correction module 24 may include: a first loss function determination unit 241, a second loss function determination unit 242, a third loss function determination unit 243, and a parameter correction unit 244;

[0294] A first loss function determination unit 241 is used to obtain a sample label corresponding to the sample data, and determine a first loss function corresponding to the initial spatial excitation prediction component according to an error between the sample spatial excitation image and the sample label;

[0295] A second loss function determining unit 242, configured to determine a second loss function corresponding to the initial time excitation prediction component according to an error between the sample time excitation image and the sample label;

[0296] A third loss function determining unit 243, configured to determine a third loss function corresponding to the initial saliency prediction component according to an error between the sample region image and the sample label;

[0297] The parameter correction unit 244 is used to correct the network parameters in the initial data detection model according to the first loss function, the second loss function and the third loss function, and determine the initial data detection model including the corrected network parameters as the data detection model.

[0298] The specific functional implementation of the first loss function determination unit 241, the second loss function determination unit 242, the third loss function determination unit 243, and the parameter correction unit 244 can be referred to above. Figure 8 Step S204 in the corresponding embodiment will not be described in detail here.

[0299] Please also see Fig.12, the first loss function determination unit 241 may include: a cross entropy loss generation subunit 2411, a structural similarity loss subunit 2412, an intersection-over-union loss subunit 2413, and a loss function addition subunit 2414;

[0300] A cross entropy loss generating subunit 2411 is used to obtain the category probability distribution corresponding to the sample space stimulus image according to the sample label, and generate a cross entropy loss function according to the category probability distribution;

[0301] The structural similarity loss subunit 2412 is used to divide the sample space stimulus image into K stimulus image blocks, divide the sample label into K label blocks, and generate a structural similarity loss function according to the structural similarity between the K stimulus image blocks and the K label blocks; K is a positive integer;

[0302] The intersection-over-union loss subunit 2413 is used to obtain the union and intersection between the sample space stimulus image and the sample label, and generate an intersection-over-union loss function according to the union and the intersection;

[0303] The loss function adding subunit 2414 is used to determine the cross entropy loss function, the structural similarity loss function and the intersection-over-union loss function as the first loss function corresponding to the initial spatial excitation prediction component.

[0304] The specific functional implementation of the cross entropy loss generation subunit 2411, the structural similarity loss subunit 2412, the intersection-over-union loss subunit 2413, and the loss function addition subunit 2414 can be found in the above Figure 8 Step S204 in the corresponding embodiment will not be described in detail here.

[0305] Please also see Fig.12 , the data processing device 2 may further include: a training times acquisition module 25, a training times determination module 26;

[0306] The training times acquisition module 25 is used to acquire the training times of the initial data detection model;

[0307] A training times judgment module 26 is used to enhance the sample content features according to the sample labels to obtain the sample incentive content features when the training times are less than or equal to the first threshold;

[0308] The training times judgment module 26 is further used to generate a candidate stimulus image according to the course learning rate, the sample label and the sample stimulus rate when the training times are greater than the first threshold and less than or equal to the second threshold, and to enhance the sample content features according to the candidate stimulus image to obtain the sample stimulus content features;

[0309] The training times judging module 26 is further configured to, when the training times are less than a second threshold, execute the step of enhancing the sample content features according to the sample excitation image to obtain the sample excitation content features.

[0310] The specific functional implementation of the training times acquisition module 25 and the training times determination module 26 can be found in the above Figure 8 Step S204 in the corresponding embodiment will not be described in detail here.

[0311] In an embodiment of the present application, two types of incentive strategies, spatial incentive and temporal incentive, can be set to train the data detection model, which can enhance the accuracy of saliency detection of video data. During the training process, half-course learning (changing the label learning rate) is used for model training, that is, at the beginning of training, features can be selectively enhanced by using labels, and then the task difficulty is gradually increased. Learnable incentive images are used instead of labels for incentives, which can improve the training efficiency of the data detection model and accelerate the convergence speed.

[0312] See also Fig.13 , Fig.13 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.13 As shown, the computer device 1000 may include: a processor 1001, a network interface 1004 and a memory 1005. In addition, the above-mentioned computer device 1000 may also include: a user interface 1003, and at least one communication bus 1002. Among them, the communication bus 1002 is used to realize the connection and communication between these components. Among them, the user interface 1003 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 1003 may also include a standard wired interface and a wireless interface. Optionally, the network interface 1004 may include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 1005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. Optionally, the memory 1005 may also be at least one storage device located away from the aforementioned processor 1001. As Fig.13 As shown, the memory 1005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0313] In such Fig.13 In the computer device 1000 shown, the network interface 1004 can provide a network communication function; the user interface 1003 is mainly used to provide an input interface for the user; and the processor 1001 can be used to call the device control application stored in the memory 1005 to achieve:

[0314] Acquire data structure features corresponding to multimedia data, and generate an excitation image corresponding to the data structure features; the excitation image is used to highlight the predicted area where the salient object in the multimedia data is located;

[0315] Obtaining salient content features corresponding to the multimedia data, and enhancing the salient content features according to the stimulus image to obtain stimulus content features;

[0316] An object region image corresponding to the stimulus content feature is generated; the object region image is used to detect salient objects in multimedia data.

[0317] It should be understood that the computer device 1000 described in the embodiment of the present application can execute the above Figure 3 The description of the data processing method in the corresponding embodiment can also be performed as described above. Fig.11 The description of the data processing device 1 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0318] See also Fig.14 , Fig.14 Schematic diagram of the structure of a computer device provided in an embodiment of the present application. Fig.14 As shown, the computer device 2000 may include: a processor 2001, a network interface 2004 and a memory 2005. In addition, the above-mentioned computer device 2000 may also include: a user interface 2003, and at least one communication bus 2002. Among them, the communication bus 2002 is used to realize the connection and communication between these components. Among them, the user interface 2003 may include a display screen (Display), a keyboard (Keyboard), and the optional user interface 2003 may also include a standard wired interface and a wireless interface. Optionally, the network interface 2004 may include a standard wired interface and a wireless interface (such as a WI-FI interface). The memory 2005 may be a high-speed RAM memory, or it may be a non-volatile memory (non-volatile memory), such as at least one disk storage. Optionally, the memory 2005 may also be at least one storage device located away from the aforementioned processor 2001. As Fig.14 As shown, the memory 2005 as a computer-readable storage medium may include an operating system, a network communication module, a user interface module, and a device control application program.

[0319] In such Fig.14 In the computer device 2000 shown, the network interface 2004 can provide a network communication function; the user interface 2003 is mainly used to provide an input interface for the user; and the processor 2001 can be used to call the device control application stored in the memory 2005 to achieve:

[0320] Obtaining sample data, obtaining sample data structure features corresponding to the sample data in an initial excitation prediction component of an initial data detection model, and generating a sample excitation image corresponding to the sample data structure features; the sample excitation image is used to highlight the prediction area where the sample object in the sample data is located;

[0321] In the initial saliency prediction component of the initial data detection model, a sample content feature corresponding to the sample data is obtained, and the sample content feature is enhanced according to the sample stimulus image to obtain the sample stimulus content feature;

[0322] Generate a sample region image corresponding to the sample data in the initial saliency prediction component;

[0323] According to the sample excitation image, the sample region image and the sample labels corresponding to the sample data, the network parameters in the initial data detection model are corrected, and the initial data detection model including the corrected network parameters is determined as the data detection model; the data detection model is used to detect salient objects contained in the multimedia data.

[0324] It should be understood that the computer device 2000 described in the embodiment of the present application can execute the above Figure 8 The description of the data processing method in the corresponding embodiment can also be performed as described above. Fig.12 The description of the data processing device 2 in the corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated here either.

[0325] In addition, it should be pointed out here that: the embodiment of the present application also provides a computer-readable storage medium, and the computer-readable storage medium stores a computer program executed by the data processing device 1 or the data processing device 2 mentioned above, and the computer program includes program instructions. When the processor executes the program instructions, it can execute the above-mentioned Figure 3 and Figure 8 The description of the data processing method in any corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer-readable storage medium embodiment involved in this application, please refer to the description of the method embodiment of this application. As an example, program instructions can be deployed on a computing device for execution, or on multiple computing devices located at one location, or on multiple computing devices distributed at multiple locations and interconnected by a communication network. Multiple computing devices distributed at multiple locations and interconnected by a communication network can constitute a blockchain system.

[0326] In addition, it should be noted that: the embodiment of the present application also provides a computer program product or a computer program, which may include computer instructions, and the computer instructions may be stored in a computer-readable storage medium. The processor of the computer device reads the computer instructions from the computer-readable storage medium, and the processor may execute the computer instructions, so that the computer device executes the above Figure 3 and Figure 8 The description of the data processing method in any corresponding embodiment will not be repeated here. In addition, the description of the beneficial effects of the same method will not be repeated. For technical details not disclosed in the computer program product or computer program embodiment involved in this application, please refer to the description of the method embodiment of this application.

[0327] It should be noted that, for the above-mentioned various method embodiments, for the sake of simplicity of description, they are all expressed as a series of action combinations, but those skilled in the art should be aware that this application is not limited by the order of the actions described, because according to this application, some steps can be performed in other orders or simultaneously. Secondly, those skilled in the art should also be aware that the embodiments described in the specification are all preferred embodiments, and the actions and modules involved are not necessarily required by this application.

[0328] The steps in the method of the embodiment of the present application can be adjusted in order, combined and deleted according to actual needs.

[0329] The modules in the device of the embodiment of the present application can be merged, divided and deleted according to actual needs.

[0330] Those skilled in the art can understand that all or part of the processes in the above-mentioned embodiments can be implemented by instructing related hardware through a computer program, and the computer program can be stored in a computer-readable storage medium, and when the program is executed, it can include the processes of the embodiments of the above-mentioned methods. The storage medium can be a disk, an optical disk, a read-only memory (ROM) or a random access memory (RAM), etc.

[0331] The above disclosure is only the preferred embodiment of the present application, which certainly cannot be used to limit the scope of rights of the present application. Therefore, equivalent changes made according to the claims of the present application are still within the scope covered by the present application.

Claims

1. A data processing method, characterized in that: include: Acquire data structure features corresponding to multimedia data, and generate a stimulus image corresponding to the data structure features; The excitation image is used to highlight the predicted area where the salient object in the multimedia data is located; Acquire a salient content feature corresponding to the multimedia data, and enhance the salient content feature according to the excitation image to obtain an excitation content feature; Generating an object region image corresponding to the stimulus content feature; the object region image is used to detect the salient object in the multimedia data; The multimedia data includes video frames T in the video data. n , n is a positive integer less than or equal to the number of video frames into which the video data is divided; The obtaining of the significant content features corresponding to the multimedia data includes: The video frame T n The saliency prediction component in the data detection model is inputted, and the video frame T is processed according to the saliency encoder in the saliency prediction component. n Perform feature extraction to obtain the video frame T n The corresponding content convolution features; According to the convolutional recurrent neural network in the saliency prediction component, a video frame T in the video data is obtained. n-1 The corresponding implicit state features; Generate the video frame T according to the hidden state feature and the content convolution feature n Corresponding salient content features.

2. The method according to claim 1, characterized in that The data structure feature includes a spatial feature, and the excitation image includes a spatial excitation image; The step of acquiring data structure features corresponding to multimedia data and generating a stimulus image corresponding to the data structure features includes: Acquire the multimedia data, and input the multimedia data into a spatial excitation prediction component in the data detection model; Extracting features of the multimedia data according to the spatial encoder in the spatial excitation prediction component to obtain the spatial features corresponding to the multimedia data; The spatial features are decoded according to the spatial decoder in the spatial excitation prediction component to generate the spatial excitation image corresponding to the multimedia data.

3. The method according to claim 2, characterized in that The spatial encoder includes a residual block and a dilated convolutional layer; The extracting features of the multimedia data according to the spatial encoder in the spatial excitation prediction component to obtain spatial features corresponding to the multimedia data includes: According to the residual block in the spatial excitation prediction component, the multimedia data is convolved to obtain residual convolution features corresponding to the multimedia data; Obtaining a dilation rate corresponding to the dilated convolution layer, and performing convolution processing on the residual convolution feature according to the dilation rate to obtain a dilated convolution feature corresponding to the residual convolution feature; The residual convolution feature and the hole convolution feature are determined as spatial features corresponding to the multimedia data.

4. The method according to claim 1, characterized in that: The multimedia data includes video data, the data structure feature includes a time feature, and the excitation image includes a time excitation image; The step of acquiring data structure features corresponding to multimedia data and generating a stimulus image corresponding to the data structure features includes: Obtain the video data, and obtain adjacent video frames T in the video data n-1 and video frame T n ; n is a positive integer less than or equal to the number of video frames into which the video data is divided; Get the video frame T n-1 and the video frame T n The optical flow image is used to characterize the pixel motion of the salient object in the video data; Inputting the optical flow image into the time excitation prediction component in the data detection model, and extracting features of the optical flow image according to the time encoder in the time excitation prediction component to obtain the time features corresponding to the optical flow image; The temporal feature is decoded according to the temporal decoder in the temporal excitation prediction component to generate the temporal excitation image corresponding to the optical flow image.

5. The method according to claim 1, characterized in that The multimedia data includes video data, the data structure features include spatial features and temporal features, and the excitation images include spatial excitation images and temporal excitation images; The step of acquiring data structure features corresponding to multimedia data and generating a stimulus image corresponding to the data structure features includes: Inputting the video data into the spatial excitation prediction component in the data detection model, obtaining the spatial features corresponding to the video data in the spatial excitation prediction component, and generating a spatial excitation image corresponding to the spatial features; Obtain an optical flow image corresponding to the video data, input the optical flow image into a temporal excitation prediction component in the data detection model, and obtain object motion features corresponding to the optical flow image according to a temporal encoder in the temporal excitation prediction component; the optical flow image is used to characterize pixel motion of the salient object in the video data; Performing feature enhancement on the object motion feature according to the spatial excitation image to obtain the temporal feature corresponding to the optical flow image; According to the temporal decoder in the temporal excitation prediction component, the temporal excitation image corresponding to the optical flow image is generated.

6. The method according to claim 5, characterized in that The step of enhancing the object motion feature according to the spatial excitation image to obtain the temporal feature corresponding to the optical flow image includes: Performing a dot multiplication operation on the spatial excitation image and the object motion feature to obtain a transition feature corresponding to the optical flow image; An excitation rate corresponding to the object motion feature is obtained, and a time feature corresponding to the optical flow image is generated according to the excitation rate, the transition feature and the object motion feature; the excitation rate is used to balance the relationship between the transition feature and the object motion feature.

7. The method according to claim 5, characterized in that The step of enhancing the salient content features according to the excitation image to obtain the excitation content features includes: Performing feature enhancement on the salient content features according to the spatial excitation image to obtain spatial excitation features corresponding to the video data; Performing feature enhancement on the salient content feature according to the temporal excitation image to obtain a temporal excitation feature corresponding to the video data; The spatial excitation feature and the temporal excitation feature are determined as excitation content features corresponding to the video data.

8. The method according to claim 1, characterized in that: The generating of the object area image corresponding to the stimulus content feature comprises: Decoding the stimulus content feature according to a saliency decoder in a saliency prediction component to generate an online stimulus image corresponding to the stimulus content feature; The salient content features are enhanced according to the online excitation image to obtain optimized excitation content features corresponding to the multimedia data, and an object region image corresponding to the optimized excitation content features is generated.

9. A data processing method, characterized in that: include: Acquire sample data, and in an initial excitation prediction component of an initial data detection model, acquire sample data structural features corresponding to the sample data, and generate a sample excitation image corresponding to the sample data structural features; the sample excitation image is used to highlight a prediction area where a sample object in the sample data is located; In the initial saliency prediction component of the initial data detection model, a sample content feature corresponding to the sample data is obtained, and the sample content feature is enhanced according to the sample stimulus image to obtain a sample stimulus content feature; Generating a sample region image corresponding to the sample data in the initial saliency prediction component; According to the sample stimulus image, the sample region image and the sample label corresponding to the sample data, the network parameters in the initial data detection model are modified, and the initial data detection model including the modified network parameters is determined as a data detection model; the data detection model is used to detect salient objects contained in multimedia data; Wherein, in the initial saliency prediction component of the initial data detection model, obtaining the sample content features corresponding to the sample data includes: Inputting the sample data into an initial saliency prediction component in the initial data detection model, encoding the sample data according to a saliency encoder in the initial saliency prediction component, and obtaining a sample convolution content feature corresponding to the sample data; According to the convolutional recurrent neural network in the initial saliency prediction component, the sample convolution content feature and the implicit state feature of the previous sample data are bidirectionally encoded to obtain the sample content feature corresponding to the sample data.

10. The method according to claim 9, characterized in that The initial excitation prediction component includes an initial spatial excitation prediction component and an initial temporal excitation prediction component; The method of obtaining, in the initial excitation prediction component of the initial data detection model, sample data structure features corresponding to the sample data and generating a sample excitation image corresponding to the sample data structure features includes: Inputting the sample data into the initial spatial excitation prediction component in the initial data detection model, and obtaining a sample spatial excitation image corresponding to the sample data in the initial spatial excitation prediction component; Obtaining a sample optical flow image corresponding to the sample data, inputting the sample optical flow image into the initial time excitation prediction component, and obtaining a sample motion feature corresponding to the sample optical flow image in the initial time excitation prediction component; The sample motion feature is enhanced according to the sample space excitation image to obtain a sample time feature corresponding to the sample data, and a sample time excitation image corresponding to the sample data is generated according to the sample time feature.

11. The method according to claim 10, characterized in that The method of modifying the network parameters in the initial data detection model according to the sample excitation image, the sample region image, and the sample labels corresponding to the sample data, and determining the initial data detection model including the modified network parameters as the data detection model, comprises: Acquire a sample label corresponding to the sample data, and determine a first loss function corresponding to the initial spatial excitation prediction component according to an error between the sample spatial excitation image and the sample label; Determining a second loss function corresponding to the initial time excitation prediction component according to an error between the sample time excitation image and the sample label; Determining a third loss function corresponding to the initial saliency prediction component according to an error between the sample region image and the sample label; According to the first loss function, the second loss function and the third loss function, the network parameters in the initial data detection model are corrected, and the initial data detection model including the corrected network parameters is determined as the data detection model.

12. The method according to claim 11, characterized in that The determining, according to the error between the sample spatial excitation image and the sample label, a first loss function corresponding to the initial spatial excitation prediction component comprises: Acquire the category probability distribution corresponding to the sample space stimulus image according to the sample label, and generate a cross entropy loss function according to the category probability distribution; The sample space stimulus image is divided into K stimulus image blocks, and the sample label is divided into K label blocks, and a structural similarity loss function is generated according to the structural similarity between the K stimulus image blocks and the K label blocks; K is a positive integer; Obtaining a union and an intersection between the sample space stimulus image and the sample label, and generating an intersection-over-union loss function according to the union and the intersection; The cross entropy loss function, the structural similarity loss function and the intersection-over-union loss function are determined as the first loss function corresponding to the initial spatial excitation prediction component.

13. The method according to claim 9, characterized in that Also includes: Obtaining the number of training times of the initial data detection model; When the number of training times is less than or equal to a first threshold, enhancing the sample content feature according to the sample label to obtain a sample incentive content feature; When the number of training times is greater than the first threshold and the number of training times is less than or equal to the second threshold, a candidate stimulus image is generated according to the course learning rate, the sample label and the sample stimulus rate, and feature enhancement is performed on the sample content feature according to the candidate stimulus image to obtain a sample stimulus content feature; When the number of training times is less than the second threshold, the step of enhancing the sample content features according to the sample excitation image to obtain the sample excitation content features is performed.

14. A data processing device, characterized in that: include: An incentive image generation module, used to obtain data structure features corresponding to multimedia data and generate an incentive image corresponding to the data structure features; The excitation image is used to highlight the predicted area where the salient object in the multimedia data is located; A first excitation module, configured to obtain a significant content feature corresponding to the multimedia data, and enhance the significant content feature according to the excitation image to obtain an excitation content feature; A saliency map generation module, used to generate an object region image corresponding to the stimulus content feature; the object region image is used to detect the salient object in the multimedia data; The multimedia data includes video frames T in the video data. n , n is a positive integer less than or equal to the number of video frames into which the video data is divided; The first incentive module includes: A saliency extraction unit is used to extract the video frame T n The saliency prediction component in the data detection model is inputted, and the video frame T is processed according to the saliency encoder in the saliency prediction component. n Perform feature extraction to obtain the video frame T n The corresponding content convolution features; A hidden state acquisition unit is used to acquire the video frame T in the video data according to the convolutional recurrent neural network in the saliency prediction component. n-1 The corresponding implicit state features; A time information adding unit, configured to generate the video frame T according to the implicit state feature and the content convolution feature. n Corresponding salient content features.

15. A data processing device, characterized in that: include: A sample excitation image generation module is used to obtain sample data, obtain sample data structure features corresponding to the sample data in the initial excitation prediction component of the initial data detection model, and generate a sample excitation image corresponding to the sample data structure features; the sample excitation image is used to highlight the prediction area where the sample object in the sample data is located; A second excitation module is used to obtain, in the initial saliency prediction component of the initial data detection model, a sample content feature corresponding to the sample data, and perform feature enhancement on the sample content feature according to the sample excitation image to obtain a sample excitation content feature; A sample image generation module, used to generate a sample region image corresponding to the sample data in the initial saliency prediction component; A network parameter correction module, used to correct the network parameters in the initial data detection model according to the sample stimulus image, the sample region image and the sample label corresponding to the sample data, and determine the initial data detection model including the corrected network parameters as the data detection model; the data detection model is used to detect salient objects included in the multimedia data; Wherein, the second excitation module is used for: Inputting the sample data into an initial saliency prediction component in the initial data detection model, encoding the sample data according to a saliency encoder in the initial saliency prediction component, and obtaining a sample convolution content feature corresponding to the sample data; According to the convolutional recurrent neural network in the initial saliency prediction component, the sample convolution content feature and the implicit state feature of the previous sample data are bidirectionally encoded to obtain the sample content feature corresponding to the sample data.

16. A computer device, characterized in that: The method comprises a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the steps of the method described in any one of claims 1 to 8, or executes the steps of the method described in any one of claims 9 to 13.

17. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, which includes program instructions. When the program instructions are executed by a processor, the steps of the method described in any one of claims 1 to 8 are executed, or the steps of the method described in any one of claims 9 to 13 are executed.

18. A computer program product, characterized in that The computer program product comprises computer instructions, which, when executed by a processor, implement the steps of the method of any one of claims 1 to 8, or perform the steps of the method of any one of claims 9 to 13.

Images