Media retrieval method and device and related product
By using image recognition technology to obtain the morphological features of an object's body parts and performing media retrieval based on feature tags, the problem of users having to manually input keywords in existing technologies is solved, achieving greater universality and accuracy.
Patent Information
- Application Number
- CN202310504757.3
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-06
- Publication Date
- 2024-11-08
AI Technical Summary
Existing media retrieval solutions rely on users inputting keywords or titles, which presents a learning curve and has poor universality and accuracy, especially in multimedia content retrieval where they fail to meet users' personalized needs.
By using image recognition technology to obtain the morphological features of the body parts of an object, and based on feature tags, media content can be retrieved. Feature tags are provided for users to customize and select, allowing media content to be retrieved directly from images, avoiding reliance on user text input.
It improves the universality and accuracy of media retrieval, lowers the retrieval threshold, ensures that retrieval results match the morphological characteristics of users' body parts, and reduces errors caused by inaccurate user morphological characteristics.
Smart Images

Figure CN118916500A_ABST
Abstract
Description
Technical Field
[0001] The present application relates to the field of computer technology, specifically to the field of media retrieval, and in particular to a media retrieval method, a media retrieval device, a computer device, a computer-readable storage medium, and a computer program product. Background Art
[0002] With the popularization and application of multimedia content such as short videos, Vlog (video blog), and Plog (Photo blog), more and more users will search for videos, pictures and other media content on related video platforms to learn about makeup, skin care, hair care and other content in the corresponding media content, which greatly meets the users' learning needs.
[0003] Currently, in media retrieval schemes for data such as videos, users usually retrieve relevant media content based on text content such as keywords and titles. This scheme requires users to understand relevant retrieval rules or learn relevant knowledge, and has a certain learning threshold, so the universality of media retrieval is poor. Summary of the invention
[0004] The embodiments of the present application provide a media retrieval method, device and related products, which can perform media retrieval based on the morphological features of objects in an image, thereby improving the universality and accuracy of media retrieval.
[0005] On the one hand, an embodiment of the present application provides a media retrieval method, the method comprising:
[0006] Get the image containing the object;
[0007] Display at least one feature tag of the object, the feature tag is used to indicate a morphological feature of at least one body part of the object;
[0008] According to the selected feature tags, one or more retrieved media contents are output, wherein the media contents include bloggers matching the selected feature tags.
[0009] On the one hand, an embodiment of the present application provides a media retrieval method, the method comprising:
[0010] receiving an image recognition request sent by a search client, wherein the image recognition request carries an image including an object;
[0011] Performing recognition processing on the image to obtain morphological features of at least one body part of the subject;
[0012] sending the morphological feature to a search client so that at least one feature tag of the object is displayed in the search client based on the morphological feature;
[0013] Receiving a media retrieval request sent by a retrieval client, where the media retrieval request carries at least one feature tag;
[0014] One or more media contents are retrieved based on the selected feature tags, and the one or more media contents are returned to the search client, wherein the media contents include bloggers matching the selected feature tags.
[0015] In one aspect, an embodiment of the present application provides a media retrieval device, the device comprising:
[0016] An acquisition unit, used for acquiring an image containing an object;
[0017] A display unit, used to display at least one feature tag of the object, the feature tag is used to indicate a morphological feature of at least one body part of the object;
[0018] The output unit is used to output the retrieved one or more media contents according to the selected feature tags, wherein the media contents include bloggers matching the selected feature tags.
[0019] In a possible implementation, the image includes at least one of the following:
[0020] The image obtained by calling the camera component to take a picture of the object;
[0021] Receive the image uploaded by the subject;
[0022] Call the camera component to shoot a video of the object to obtain a video, and extract an image from the video;
[0023] The image is acquired when an image shooting event is generated in the search page; the image shooting event includes any of the following: an event generated when a shooting search entry in the search page is triggered; an event generated when an image acquisition operation occurs in the operation area of the search page; the image acquisition operation includes any of the following: a single-click operation, a double-click operation, a long-press operation, a gesture operation, and a floating gesture;
[0024] The method of extracting an image from a video includes any of the following: randomly extracting an image frame from a video as an image; extracting a specified image frame from a video as an image; extracting an image frame that meets a preset image condition from a video as an image;
[0025] The preset image conditions include any one of the following: the image includes a specified body part, the shooting posture is a fixed posture, the shooting angle is a fixed angle, and the image clarity reaches a clarity threshold.
[0026] In a possible implementation, the morphological features of any body part of the object are indicated by one or more feature tags;
[0027] The display unit displays at least one characteristic tag of the object, including at least one of the following:
[0028] Displaying partial or complete feature labels of each body part of the object at the display position of each body part of the object in the image;
[0029] Displaying partial or complete feature labels of each body part of the object at the display position of each body part of the virtual image of the object, respectively, so that the body parts of the virtual image correspond to the body parts of the object one by one;
[0030] Displaying at least one characteristic label of the object in the label display area;
[0031] Displaying a portion of the feature labels of the object in the image, and displaying the remaining portion of the feature labels in the label display area;
[0032] Synchronously displaying at least one characteristic label of the object in the image and label display area;
[0033] Arrange and display at least one characteristic label of the object in the label display area;
[0034] Displaying at least one characteristic label of the object in a label display area according to body part clustering;
[0035] The label display area includes at least one of the following: an interface independent of the image display interface; a window independent of the image display interface; a fixed area defined in the image display interface; a floating layer suspended on the image display interface.
[0036] In a possible implementation, the media retrieval apparatus further includes a processing unit, which is configured to perform the following operations:
[0037] In response to a zooming operation on a body part of the object, the displayed number of feature labels corresponding to each body part of the object is dynamically adjusted following the zooming operation.
[0038] In a possible implementation, the processing unit is further configured to perform the following operations:
[0039] In response to a circle operation on a body part of the subject, highlighting the circled body part;
[0040] Among them, the highlighted display includes any one or more of the following: enlarged display, flashing display, animated display, distinguishing the circled body parts and uncircled body parts in the display object by brightness, and distinguishing the circled body parts and uncircled body parts in the display object by color.
[0041] In a possible implementation, the processing unit is further configured to perform the following operations:
[0042] Following the highlighting of the circled body part, the display mode of the feature label of the circled body part is dynamically adjusted;
[0043] Among them, the dynamic adjustment includes any one or more of the following: dynamically adjusting the display quantity of the feature labels of the circled body parts, dynamically adjusting the display status of the feature labels of the circled body parts, and dynamically adjusting the display form of the feature labels of the circled body parts.
[0044] In a possible implementation, any feature tag supports being edited, and editing includes at least one of the following operations: modification, deletion, addition, and selection;
[0045] If the body part of the subject includes a face, the morphological features of the face indicated by the facial feature label include any one or more of the following: facial skin texture, skin color, organ features, facial morphology, and pore status;
[0046] If the body part of the object includes a head, the morphological features of the head indicated by the feature label of the head include any one or more of the following: head outline, cranial features, hair color type, and hair quality features;
[0047] If the body parts of the object include limbs, the morphological features of the limbs indicated by the feature labels of the limbs include any one or more of the following: limb position, limb outline, limb height, and body shape features.
[0048] In a possible implementation, the feature label is displayed in a display interface of the image, and an inspiration search control is provided in the display interface; the processing unit is further used to perform the following operations:
[0049] Receiving a selection operation of at least one feature tag displayed in the display interface;
[0050] In response to a triggering operation on the inspiration retrieval control, one or more media contents are retrieved based on the selected feature tags;
[0051] The media content includes any one of the following: media content including some selected feature tags, and media content including all selected feature tags.
[0052] In a possible implementation, a directional search control is further provided in the display interface; after receiving a selection operation on at least one feature tag displayed in the display interface, the processing unit is further configured to perform the following operations:
[0053] After detecting that the directional search control is triggered, displaying the input panel;
[0054] In response to a text input operation detected in the input panel, obtaining a search category of the input;
[0055] Retrieve one or more media contents based on the selected feature tags and search categories;
[0056] The media content includes any of the following: media content including some of the selected feature tags and matching the search category, and media content including all of the selected feature tags and matching the search category.
[0057] In a possible implementation, the number of retrieved media contents is P, where P is a positive integer; and the outputting methods of the P media contents include at least one of the following:
[0058] Arrange and display P media contents;
[0059] Surround display of P media contents;
[0060] Clustering displays P media contents;
[0061] Switch to display P media contents;
[0062] The arrangement and display method includes at least one of the following: arranging and displaying in random order, arranging and displaying in order of the release time of the P media contents, arranging and displaying in order of the length of the media duration of the P media contents, and arranging and displaying in order of the feature similarity between the blogger and the object in the P media contents;
[0063] The surround display method includes at least one of the following: taking the media content with the greatest feature similarity to the image as the center point, and the other media content in the P media content except the center point as the points on the circumference; taking the media content with the latest release time as the center point, and the other media content in the P media content except the center point as the points on the circumference; taking the media content with the longest media duration as the center point, and the other media content in the P media content except the center point as the points on the circumference;
[0064] The cluster display method includes at least one of the following: cluster display according to the type of the P media contents, cluster display according to the source of the P media contents, cluster display according to the media duration of the P media contents, and cluster display according to the release time of the P media contents;
[0065] The switching display method includes at least one of the following: switching and displaying P media contents according to a specified number, switching and displaying P media contents according to a specified duration, switching and displaying P media contents according to type, and switching and displaying P media contents according to source.
[0066] In a possible implementation, the retrieved media content is P videos, where P is a positive integer; the output unit outputs one or more media content retrieved based on the feature tag, and is used to perform the following operations:
[0067] Displaying P videos retrieved based on feature tags in the same display interface;
[0068] Play P videos in a preset manner;
[0069] Among them, the preset method includes any of the following: playing P videos in sequence according to the arrangement order of each video, playing P videos in sequence according to the video length of each video, playing P videos in sequence according to the release time of each video, and playing P videos in sequence at a fixed time interval.
[0070] In a possible implementation, the retrieved media content is P videos, where P is a positive integer; any video has one or more video segments that match the feature tag;
[0071] The output unit outputs one or more media contents retrieved based on the feature tag, including at least one of the following:
[0072] Output one or more video clips in P videos that match the feature labels;
[0073] Output P videos, and highlight one or more corresponding video clips in each video;
[0074] The output mode of each video segment in the P videos includes any one of the following: automatically playing each video segment, randomly playing each video segment, playing each video segment according to a fixed duration, and playing each video segment according to a selected feature tag;
[0075] The highlighting includes any of the following: highlighting a progress bar corresponding to a video clip, magnifying and displaying the screen content corresponding to the video clip, animating and displaying the screen content corresponding to the video clip, and distinguishing the color or brightness of the screen content of the video clip.
[0076] In a possible implementation, the processing unit is further configured to perform the following operations:
[0077] During the playback of the video clip, a feature tag matching the video clip is displayed;
[0078] In response to a trigger operation of the feature tag, outputting a media result indicated by the trigger operation;
[0079] Among them, the media results include any of the following: video images that match the feature tags; retrieved historical media content that matches the feature tags; any one or more of the upcoming live videos, conferences, forums, and lectures that match the feature tags; and the advertiser's main page that matches the feature tags.
[0080] In a possible implementation, the media content includes a makeup and hair video or a dressing video; and the processing unit is further configured to perform the following operations:
[0081] During the playback of the makeup and hair video or the outfit video, in response to an object interaction event, a preview screen is displayed;
[0082] Among them, the preview screen is used to present the video screen after the object in the makeup and hair video or the dressing video is replaced with the object in the image; or, the preview screen is used to present the video screen after the object in the makeup and hair video or the dressing video is replaced with the virtual image of the object in the image.
[0083] In a possible implementation, the processing unit is further configured to perform the following operations:
[0084] Displaying a historical search interface, the historical search interface includes one or more historical search records, and a historical search record is used to record any one or more of the following: feature tags of objects identified during a historical search process, and feature tags selected for retrieval;
[0085] In response to a selection operation on a target historical search record, one or more feature tags in the target historical search record are displayed.
[0086] In a possible implementation, the processing unit is further configured to perform the following operations:
[0087] In response to any one of the historical search records being selected, outputting one or more historical media contents retrieved corresponding to the historical search record;
[0088] Among them, historical search records support any one or more operations of being forwarded, shared, edited, and downloaded.
[0089] In one aspect, an embodiment of the present application provides a media retrieval device, the device comprising:
[0090] A receiving unit, configured to receive an image recognition request sent by a search client, wherein the image recognition request carries an image including an object;
[0091] a processing unit, configured to perform recognition processing on the image to obtain a morphological feature of at least one body part of the object;
[0092] A sending unit, configured to send the morphological feature to a search client, so that at least one feature tag of the object is displayed in the search client based on the morphological feature;
[0093] The receiving unit is further used to receive a media retrieval request sent by a retrieval client, wherein the media retrieval request carries at least one feature tag;
[0094] The processing unit is further configured to retrieve one or more media contents based on the feature tags, and return the one or more media contents to the retrieval client.
[0095] In a possible implementation, the processing unit performs recognition processing on the image to obtain a morphological feature of at least one body part of the object, and performs the following operations:
[0096] Performing virtual modeling processing on the object in the image to obtain a virtual image;
[0097] Performing multi-dimensional feature analysis processing on the virtual image to obtain morphological features of at least one body part of the object, the feature analysis processing including: bone binding processing;
[0098] The morphological features include any one or more of model features, texture features, and bone features.
[0099] In a possible implementation, the feature tag is used to indicate a morphological feature of at least one body part of the object; the processing unit retrieves one or more media content based on the feature tag, and performs the following operations:
[0100] Based on the morphological features indicated by the feature tags, at least one matching model matching the virtual image is retrieved from a virtual model database, wherein the virtual model database includes: a plurality of virtual models obtained by performing virtual modeling processing on objects in a plurality of media contents to be retrieved;
[0101] The media content associated with the matching model is determined as the retrieved media content.
[0102] In a possible implementation, the processing unit retrieves at least one matching model that matches the virtual image from the virtual model database based on the morphological feature indicated by the feature tag, and performs the following operations:
[0103] Acquire the morphological features of each virtual model in the virtual model database;
[0104] Calculating feature similarity between the morphological feature indicated by the feature label and the morphological features of each virtual model in the virtual model database;
[0105] The virtual model corresponding to the feature similarity that meets the feature similarity threshold is determined as the matching model that matches the virtual image.
[0106] In a possible implementation, the morphological features of any virtual model include: any one or more of model features, mapping features, and skeleton features; the process of the processing unit calculating the feature similarity between the morphological features indicated by the feature tag and the morphological features of any virtual model in the virtual model database specifically includes:
[0107] Obtaining a first similarity between a model feature of the virtual image and a model feature of any virtual model;
[0108] Obtaining a second similarity between the texture feature of the virtual image and the texture feature of any virtual model;
[0109] Obtaining a third similarity between the skeleton features of the virtual image and the skeleton features of any virtual model;
[0110] Based on the first similarity, the second similarity, and the third similarity, the feature similarity is calculated.
[0111] On the one hand, an embodiment of the present application provides a computer device, which includes a memory and a processor, wherein the memory stores a computer program, and when the computer program is executed by the processor, the processor executes the above-mentioned media retrieval method.
[0112] On the one hand, an embodiment of the present application provides a computer-readable storage medium, which stores a computer program. When the computer program is read and executed by a processor of a computer device, the computer device executes the above-mentioned media retrieval method.
[0113] On the one hand, an embodiment of the present application provides a computer program product or a computer program, which includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device performs the above-mentioned media retrieval method.
[0114] In an embodiment of the present application, an image containing an object can be obtained, and at least one feature tag of the object can be displayed, and the feature tag is used to indicate the morphological features of at least one body part of the object. Next, one or more retrieved media contents can be output according to the selected feature tags, and the media contents include bloggers matching the selected feature tags. It can be seen that, on the one hand, when performing media retrieval, the present application can complete the retrieval of media content by providing an image. Specifically, the system can analyze the morphological features of each body part of the object in the image, and provide corresponding feature tags for user retrieval, so as to facilitate the user to customize the selection of corresponding feature tags to retrieve matching media content. There is no need for the user to manually enter the keywords and other content of the media retrieval, which can reduce the retrieval threshold and thus improve the universality of media retrieval. On the other hand, in the process of media retrieval, the media content is matched with the morphological features of the user's body parts, avoiding the retrieval error caused by the user's inaccurate understanding of his own morphological features, thereby improving the accuracy of media retrieval. BRIEF DESCRIPTION OF THE DRAWINGS
[0115] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.
[0116] Figure 1 is a schematic diagram of the architecture of a media retrieval system provided in an embodiment of the present application;
[0117] Figure 2 It is a flowchart of a media retrieval method provided in an embodiment of the present application;
[0118] Figure 3a is a schematic diagram of an image acquisition interface provided by an embodiment of the present application;
[0119] Figure 3b is a schematic diagram of another image acquisition interface provided in an embodiment of the present application;
[0120] Figure 3c is a schematic diagram of another image acquisition interface provided in an embodiment of the present application;
[0121] Figure 3d is a schematic diagram of another image acquisition interface provided in an embodiment of the present application;
[0122] Figure 4a This is a schematic diagram of an interface for displaying feature labels provided in an embodiment of the present application;
[0123] Figure 4bIt is another schematic diagram of an interface for displaying feature labels provided in an embodiment of the present application;
[0124] Figure 4c It is another schematic diagram of an interface for displaying feature labels provided in an embodiment of the present application;
[0125] Figure 4d It is another schematic diagram of an interface for displaying feature labels provided in an embodiment of the present application;
[0126] Figure 4e It is another schematic diagram of an interface for displaying feature labels provided in an embodiment of the present application;
[0127] Figure 4f It is another schematic diagram of an interface for displaying feature labels provided in an embodiment of the present application;
[0128] Figure 4g It is another schematic diagram of an interface for displaying feature labels provided in an embodiment of the present application;
[0129] Figure 4h It is another schematic diagram of an interface for displaying feature labels provided in an embodiment of the present application;
[0130] Figure 4i This is a schematic diagram of an interface for retrieving operation history records provided in an embodiment of the present application;
[0131] Figure 4j This is another interface schematic diagram of operation history retrieval records provided by an embodiment of the present application;
[0132] Figure 5a This is a schematic diagram of an interface for adjusting the number of displayed feature tags provided in an embodiment of the present application;
[0133] Figure 5b This is another schematic diagram of an interface for adjusting the number of displayed feature tags provided in an embodiment of the present application;
[0134] Figure 5c is a schematic diagram of an interface for highlighting body parts provided in an embodiment of the present application;
[0135] Figure 5d This is a schematic diagram of an interface for dynamically adjusting the display of feature labels provided in an embodiment of the present application;
[0136] Figure 5e This is another schematic diagram of an interface for dynamically adjusting the display of feature labels provided in an embodiment of the present application;
[0137] Figure 6a It is a schematic diagram of an interface of a media retrieval method provided in an embodiment of the present application;
[0138] Figure 6bis a schematic diagram of an interface of another media retrieval method provided in an embodiment of the present application;
[0139] Figure 6c is a schematic diagram of an interface of another media retrieval method provided in an embodiment of the present application;
[0140] Figure 7a is a schematic diagram of an interface for displaying media content provided by an embodiment of the present application;
[0141] Figure 7b is another schematic diagram of an interface for displaying media content provided by an embodiment of the present application;
[0142] Figure 7c is another schematic diagram of an interface for displaying media content provided by an embodiment of the present application;
[0143] Figure 7d is another schematic diagram of an interface for displaying media content provided by an embodiment of the present application;
[0144] Figure 8a is a schematic diagram of an interface for outputting media content provided by an embodiment of the present application;
[0145] Figure 8b is a schematic diagram of another interface for outputting media content provided by an embodiment of the present application;
[0146] Figure 8c is a schematic diagram of another interface for outputting media content provided by an embodiment of the present application;
[0147] Figure 9a is a schematic diagram of an interface for outputting media results provided in an embodiment of the present application;
[0148] Figure 9b is a schematic diagram of another interface for outputting media results provided in an embodiment of the present application;
[0149] Fig.9c is a schematic diagram of another interface for outputting media results provided in an embodiment of the present application;
[0150] Fig.10 is a schematic diagram of a preview screen provided in an embodiment of the present application;
[0151] Fig.11 is a flowchart of another media retrieval method provided in an embodiment of the present application;
[0152] Fig.12a is a schematic diagram of collecting training data provided by an embodiment of the present application;
[0153] Figure 12b It is a schematic diagram of a process of recognizing a training image provided by an embodiment of the present application;
[0154] Fig.12c is a schematic diagram of a virtual model and object map provided in an embodiment of the present application;
[0155] Fig.12d It is a schematic diagram of a process of performing skeleton binding processing on a virtual image provided by an embodiment of the present application;
[0156] Fig.13 is a schematic diagram of virtual image matching provided in an embodiment of the present application;
[0157] Fig.14 is a structural schematic diagram of a media retrieval device provided in an embodiment of the present application;
[0158] Fig.15 is a schematic diagram of the structure of another media retrieval device provided in an embodiment of the present application;
[0159] Fig.16 It is a structural diagram of a computer device provided in an embodiment of the present application. DETAILED DESCRIPTION
[0160] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of this application.
[0161] The following is an introduction to the key technical terms involved in the embodiments of the present application:
[0162] 1. Morphological characteristics:
[0163] Morphological features refer to features determined after the body parts of the object in the image are identified (such as AI identification, feature extraction, etc.). Among them, the object refers to a living being with behavioral capabilities, such as: humans, animals, etc.; the body parts of the object may include: face, head, limbs, etc. Specifically, the face may include but are not limited to: nose, eyes, ears, mouth, eyebrows, etc.; the head may include but are not limited to: hair, neck, etc.; the limbs may include but are not limited to: hands, feet, waist, head, etc. It should be understood that different body parts reflect different morphological features. For example, if the body parts of the object include the face, the morphological features of the object include at least one or more of the following: facial skin quality (such as dry skin, oily skin, mixed dry skin, mixed oily skin, etc.), skin color (such as yellow skin, cold white skin, warm white skin, black skin, etc.), organ features (such as hooked nose, collapsed nose bridge, big nose tip, double eyelids, phoenix eyes, buck teeth, etc.), facial morphology (such as crow's feet, square face, facial depression, etc.), pore state (such as enlarged pores). For another example, if the body part of the object includes the head, the morphological characteristics of the object include at least one or more of the following: head outline (e.g., big head, round head, etc.), cranial features (e.g., high cranial features), hair color type (e.g., black, brown, yellow, etc.), hair quality features (e.g., fine and soft hair, rough hair, frizzy hair, sparse hair, etc.). For another example, if the body part of the object includes limbs, the morphological characteristics of the object include at least one or more of the following: limb position, limb outline (e.g., head outline, waist outline, leg outline), limb height, body shape features (e.g., fat, thin, slightly fat, normal, etc.). In addition, the morphological characteristics corresponding to a body part can be one or more, for example, the morphological characteristics corresponding to the nose part can include: collapsed nose bridge, large nose wings, wide nose bridge, etc.
[0164] 2. Feature tags:
[0165] The feature tag is used to indicate the morphological features of the body parts of the object, that is, the feature tag is a way of displaying the morphological features. The present application can display the morphological features of each body part of the object in the interface of the retrieval client in the form of corresponding feature tags. Among them, the display form of the feature tag can include, for example, any of the following: rectangle, square, circle, line plus text and other types. The display forms of the feature tags corresponding to different morphological features can be the same or different; in addition, the display color of the feature tag can be any color, such as black, red, yellow, etc. Similarly, the display colors of the feature tags corresponding to different morphological features can be the same or different. Finally, the display state of the feature tag can be a static display or a dynamic display (such as flashing, animation, etc.), and the present application does not make specific limitations on this.
[0166] 3. Media Content:
[0167] The media content is the media data obtained after media retrieval based on the feature tags of one or more selected objects. The type of the media content may include, but is not limited to: any one or more of video (such as long video, short video, film and television series, variety show, vlog, etc.), audio, image (such as plog), text, etc. It should be understood that the media retrieval method in the embodiment of the present application can be applied to different business scenarios, and the types of media content retrieved in different business scenarios may be different. For example, the media retrieval method is applied to makeup and hair scenes (such as makeup scenes, skin care scenes, dressing scenes, etc.), then the retrieved media content may include video data, such as makeup videos, makeup removal videos, skin care videos, dressing videos, etc.; for another example, the media retrieval method is applied to hairdressing scenes (such as hair cutting, hair dyeing, perming, etc.), then the retrieved media content may include video data or picture data. It should be noted that for the convenience of explanation, the embodiments of the present application will be described accordingly with the media content as video data.
[0168] 4. Blogger:
[0169] The so-called blogger is the subject who publishes or creates media content. Generally speaking, the blogger involved in the media content refers to the professional and technical personnel in the field to which the media content belongs (such as the field of beauty and makeup, the field of skin care, the field of dressing, the field of perming and dyeing, etc.). Specifically, different types of media content can include bloggers in different fields. For example, if the media content is a makeup video (or a makeup removal video), the media content may include bloggers in the makeup field; if the media content is a skin care video, the media content may include bloggers in the skin care field; and if the media content is a dressing video, the media content may include bloggers in the dressing field.
[0170] 5. Artificial Intelligence
[0171] Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results. Artificial intelligence technology is an interdisciplinary subject that covers a wide range of fields, including both hardware-level and software-level technologies. The basic technologies of artificial intelligence generally include technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, large media retrieval technology, operation / interaction systems, and mechatronics. Artificial intelligence software technology mainly includes computer vision technology, speech processing technology, natural language processing technology, and machine learning / deep learning.
[0172] 1) AI Model:
[0173] An AI model (also known as a neural network model) is a software system that uses artificial intelligence technology to learn capabilities from historical data and predict new data based on the learned capabilities. AI models are usually implemented based on mathematical methods such as linear regression, decision trees, support vector machines, neural networks, etc. AI models adapt to data features through training (also known as learning) and predict the output of unknown data through reasoning. Trained AI models can be applied to applications in a variety of fields, such as image classification, speech recognition, natural language processing, predictive analysis, etc.
[0174] 2) AI training:
[0175] The AI models used in business scenarios are usually obtained after AI training. The so-called AI training is a process of guiding the AI model on how to learn from data. Among them, this learning process usually includes the following steps:
[0176] 1. Prepare training data, such as images, speech, text, etc.
[0177] 2. Choose a machine learning algorithm, such as neural network, support vector machine, etc.
[0178] 3. Configure the parameters of the AI model, such as learning rate, number of hidden layers, loss function, etc.
[0179] 4. Model training: Let the AI model learn the learning patterns in the training data. For example, if the training data is an image, it can learn how to extract features from the image (such as identifying the target object in the image, analyzing the morphological features of the object's body parts, etc.).
[0180] 5. Evaluate model performance and verify whether the trained model can accurately predict through test data.
[0181] 6. Adjust model parameters and repeat training to improve model performance.
[0182] In summary, AI model training is a cyclic process until the model can predict data with a preset accuracy. The trained model can be used in practical applications and continue to improve performance by learning on new data. In one possible implementation, the embodiment of the present application relates to artificial intelligence technology. Specifically, the image containing the object can be identified and processed based on the deep learning technology in the field of artificial intelligence to obtain at least one feature label of the object. The so-called deep learning (DL) is a machine learning method in the field of artificial intelligence, which performs image recognition, feature extraction and other processing based on a neural network model. Among them, deep learning processes data in layers, from simple feature extraction to complex feature representation and other media retrieval processes. The neural network model that performs deep learning is realized after training based on a large amount of sample data to achieve efficient learning and prediction capabilities of the neural network model. At present, deep learning has achieved remarkable results in many fields, such as computer vision, natural language processing, speech recognition, etc. Therefore, the present application can accurately and quickly realize the recognition and processing of the morphological features of the body parts of the object based on deep learning technology. For example, if the image includes the face of an object, the AI model can be used to identify the object’s facial skin texture, skin color, organ features, facial shape, pore status and other features; for another example, if the image includes the object’s head, the AI model can be used to identify the object’s head outline, skull features, hair color type, hair quality and other features; for another example, if the image includes the object’s limbs, the AI model can be used to identify the object’s limb position, limb outline, limb height, body characteristics and other features.
[0183] Based on the above description, an embodiment of the present application provides a media retrieval scheme, which includes: obtaining an image containing an object, and displaying at least one feature tag of the object, the feature tag is used to indicate the morphological features of at least one body part of the object, and then outputting one or more media contents retrieved based on the feature tag. It can be seen that, on the one hand, when performing media retrieval, the present application can complete the retrieval of media content by providing an image. Specifically, the system can analyze the morphological features of each body part of the object in the image, and provide corresponding feature tags for user retrieval, so as to facilitate the user to customize the selection of corresponding feature tags to retrieve matching media content. There is no need for the user to manually enter keywords and other content for media retrieval, which can lower the retrieval threshold and thus improve the universality of media retrieval. On the other hand, in the process of media retrieval, the media content is matched with the morphological features of the user's body parts, avoiding the retrieval error caused by the user's inaccurate understanding of his or her own morphological features, thereby improving the accuracy of media retrieval.
[0184] It is particularly important to note that in the subsequent specific implementation methods of this application, data related to user information (such as collecting images of the user's face, head, limbs and other body parts) is required. When the above embodiments of this application are applied to specific products or technologies, user permission or consent is required, and the collection, use and processing of relevant data must comply with relevant laws, regulations and standards of relevant countries and regions.
[0185] In order to better understand the media retrieval solution proposed in the embodiment of the present application, the embodiment of the present application is combined with Figure 2 Give an example media retrieval scenario. See Figure 1 ,Should Figure 1 A schematic diagram of the architecture of a media retrieval system provided in an embodiment of the present application is shown; Figure 1 As shown, the media retrieval system may include at least: a terminal device 101 and a server 102, wherein the terminal device 101 and the server 102 may be directly or indirectly connected in a wired or wireless manner, and the embodiment of the present application does not limit the connection method between the target terminal and the server. It should be understood that the embodiment of the present application does not limit the number, type, and naming of the terminal device 101 and the server 102.
[0186] Next, the types and functions of each device in the media retrieval system are introduced in detail:
[0187] 1) Terminal device 101:
[0188] ① The terminal device 101 may include but is not limited to: smart phones (such as Android phones, iOS phones, etc.), tablet computers, portable personal computers, mobile Internet devices (Mobile Internet Devices, MID), smart TVs, vehicle-mounted devices, head-mounted devices, and other smart devices that can perform touch screens. The embodiments of the present application do not limit the type of terminal device; the terminal device includes a display screen, which can be a physical screen, a touch screen, etc. The display screen can be called a terminal screen, a display screen, etc.
[0189] ② A search client with the above-mentioned media search function is running in the terminal device 101, and the search client allows the object to perform a search operation on the media content, so as to retrieve the matching media content. In detail, a search APP (Application) can be run in the search client. The so-called search application refers to a computer program for completing one or more specific tasks (such as the media search mentioned above in this application). By classifying the search application according to different dimensions (such as the operation mode and function of the application), the types of the same search application in different dimensions can be obtained, among which: a. According to the operation mode of the search application, the search application may include but is not limited to: a client installed in the terminal, a small program that can be used without downloading and installing, a web application opened through a browser, etc. b. According to the functional type of the application, the application may include but is not limited to: an instant messaging application, a content interaction application, etc. Among them, the instant messaging application refers to an application for instant communication messages and social interaction based on the Internet. The instant messaging application may include but is not limited to: a social application with communication functions, a map application with social interaction functions, a game application, etc. Content interaction applications refer to applications that can implement content interaction, such as online banking, sharing platforms, personal spaces, news, etc. It should be understood that the present application embodiment does not limit which type of application the search application is, and is specifically described here.
[0190] 2) Server 102:
[0191] ① Server 102 can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers. It can also be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN (Content Delivery Network), as well as big data and artificial intelligence platforms.
[0192] ② Server 102 refers to the background server of the terminal device 101, which is used to interact with the terminal device 101 for data exchange, so as to provide computing and application service support for the terminal device 101. In detail, the server 102 is the background server of the retrieval application running in the terminal device 101, so that the background server is used to provide computing and application service support for the retrieval application running in the terminal device 101. For example, the server 102 can perform feature recognition processing for the image provided in the retrieval application running in the terminal device 101 to identify the morphological features of at least one body part of the object in the image. Subsequently, the server 102 can send the identified morphological features to the terminal device 101 to display them in the terminal device 101 in the form of feature tags, so that the user can customize the selection or edit the corresponding feature tags and then perform the retrieval operation of the media content.
[0193] Optionally, the specific process of performing feature recognition processing on the image by the server 102 involves a large amount of data calculation and data storage services. The above process requires a lot of computer operating costs. Therefore, the present application can implement corresponding image processing and other related operation processes based on cloud computing technology, thereby improving the efficiency of image recognition and processing.
[0194] The media retrieval solution provided in the embodiment of the present application can be jointly executed by the terminal device 101 and the server 102 in the above-mentioned media retrieval system; or, it can be jointly executed by the retrieval application running in the terminal device 101 and the server 102; the embodiment of the present application does not limit the specific method used to execute the media retrieval solution proposed in the embodiment of the present application. Figure 1 Taking the terminal device 101 and the server 102 jointly executing the media retrieval solution proposed in the embodiment of the present application as an example, the specific interaction process involved in the media retrieval solution is introduced accordingly:
[0195] 1) The terminal device 101 can display an image containing an object. Specifically, the image can include at least one body part (such as face, head, limbs, etc.) of the object. In addition, the image can be an image obtained by real-time shooting, the image can also be an image uploaded by the object, and the image can also be an image extracted from a video. This application does not specifically limit the method of obtaining the image.
[0196] 2) The terminal device 101 sends the image to the server 102. The server 102 can perform recognition processing on the received image. The specific recognition method can be to perform target detection, feature extraction and other recognition processing on the image based on an AI model (such as an image recognition model), so as to obtain the morphological characteristics of at least one body part of the object. For example, if the image includes the face of the object, the server 102 can use the features of the facial organs of the identified object as morphological features (such as yellow skin, wide nose, collapsed nose bridge, square face, crow's feet, large pores, etc.); for another example, if the image includes the head of the object, the server 102 can identify the head features as morphological features (such as reddish-brown hair color, long hair, high skull, etc.).
[0197] 3) The server 102 returns at least one morphological feature that has been identified to the terminal device 101. Optionally, when returning the morphological feature, the server 102 may carry the identifier of the terminal device 101 (for example, encrypting the data based on the identifier), so that the identification result can be encrypted and transmitted to the terminal device 101, thereby avoiding data leakage and improving the security of data interaction.
[0198] 4) After receiving the morphological features returned by the server 102, the terminal device 101 can display the morphological features in the display interface with corresponding feature tags. At least one feature tag displayed in the display interface supports editing (modification, addition, deletion, etc.) or selection. For example, the user can select one or more feature tags in the display interface and perform a media retrieval operation based on the selected feature tags.
[0199] 5) When a media retrieval event generated in the display interface is detected, the terminal device 101 may generate a media retrieval request and send the media retrieval request to the server 102, wherein the media retrieval request carries at least one feature tag selected for retrieval.
[0200] 6) The server 102 performs a media retrieval operation based on at least one characteristic tag carried in the media retrieval request in response to the media retrieval request sent by the terminal device 101, thereby retrieving one or more media contents matching the characteristic tags from the media content library. The retrieved media contents may include any type of data such as video, audio, picture, text, etc.
[0201] 7) The server 102 returns the retrieved one or more media contents to the terminal device 101. These media contents may be outputted in the terminal device 101, wherein the outputting methods of the media contents may include but are not limited to: displaying, playing (such as audio playing, video playing, audio and video playing), and the like.
[0202] It can be seen that, on the one hand, when performing media retrieval, the present application can complete the retrieval of media content by providing an image. Specifically, the system can analyze the morphological features of each body part of the object in the image, and provide corresponding feature tags for user retrieval, so that the user can customize the selection of corresponding feature tags to retrieve matching media content. There is no need for the user to manually enter keywords and other content for media retrieval, which can lower the retrieval threshold and improve the universality of media retrieval. On the other hand, in the media retrieval process, the media content is matched with the morphological features of the user's body parts, avoiding retrieval errors caused by the user's inaccurate understanding of his or her own morphological features, thereby improving the accuracy of media retrieval.
[0203] It should be noted that when the embodiments of the present application are applied to specific products or technologies, user permission or consent is required (for example, when calling a camera component to take a picture of an object, the permission or consent of the object is required), and the relevant data collection, use and processing process must comply with relevant laws, regulations and standards of the country and region, comply with the principles of legality, legitimacy and necessity, and do not involve the acquisition of data types prohibited or restricted by laws and regulations. In some optional embodiments, the relevant data (such as media content) involved in the embodiments of the present application is obtained after the object has authorized it separately. In addition, when obtaining the separate authorization of the object, the purpose of the relevant data involved is indicated to the object.
[0204] Based on the media retrieval scheme described above, the embodiment of the present application proposes a more detailed media retrieval method. The media retrieval method proposed in the embodiment of the present application will be described in detail below in conjunction with the accompanying drawings.
[0205] Figure 2 A flow chart of a media retrieval method provided by an embodiment of the present application is shown; the media retrieval method can be performed by Figure 1 The media retrieval method may be performed by the terminal device 101 in the media retrieval system shown in the figure. The media retrieval method may include but is not limited to steps S201-S203:
[0206] S201: Acquire an image containing an object.
[0207] In an embodiment of the present application, an image refers to an image including body parts of an object. Specifically, the image may be: an image including the face of an object (such as nose, eyes, ears, mouth, eyebrows, etc.), or an image including the head of an object (such as hair, neck, etc.), or an image including limbs of an object (such as hands, feet, waist, head, etc.).
[0208] Next, the method of obtaining the image is introduced in detail with reference to the attached figure:
[0209] ① The image obtained by calling the camera component to take a picture of the object.
[0210] Specifically, the camera component may include, but is not limited to: a front camera of a mobile phone, a rear camera of a mobile phone, a camera, a computer camera, and other devices with a picture taking function.
[0211] In one possible implementation, the image is acquired when an image capture event is generated in a search page; the image capture event includes any one of the following: an event generated when a shooting search entry in the search page is triggered; an event generated when an image acquisition operation occurs in the operation area of the search page; the image acquisition operation includes any one of the following: a single-click operation, a double-click operation, a long press operation, a gesture operation, and a floating gesture.
[0212] See also Figure 3a , Figure 3a Schematic diagram of an image acquisition interface provided by an embodiment of the present application. Figure 3aThe page S301 shown in the figure may be a search page, and a shooting search entry is provided in the search page S301. The shooting search entry may include but is not limited to: controls, buttons, options, etc. For example, the shooting search entry is the shooting search control 3011, and the user can click (for example, single-click, double-click, long-press any one of) the shooting search control 3011 to trigger the generation of an image shooting event. Next, in response to the image shooting event detected in the search page, a shooting category interface S302 may be displayed, and at least one option about the image shooting category may be displayed in the shooting category interface S302, including but not limited to: "outfit measurement" option, "beauty and skin measurement" option, "decoration measurement" option, etc., wherein one option corresponds to one shooting category, for example, the "outfit measurement" option is used to correspond to the shooting of the user's full-body image, and the "beauty and skin measurement" option is used to correspond to the shooting of the user's facial image, etc.; in addition, one option can also be used to determine the category of the feature label, for example For example, the feature labels used to determine the object in the "Beauty Measurement" option are: limb position, limb contour (such as head contour, waist contour, leg contour), limb height, body shape (such as fat, thin, slightly fat, normal, etc.); and the feature labels used to determine the object in the "Beauty Skin Test" option are: facial skin type (such as dry skin, oily skin, combination dry skin, combination oily skin, etc.), skin color (such as yellow skin, cold white skin, warm white skin, black skin, etc.), organ characteristics (such as collapsed nose bridge, big nose, double eyelids, almond eyes, buck teeth, etc.), facial shape (such as crow's feet, square and round face, facial depression, etc.), pore status (such as enlarged pores). If the user can select the "Beauty and Skin Testing" option 3021 in the shooting category interface S302, the shooting interface S303 can be displayed. A picture shooting area 3031 is displayed in the shooting interface S303. The user can take corresponding pictures in real time based on the shooting component. Since the shooting category selected by the user is the beauty and skin testing category, the captured image must include a picture of the user's face or face. During the image shooting process, shooting functions such as "countdown shooting" and "beauty shooting" are allowed to be used, so that the image taken by the user is displayed in the picture shooting area 3031 in the shooting interface S303.
[0213] See also Figure 3b , Figure 3b FIG. 1 is another schematic diagram of an interface for image acquisition provided in an embodiment of the present application. Figure 3bThe page S301 in the search page may be a search page, and an operation area 3012 is provided in the search page S301, allowing the user to perform an image acquisition operation in the operation area. For example, when a "V" gesture operation is detected by the user, an image capture event may be generated. Next, in response to the image capture event detected in the search page, a capture mode selection interface S304 may be displayed, and at least one capture mode selection item may be displayed in the capture mode selection interface S304, such as a picture capture mode, a video capture mode, an album upload mode, etc.; after the user selects the picture capture mode 3041, a capture interface S305 may be displayed. Finally, the captured image may be displayed in the capture interface S305.
[0214] ② Receive the image uploaded by the target.
[0215] In one possible implementation, a retrieval page is displayed, in which an image upload entry is provided; in response to a trigger operation on the image upload entry, an image browsing interface is displayed, in which the image browsing interface includes at least one image containing an object; and a user is allowed to select any image from the image browsing interface as an image for media retrieval.
[0216] See also Figure 3c , Figure 3c FIG. 1 is another schematic diagram of an interface for image acquisition provided in an embodiment of the present application. Figure 3c As shown, an image upload entry 3061 is provided in the search page S306. The user can click the image upload entry 3061 to display an image browsing interface S307. The image browsing interface S307 displays multiple images. The user can select an image including an object from the image browsing interface S307 to upload. Optionally, before displaying the image browsing interface S307, the user's authorization or consent is required before accessing the images in the album. Based on this approach, the user's album information can be prevented from being leaked, thereby improving data security.
[0217] ③Call the camera component to shoot a video of the object and extract the image from the video.
[0218] In a possible implementation, the method of extracting an image from a video includes any of the following: randomly extracting an image frame from a video as an image; extracting a specified image frame (e.g., the nth frame, where n is a positive integer) from a video as an image; extracting an image frame that satisfies a preset image condition from a video as an image. The preset image condition includes any of the following: the image screen includes a specified body part (e.g., in a beauty retrieval scenario, the image screen must include the user's face, and in a hairdressing scenario, the image screen must include the user's head, and in a dressing scenario, the image screen must include the user's entire body); the shooting posture is a fixed posture (e.g., frontal shooting, back shooting); the shooting angle is a fixed angle (e.g., horizontal shooting, overhead shooting); the image clarity reaches a clarity threshold. The so-called clarity is an indicator used to measure the clarity of an image. The higher the clarity, the clearer the image and the better the image quality; conversely, the lower the clarity, the blurrier the image and the poorer the image quality.
[0219] See also Figure 3d , Figure 3d FIG. 1 is another schematic diagram of an interface for image acquisition provided in an embodiment of the present application. Figure 3d The page S301 shown in the figure can be a search page, and a shooting search entry is set in the search page S301. The shooting search entry may include but is not limited to: controls, buttons, options, etc. For example, the shooting search entry is a shooting search control 3011, and the user can click (for example, single-click, double-click, long press) the shooting search control 301 to trigger the generation of an image shooting event. Next, in response to the image shooting event detected in the search page, a shooting mode selection interface S308 can be displayed, and at least one shooting mode selection item can be displayed in the shooting mode selection interface S308, such as: picture shooting mode, video shooting mode, album upload mode, etc.; after the user selects the video shooting mode 3081, the shooting interface S309 can be displayed, and the shooting interface S309 can display prompt information 3091, which is used to indicate the video shooting mode, for example, the prompt information can be: please move your head slowly from various angles and press the shutter key to scan. The user can shoot a video according to the video shooting mode indicated by the prompt information, and after the video shooting is completed, the image extracted from the video can be displayed in the shooting interface S3010.
[0220] It should be noted that, in the above process of calling the shooting component to collect user images, the user's authorization or permission must be obtained first before the shooting component can be called to collect images containing objects.
[0221] S202: Display at least one feature tag of the object, where the feature tag is used to indicate a morphological feature of at least one body part of the object.
[0222] In specific implementation, the morphological features of any body part of the object are indicated by one or more feature tags. If the body part of the object includes the face, the morphological features of the face indicated by the feature tag of the face include any one or more of the following: facial skin texture, skin color, organ features, facial morphology, pore status; if the body part of the object includes the head, the morphological features of the head indicated by the feature tag of the head include any one or more of the following: head outline, cranial features, hair color type, hair quality features; if the body part of the object includes limbs, the morphological features of the limbs indicated by the feature tag of the limbs include any one or more of the following: limb position, limb outline, limb height, body shape features.
[0223] (1) The display method of the feature label is introduced in detail with reference to the attached figure:
[0224] ① Displaying partial or complete feature labels of each body part of the object at the display position of each body part of the object in the image.
[0225] See also Figure 4a , Figure 4a is a schematic diagram of an interface for displaying feature tags provided in an embodiment of the present application. Figure 4a As shown, interface S401 is a recognition interface, in which an image containing an object is displayed, and part or all of the feature labels of each body part are respectively displayed at the display position of each body part of the object displayed in the recognition interface S401. Specifically, one body part corresponds to one or more feature labels. For example, if the body part is a nose, the feature label corresponding to the nose (e.g., large nose wing) can be displayed at the display position of the nose in the recognition interface; if the body part is a cheek, the feature label corresponding to the cheek (e.g., square face) can be displayed at the display position of the cheek; if the body part is an eye, the feature label corresponding to the eye (e.g., crow's feet) can be displayed at the display position of the eye; if the body part is a mouth, the feature label corresponding to the mouth (e.g., pouty lips) can be displayed at the display position of the mouth.
[0226] ② Partial or complete feature labels of each body part of the object are displayed at the display position of each body part of the virtual image of the object; the body parts of the virtual image correspond to the body parts of the object one by one.
[0227] Specifically, the so-called virtual image can be generated by virtual modeling of an object based on an AI model. The virtual image refers to a non-real character image, that is, a character image in a virtual scene, such as: a game character image in a game scene, a comic character image in a comic scene, and a VR character image / AR character image in a VR / AR scene. Figure 4b , Figure 4bis another interface schematic diagram of displaying feature tags provided by an embodiment of the present application. Figure 4b As shown, part or all of the feature labels of the various body parts of the object are displayed at the display positions of the various body parts of the virtual image of the object. For example, if the body part is a nose, the feature label corresponding to the nose (e.g., a wide nose bridge) can be displayed at the display position of the nose in the recognition interface; if the body part is a cheek, the feature label corresponding to the cheek (e.g., a square face) can be displayed at the display position of the cheek; if the body part is an eye, the feature label corresponding to the eye (e.g., crow's feet) can be displayed at the display position of the eye.
[0228] ③ Display at least one characteristic label of the object in the label display area.
[0229] In a specific implementation, the label display area includes at least one of the following: an interface independent of the display interface of the image; a window independent of the display interface of the image; a fixed area defined in the display interface of the image; a floating layer suspended on the display interface of the image. The so-called independent interface refers to another interface that is different from and does not intersect with the display interface of the image. For example, if the display interface of the image is interface a, the label display area can be interface b. Interface a does not intersect with interface b and interface a is different from interface b. The so-called independent window refers to a window independent of the display interface of the image. For example, if the display interface of the image is interface a, the label display area can be window c. Interface a does not intersect with window c and interface a and the window belong to different interfaces.
[0230] For example, see Figure 4c , Figure 4c is another interface schematic diagram of displaying feature tags provided by an embodiment of the present application. Figure 4c As shown, the label display area may be another interface S403 independent of the image display interface S403, and at least one characteristic label of the object (eg, large nose wing, wide nose bridge, crow's feet, square face) is displayed in the interface S403.
[0231] For example, see Figure 4d , Figure 4d is another interface schematic diagram of displaying feature tags provided by an embodiment of the present application. Figure 4d As shown, the label display area can be a floating layer 4031 suspended on the display interface of the image, and at least one characteristic label of the object (such as large nose wings, wide nose bridge, crow's feet, square face) is displayed in the floating layer 4031.
[0232] Also, see Figure 4e , Figure 4e is another interface schematic diagram of displaying feature tags provided by an embodiment of the present application. Figure 4eAs shown, the label display area refers to a fixed area 4011 defined in the display interface S401 of the image, and at least one characteristic label of the object (such as raised eyebrows, crow's feet, chin hair, wide nose bridge, large nose wings, square face, and slightly protruding teeth) is displayed in the fixed area 4011.
[0233] ④ Display part of the feature labels of the object in the image, and display the remaining feature labels in the label display area.
[0234] See also Figure 4f , Figure 4f is another interface schematic diagram of displaying feature tags provided by an embodiment of the present application. Figure 4f As shown, feature labels corresponding to facial organs (such as eyes, nose, ears, eyebrows, and mouth) can be displayed in the image. For example, image 4041 can display some feature labels such as large nose wings, crow's feet, and square face; feature labels corresponding to parts such as the head and limbs can be displayed in the label display area. For example, label display area 4042 can display another part of feature labels such as high skull top, black hair, long hair, and round head shape.
[0235] ⑤ Display at least one characteristic label of the object both in the image and in the label display area simultaneously. The label display area is a fixed area defined in the display interface of the image.
[0236] See also Figure 4g , Figure 4g is another interface schematic diagram of displaying feature tags provided by an embodiment of the present application. Figure 4g As shown, if the feature labels of the object include: large nose wings, crow's feet, and square face, the corresponding feature labels (large nose wings, crow's feet, and square face) can be displayed at the display position of each body part in image 4051, and these feature labels (large nose wings, crow's feet, and square face) can be displayed in the label display area 4052.
[0237] Furthermore, the specific display mode of each feature label displayed in the label display area is as follows:
[0238] a. Arrange and display at least one characteristic tag of the object in the tag display area.
[0239] Specifically, the so-called arrangement display may include any of the following: an arrangement from top to bottom, an arrangement from left to right, or a random arrangement. Figure 4d The various feature labels in the label display area 4031 shown can be displayed in random order. Figure 4e In the label display area 4011 shown, the feature labels can be displayed in an arranged manner from left to right.
[0240] b. Displaying at least one characteristic label of the object in the label display area according to the body part cluster.
[0241] Specifically, the feature labels corresponding to one body part can be clustered and displayed, and the feature labels corresponding to another body part can be clustered and displayed. For example, the feature labels corresponding to the eyes include: double eyelids, crow's feet; the feature labels corresponding to the nose include: wide bridge of the nose, large nose wings, collapsed nose bridge; the feature labels corresponding to the face include: square and round face, then Figure 4e As shown, the feature labels corresponding to the eyes (double eyelids, crow's feet) can be displayed in the first row in the label display area 4011; the feature labels corresponding to the nose (wide nose bridge, large nose wings, collapsed nose bridge) can be displayed in the second row in the label display area 4011; and the feature labels corresponding to the face (square and round face) can be displayed in the third row in the label display area 4011.
[0242] ⑥Trigger the display of feature tags based on historical search records.
[0243] In a possible implementation, a historical search interface is displayed, and the historical search interface includes one or more historical search records, and a historical search record is used to record any one or more of the following: feature tags of the object identified in a historical search process, feature tags selected for retrieval; in response to a selection operation on a target historical search record, one or more feature tags in the target historical search record are displayed. See Figure 4h , Figure 4h is another interface schematic diagram of displaying feature tags provided by an embodiment of the present application. Figure 4h As shown, a historical search entry is provided in the search interface S406. For example, the historical search entry is a historical search control 4061. The user can click the historical search control 4061 to trigger the display of the historical search interface S407. A plurality of cards are displayed in the historical search interface S407. Each card represents a historical search record (for example, historical search records 4071, 4072, and 4073). The user can click any historical search record (for example, 4071) to trigger the display of one or more feature tags recorded in the historical search record 4071 in the interface S408.
[0244] Further, in response to any historical search record being selected, one or more historical media contents retrieved corresponding to the historical search record are output; wherein the historical search record supports any one or more operations of being forwarded, shared, edited, and downloaded. Figure 4i , Figure 4i Schematic diagram of an interface for retrieving operation history records provided in an embodiment of the present application. Figure 4iAs shown, the user can click (e.g., long press) any historical search record in the historical search interface S407, so that the interface S409 displays multiple retrieved historical media contents (video 1, video 2, video 3, video 4) corresponding to the historical search record. Figure 4j , Figure 4j 1 is another interface diagram of an operation history retrieval record provided by an embodiment of the present application. Figure 4j As shown, the user can long press any historical search record in the historical search interface S407 to display the forwarding item. The user can trigger the sharing of the selected historical search record to the social session (e.g., XXX group chat) based on the forwarding item (as shown in interface S410), so that other social users can view the corresponding historical media content based on the historical search record.
[0245] (2) The dynamic adjustment process of the feature label display is introduced in detail with reference to the accompanying drawings:
[0246] ① The zoom operation triggers the dynamic adjustment of feature labels:
[0247] In a possible implementation, in response to a zoom operation on a body part of the object, the number of displayed feature labels corresponding to each body part of the object is dynamically adjusted following the zoom operation. In a specific implementation, if a zoom operation is performed on a body part of the object, the number of displayed feature labels corresponding to each body part of the object can be increased following the zoom operation; if a zoom operation is performed on a body part of the object, the number of displayed feature labels corresponding to each body part of the object can be reduced following the zoom operation.
[0248] See also Figure 5a , Figure 5a Schematic diagram of an interface for adjusting the number of displayed feature tags provided in an embodiment of the present application. Figure 5aAs shown, the feature labels displayed in the image in the display interface S501 include: wide nose, crow's feet, and round face; in response to the zoom operation on the body part of the object in the display interface S501, the number of feature labels displayed about the body part of the object can be increased in the display interface S502, for example, the feature labels displayed in the display interface S502 can include: wide nose bridge, raised eyebrows, wide nose, crow's feet, and round face. Optionally, the added feature labels can be highlighted in the display interface S502, for example, the feature labels "wide nose bridge" and "raised eyebrows" that are triggered by the zoom operation can be displayed in a flashing manner, and the original feature labels "wide nose bridge" and "raised eyebrows" can be displayed normally; for example, the feature labels "wide nose bridge" and "raised eyebrows" that are triggered by the zoom operation can be displayed in red, and the original feature labels "wide nose bridge" and "raised eyebrows" can be displayed in black; for example, the fonts of the feature labels "wide nose bridge" and "raised eyebrows" that are triggered by the zoom operation can be displayed in bold, and the original feature labels "wide nose bridge" and "raised eyebrows" can be displayed in normal font size.
[0249] See also Figure 5b , Figure 5b FIG. 1 is another interface diagram for adjusting the display quantity of feature tags provided in an embodiment of the present application. Figure 5b As shown, the feature labels displayed in the image in the display interface S501 include: large nose wings, crow's feet, and square face; in response to a zoom-out operation on a body part of the object in the display interface S501, the number of displayed feature labels about the body part of the object can be reduced in the display interface S503. For example, the feature labels displayed in the display interface S503 may include: large nose wings, and crow's feet.
[0250] ② The circling operation triggers the dynamic adjustment of feature labels:
[0251] In one possible implementation, in response to a circle operation on a body part of an object, the circled body part is highlighted; wherein the highlighted display includes any one or more of the following: enlarged display, flashing display, animated display (for example, rotating or shaking the image corresponding to the circled body part), displaying the circled body part and uncircled body parts of the object by brightness distinction (for example, displaying the brightness of the circled body part as 100, and displaying the brightness of the uncircled body part as 50), displaying the circled body part and uncircled body parts of the object by color distinction (for example, displaying the color of the circled body part as red, and displaying the color of the uncircled body part as black).
[0252] See also Figure 5c , Figure 5cis a schematic diagram of an interface for highlighting body parts provided in an embodiment of the present application. Figure 5c As shown, a circle tool 5011 (e.g., similar to a paintbrush) is provided in the display interface S501 of the image, and the user can click the circle tool 5011 to trigger the circle operation of the body part. For example, the user can circle the nose part of the object based on the circle tool 5011, so that the circled nose part can be focused and enlarged in the display interface S504 (as shown in 5041). Optionally, the circled nose part can also be displayed in a flashing manner, or the circled nose part can be displayed in red, and the other uncircled body parts (e.g., eyebrows, eyes, mouth, ears, etc.) are displayed in black.
[0253] In one possible implementation, the display mode of the feature label of the circled body part is dynamically adjusted as the circled body part is highlighted; wherein the dynamic adjustment includes any one or more of the following: dynamically adjusting the display quantity of the feature label of the circled body part, dynamically adjusting the display state of the feature label of the circled body part (e.g., dynamic display, static display), dynamically adjusting the display form of the feature label of the circled body part (e.g., font, font size, color, etc.). Specifically, if the highlighting mode of the circled body part is: enlarged display, then the display mode of the feature label of the circled body part may be: increase the display number of the feature label of the circled body part (wherein, the feature label additionally displayed in the feature label of the circled body part may be a hidden feature label, or may be a feature label determined after re-identifying the circled body part); if the highlighting mode of the circled body part is: flashing display (or animation display, or color difference display, etc.), then the display mode of the feature label of the circled body part may be: dynamically adjust the display form of the corresponding body part according to the same highlighting mode as the circled body part, for example, if the circled body part is flashing, then the feature label of the circled body part is also flashing; for example, if the circled body part is displayed based on the first color (such as red), then the feature label of the circled body part is also displayed based on red.
[0254] See also Figure 5d , Figure 5d is a schematic diagram of an interface for dynamically adjusting the display of feature labels provided in an embodiment of the present application. Figure 5dAs shown, the user clicks the circle tool 5011 in the display interface S501 to circle the nose part, and then enlarges the nose part in the display interface S504. Then, as the nose part is enlarged, the number of feature labels corresponding to the nose part can be increased. For example, before the nose part is circled, the feature labels displayed in the display interface S505 include a feature label of large nose wings. As the nose part is circled, the feature labels displayed in the display interface S506 after the nose part is circled may include three feature labels: wide nose, bulbous nose and large nose wings.
[0255] Optionally, the additional feature labels displayed in the feature labels of the circled parts can also be displayed differently. For example, the feature label of large nose wing is displayed in the default display mode (such as black font, regular font, font size 4, etc.), and the additional feature labels of "wide nose" and "bulky nose" are displayed in red font, Song font, font size 5, etc. Please refer to Figure 5e , Figure 5e is another interface schematic diagram for dynamically adjusting the display of feature tags provided by an embodiment of the present application. Figure 5e As shown, after the nose area is circled in the display interface, the nose area is enlarged and displayed (as shown at 5051 in page S505), and then based on the enlarged display of the circled nose area, the number of feature labels displayed for the nose area is increased, and the feature labels that are additionally displayed due to the circling operation (i.e., wide nose, bulbous nose) are displayed in italics, bold, etc., and the feature labels before the circling operation is performed (i.e., large nose wings) are displayed in the default display mode. Based on this method, it is easy to view the dynamically adjusted feature labels, thereby facilitating the user to analyze the feature labels.
[0256] Through the above method, the circled body parts are highlighted, and then the feature labels of the corresponding body parts can be dynamically adjusted based on the highlighted body parts, so that users can view more detailed local labels and enrich the display method of feature labels.
[0257] (3) The process of media retrieval based on feature tags is described in detail with reference to the accompanying drawings:
[0258] It should be understood that any feature tag supports being edited, and the editing here includes at least one of the following operations: modification, deletion, addition, and selection (such as any one of single selection and multiple selection). Figure 5e As shown in the middle interface S501, the interface S501 is provided with an add control 5012, and the user can click the add control 5012 to customize and add corresponding feature labels, for example, "round head shape", "slightly protruding teeth", etc. (such as Figure 5e(As shown in the interface S505). In one possible implementation, a selection operation for at least one feature tag is received; based on the selected feature tag, one or more media content matching the selected feature tag is retrieved. The matching media content includes any of the following: media content including a selected portion of the feature tags, media content including all of the selected feature tags, and media content including the selected feature tag and the input search category. Specifically, if the media content is a video, then if the video content or title contains the selected feature tag, then the video can be determined as a video matching the selected feature tag; if the media content is a picture, then if the feature tag corresponding to the object in the picture contains the selected feature tag, then the picture can be determined as a picture matching the selected feature tag.
[0259] ① Edit the feature tag in the tag display area and perform media search.
[0260] In a possible implementation, the feature tags are displayed in the display interface of the image, and an inspiration search control is provided in the display interface. First, a selection operation of at least one feature tag displayed in the display interface may be received; then, in response to a trigger operation of the inspiration search control, one or more media contents are retrieved based on the selected feature tags; wherein the media contents include any of the following: media contents including some of the selected feature tags, and media contents including all of the selected feature tags. See Figure 6a , Figure 6a Schematic diagram of an interface of a media retrieval method provided in an embodiment of the present application. Figure 6a As shown, the feature tags displayed in the tag display area support selection (single or multiple selection), and the display interface S601 is provided with an inspiration retrieval control 6011. The user can directly click on the feature tags for media retrieval in the tag display area (for example, select "wide nose bridge", "big nose wings", "square and round face"), and then click on the inspiration retrieval control 6011 to trigger media retrieval, and display the retrieval result interface S602. The media retrieval interface displays four media contents retrieved based on the selected feature tags (i.e. "wide nose bridge", "big nose wings", "square and round face"). Among them, any one of the media contents includes all the above selected feature tags. Optionally, the user can edit the feature tags in the tag display area, and then perform media retrieval based on the edited feature tags. For example, the user can modify "big nose wings" to "collapsed nose bridge" and add "bulky nose", and then select the corresponding feature tags to trigger media retrieval.
[0261] ② Select feature tags from images for direct media retrieval.
[0262] See also Figure 6b , Figure 6b FIG. 1 is a schematic diagram of an interface of another media retrieval method provided in an embodiment of the present application. Figure 6b As shown, corresponding feature tags are displayed at the display position of each body part in the image, and the user can directly click on the feature tag displayed at the body part to trigger media retrieval. For example, after the user clicks on the feature tag "big nose wing" in the display interface S601, media retrieval can be performed based on "big nose wing", thereby obtaining two media contents matching the big nose wing (as shown in the retrieval result interface S603). It should be noted that when selecting a feature tag in an image to trigger media retrieval, it usually means that media retrieval is automatically triggered after selecting a feature tag displayed in the image.
[0263] ③ Comprehensively perform media retrieval based on the selected feature tags and input retrieval categories.
[0264] In a possible implementation, a directional search control is also provided in the display interface; after receiving a selection operation of at least one feature tag displayed in the display interface, the method further includes: after detecting that the directional search control is triggered, displaying an input panel; in response to a text input operation detected in the input panel, obtaining an input search category; based on the selected feature tag and the search category, retrieving one or more media content; wherein the media content includes any of the following: media content including some of the selected feature tags and matching the search category, and media content including all of the selected feature tags and matching the search category. See Figure 6c , Figure 6c FIG. 1 is a schematic diagram of an interface of another media retrieval method provided in an embodiment of the present application. Figure 6c As shown, a directional search control 6012 is also provided in the display interface S601. After the user selects or edits the corresponding feature tag in the display interface S601, the user can also click on the directional search control 6012 to display the input panel 6041, allowing the user to input the search category for media retrieval in the input panel 6041. For example, in the beauty makeup scene, the user can input search categories such as "whitening" and "concealer", and then after the confirmation operation is detected in the input panel (such as clicking the confirmation control in the input panel), a comprehensive media search based on the selected feature tag (such as "wide nose bridge", "big nose wing", "square and round face") and the input search category (such as "whitening" or "concealer") can be triggered to display the three retrieved media contents in the search result interface S605.
[0265] S203: Retrieve one or more media contents based on the selected feature tags, and return the one or more media contents to the search client, wherein the media contents include bloggers matching the selected feature tags.
[0266] It should be understood that the types of media content may include but are not limited to: any one or more of video, audio, picture, text, etc. Moreover, the types of the retrieved media content may be the same or different.
[0267] (1) The number of retrieved media contents is P, where P is a positive integer. Next, the output method of the retrieved P media contents is described in detail in conjunction with the accompanying drawings:
[0268] ① Arrange and display P media contents.
[0269] Specifically, the arrangement and display methods include at least one of the following: arranging and displaying in random order; arranging and displaying in order of the release time of the P media contents (for example, the media contents with later release time are displayed earlier); arranging and displaying in order of the length of the media duration of the P media contents (for example, the media contents with longer media duration are displayed earlier); arranging and displaying in order of the feature similarity between the blogger and the object in the P media contents (for example, the media contents with higher feature similarity are displayed earlier). Please refer to Figure 7a , Figure 7a is a schematic diagram of an interface for displaying media content provided by an embodiment of the present application. Figure 7a As shown, the media content displayed in interface S701 includes: video 1, video 2, video 3, and video 4. If the media duration t1 of video 1 is greater than the media duration t2 of video 2, the media duration t3 of video 3, and the media duration t4 of video 4, then the four videos may be sorted and displayed as follows: video 1, video 2, video 3, video 4. The specific sorting and display methods may include any one of the following: displaying in sequence from top to bottom, or displaying in sequence from left to right, or displaying in sequence from top to bottom and from left to right respectively.
[0270] ②Surround display of P media contents.
[0271] Specifically, the surround display method includes at least one of the following: taking the media content with the greatest similarity to the image as the center point (the so-called similarity refers to the similarity between the object features in the image and the object features in the media content, which can be calculated by the server, and the specific calculation process will be described in detail on the subsequent technical side), taking the media content with the latest release time as the center point, and taking the media content with the longest media duration as the center point, and taking the media content with the longest media duration as the center point, and taking the media content with the longest media duration as the center point. Please refer to Figure 7b , Figure 7b is another interface schematic diagram for displaying media content provided by an embodiment of the present application. Figure 7bAs shown, assuming that P=5 (video 1, video 2, video 3, video 4, video 5), and the similarities between the object features in these five videos and the object features in the image are s1>s2>s3>s4>s5, then as shown in interface S702, video 1 can be taken as the center of the circle, and video 2, video 3, video 4, and video 5 can all be displayed as points on the circumference.
[0272] ③ Clustering displays P media contents.
[0273] Specifically, the clustering display method includes at least one of the following: clustering display according to the type of P media content (for example, video type media content is one category, image type media content is another category, audio type media content is another category, and text type media content is another category), clustering display according to the source of P media content (for example, media content from the same app is in the same category), clustering display according to the media duration of P media content (for example, media content with a media duration greater than a certain time threshold is in the same category), and clustering display according to the release time of P media content (for example, media content released in the same time period is in the same category). Please refer to Figure 7c , Figure 7c is another schematic diagram of an interface for displaying media content provided by an embodiment of the present application. Figure 7b As shown, assuming that video 1 and video 3 are from app 1, and video 2 and video 3 are from app 2, as shown in interface S703, video 1 and video 3 can be displayed as one category, and video 2 and video 4 can be displayed as another category. It should be noted that media content of the same category can be displayed randomly or in a specified order (e.g., from left to right, from top to bottom).
[0274] ④ Switch to display P media contents.
[0275] Specifically, the switching display method includes at least one of the following: switching to display a specified number of media contents from P media contents (for example, displaying M media contents each time, M is a positive integer and M<P), switching to display different types of media contents from P media contents (for example, after displaying video type media contents, switching to display picture type media contents, and then switching to display text type media contents), switching to display media contents from different sources from P media contents. Figure 7d , Figure 7d is another schematic diagram of an interface for displaying media content provided by an embodiment of the present application. Figure 7dAs shown, assuming that two media contents are displayed in the search result interface each time, video 1 and video 2 may be displayed preferentially in the search result interface (for example, as shown in interface S704), and then video 1 and video 2 may be switched to video 3 and video 4 in the search result interface (for example, as shown in interface S705).
[0276] (2) The retrieved media content is P videos, where P is a positive integer. Next, the output method of P videos is described in detail with reference to the accompanying drawings:
[0277] ①Play P videos.
[0278] In a possible implementation, P videos retrieved based on feature tags are displayed in the same display interface, and the P videos are played in a preset manner. The preset manner includes any of the following: playing the P videos in the order in which they are arranged, playing the P videos in order according to the length of each video (for example, giving priority to playing videos with longer durations), playing the P videos in order according to the release time of each video (for example, giving priority to playing videos with later releases), and playing the P videos in order at fixed time intervals (for example, playing the next video every 10 seconds). Please refer to Figure 8a , Figure 8a is a schematic diagram of an interface for outputting media content provided by an embodiment of the present application, such as Figure 8a As shown in the figure, video 1 can be played first in interface S801; when video 1 is played, video 2 can be played in interface S802; similarly, when video 2 is played, video 3 can be played in interface S803; finally, when video 3 is played, video 4 can be played in interface S804. It should be understood that during the playback of any video, the user is allowed to manually adjust the playback status (e.g., pause, play) and playback progress (e.g., 2.0 times playback) of the video.
[0279] ② In any video, there are one or more video segments that match the feature label, and one or more video segments that match the feature label in P videos are output.
[0280] It is understandable that the video clips matching the feature tag may include any of the following: video clips containing the feature tag (for example, if the feature tag is wide nose bridge, the video screen with wide nose bridge or the audio content with wide nose bridge can be displayed as a video clip containing the feature tag), and the video clip whose matching degree with the feature tag is greater than or equal to the matching degree threshold. It should be understood that the present application does not limit the arrangement method between the video clips. For example, the arrangement method may include but is not limited to: random arrangement, arrangement according to the length of the clip, and arrangement according to the release time of the corresponding video. In specific implementation, for any retrieved video, the system will perform AI analysis on all its video content (including but not limited to: keywords, text, sound, image and other information related to the video), and after comprehensive analysis of the above information, the video clip matching the feature tag can be extracted. For example, in a 3-minute makeup video, 20 seconds of which are about whitening and concealing, the system will extract this 20-second video clip and directly output the video clip. In this way, it can save users time in screening information and improve users' viewing experience.
[0281] See also Figure 8b , Figure 8b is another interface schematic diagram for outputting media content provided by an embodiment of the present application. Figure 8b As shown, assuming that the retrieved media content includes: video 1, video 2, video 3, video 4, where the video segment in video 1 that matches the feature tag is segment 1 (the video content includes a wide nose bridge), the video segment in video 2 that matches the feature tag is segment 2 (the video content includes a wide nose bridge), the video segment in video 3 that matches the feature tag is segment 3 (the video content includes a wide nose), and the video segment in video 4 that matches the feature tag is segment 4 (the video content includes a large nose wing), then the video segments in each video can be directly output in the interface: segment 1, segment 2, segment 3, segment 4. Furthermore, the output method of each video segment in P videos includes any of the following: automatically play each video segment, randomly play each video segment, play each video segment according to a fixed duration, and play each video segment according to a selected feature tag. For example, segment 1 can be automatically played in interface S805; when segment 1 is played, segment 2 can be automatically played in interface S806 after 10 seconds; when segment 2 is played, segment 3 can be automatically played in interface S807 after 10 seconds; when segment 3 is played, segment 4 can be automatically played in interface S808 after 10 seconds.
[0282] ③ Output P videos and highlight one or more corresponding video clips in each video.
[0283] Specifically, the highlighting includes any of the following: highlighting the progress bar corresponding to the video clip, magnifying the screen content corresponding to the video clip, animating the screen content corresponding to the video clip, and distinguishing the color or brightness of the screen content of the video clip. Figure 8c , Figure 8c is another interface schematic diagram for outputting media content provided by an embodiment of the present application. Figure 8c As shown, any retrieved video can be displayed in interface S809, and the progress bar 8091 of the corresponding video segment (for example, including two video segments matching "wide mountain root") can be highlighted in the video, and then the two video segments can be automatically played in interface S809. Of course, the screen content of the video segment can also be displayed in different colors or brightness (for example, the screen content of the video segment is displayed in color, and other screens in the video that are not video segments are displayed in black and white). Based on this method, the screen content of the video segment can be highlighted to enhance the user's viewing experience.
[0284] ④Perform interactive operations based on feature tags.
[0285] In one possible implementation, during the playback of a video clip, a feature tag matching the video clip is displayed; in response to a trigger operation of the feature tag, a media result indicated by the trigger operation is output; wherein the media result includes any one of the following: a video screen matching the feature tag; retrieved historical media content matching the feature tag; any one or more of videos, conferences, forums, and lectures to be broadcast live that match the feature tag; an advertiser's main page matching the feature tag.
[0286] See also Figure 9a , Figure 9a is a schematic diagram of an interface for outputting media results provided in an embodiment of the present application. Figure 9a As shown, during the playback of a video clip, a feature tag matching the video clip may be displayed (for example, the feature tags in interface S901 are: wide nose bridge, large nose wing, wide nose), and the user may click on the feature tag "wide nose bridge" to switch the current video screen (such as the screen shown in interface S901) to the video screen matching "wide nose bridge" (such as the screen shown in interface S902). Based on this method, during the video playback, the matching screen may be quickly located based on the feature tags, and the video may be played conveniently and quickly.
[0287] See also Figure 9b , Figure 9b is a schematic diagram of another interface for outputting media results provided by an embodiment of the present application. Figure 9bAs shown, during the playback of the video clip, the feature tags matching the video clip can be displayed (for example, the feature tags: wide nose bridge, large nose wing, wide nose) in the interface S901. The user can click on the feature tag "wide nose bridge" to display the video list 9031 retrieved based on the "wide nose bridge" in the interface S903. The video list 9031 is used to present the retrieved historical media content (such as videos) matching the "wide nose bridge"; or, the video list 9031 can also be used to present the live broadcast that is about to be broadcasted that matches the "wide nose bridge"; or, the video list 9031 can also be used to present the lecture that is about to be broadcasted that matches the "wide nose bridge", etc. It should be understood that the user is allowed to play (or reserve) the corresponding video, live broadcast, or lecture in the video list in the interface S903. In this way, historically retrieved videos, videos that are about to be broadcasted, etc. can be obtained based on the feature tags, enriching the media retrieval method, thereby providing users with richer media resources.
[0288] See also Fig.9c , Fig.9c is a schematic diagram of another interface for outputting media results provided by an embodiment of the present application. Fig.9c As shown, during the playback of the video clip, feature tags matching the video clip may be displayed (for example, the interface S901 displays the feature tags: wide nose bridge, large nose wings, wide nose), and the user may click on the feature tag "wide nose bridge" to display a product link 9041 associated with the feature tag in the video screen. A purchase control 9041 is provided in the product link 9041, and the user may click on the purchase control 9041 to jump to the advertiser's page S905 regarding the product (XXX powder), and the advertiser's page S905 displays detailed information regarding the XXX powder to facilitate the user's subsequent purchase of the corresponding product.
[0289] In a possible implementation, the media content includes a makeup and hair video or a dressing video. Then, during the playback of the makeup and hair video or the dressing video, in response to an object interaction event, a preview screen is displayed. The preview screen is used to present the video screen after the object in the makeup and hair video or the dressing video is replaced with the object in the image; or the preview screen is used to present the video screen after the object in the makeup and hair video or the dressing video is replaced with the virtual image of the object in the image. See Fig.10 , Fig.10 is a schematic diagram of a preview screen provided in an embodiment of the present application. Fig.10As shown, during the playback of a makeup video or a dressing video, an interactive control 10011 may be displayed, and then the user clicks on the interactive control 10011 to generate an object interaction time, which can be displayed in the preview page S1002: the video screen after the object in the current video screen (such as object 1 shown in interface S1001) is converted into object 2 in the image; of course, after the user clicks on the interactive control in the preview page, the original playback interface S1001 can be restored. In the above-mentioned object interaction scene, specifically, if in a makeup scene, object 1 in the video can be converted into object 2 in the image; if in a hairdressing scene, object 1 in the video can be converted into a virtual image corresponding to object 2 in the image. Based on this method, it is convenient for users to preview makeup effects or dressing effects.
[0290] In an embodiment of the present application, an image containing an object can be obtained, and at least one feature tag of the object can be displayed. The feature tag is used to indicate the morphological features of at least one body part of the object. Then, one or more media contents retrieved based on the feature tag can be output. It can be seen that, on the one hand, when performing media retrieval, the present application can complete the retrieval of media content by providing an image. Specifically, the system can analyze the morphological features of each body part of the object in the image, and provide corresponding feature tags for user retrieval, so that the user can customize the selection of corresponding feature tags to retrieve matching media content. There is no need for the user to manually enter keywords and other content for media retrieval, which can improve the efficiency of media retrieval. On the other hand, during the media retrieval process, the media content is matched with the morphological features of the user's body parts, avoiding retrieval errors caused by the user's inaccurate understanding of his or her own morphological features, thereby improving the accuracy of media retrieval.
[0291] Next, the background technology involved in the media retrieval solution provided in the embodiment of the present application is described in conjunction with the accompanying drawings. Fig.11 , Fig.11 is a flow chart of another media retrieval method provided in an embodiment of the present application, the media retrieval method is applied to Figure 1 The server in the media retrieval system shown in FIG. Fig.11 As shown, the media retrieval method may specifically include steps S1101-S1105:
[0292] S1101: receiving an image recognition request sent by a search client, wherein the image recognition request carries an image including an object.
[0293] In one possible implementation, when an image is received in a retrieval client, generation of an image recognition request may be triggered, wherein the method of obtaining the image may include but is not limited to at least one of the following: calling a camera component to take a picture of an object to obtain an image; receiving an image uploaded by the object; calling a camera component to take a video of the object to obtain a video, and extracting the image from the video.
[0294] S1102: Perform recognition processing on the image to obtain morphological features of at least one body part of the object.
[0295] In a possible implementation, the server performs recognition processing on the image to obtain the morphological features of at least one body part of the object in the following specific process: performing virtual modeling processing on the object in the image to obtain a virtual image; performing multi-dimensional feature analysis processing on the virtual image to obtain the morphological features of at least one body part of the object, the feature analysis processing includes: bone binding processing; wherein the morphological features include: any one or more of model features, mapping features, and bone features.
[0296] For the convenience of explanation, this application takes the head of an object in an image as an example for explanation. Next, the modeling process of the virtual image of the object (virtual model + object map) is introduced in detail:
[0297] 1.1 AI training data set collection: Collect representative head data as training data, such as head photos or 3D scan data. Fig.12a , Fig.12a is a schematic diagram of collecting training data provided by an embodiment of the present application, such as Fig.12a As shown in the figure, when collecting photos of objects, pictures can be taken from multiple angles for the same object (such as left side shot, front shot, right side shot, etc.), so that the collected training data is richer; in addition, these training data in the training data set must also fully cover head photos of different ages, genders, countries and other categories, especially selecting data set characters with different facial features and makeup features, so that the generated virtual image is more representative and universal.
[0298] 1.2 AI training data set annotation: manually label the images in the training data set, specifically the features of the facial features (such as eyes, nose, mouth, ears, eyebrows, etc.), makeup features, facial features, such as the user's facial skin texture, facial shape, shape and contour of each part, defects and advantages, pore status, hair quality, etc. Then generate specific feature labels based on the results of manual annotation, such as: dark yellow skin, dry and oily mixed skin, hooked nose, collapsed nose bridge, sparse hair, etc.
[0299] 1.3 AI training data set production: semi-manual and semi-automatic processing of training images in the training data set (see Figure 12b , Figure 12b is a flow chart of a method for recognizing a training image provided by an embodiment of the present application, such as Figure 12b As shown in the figure, for the self-portrait data (multiple pictures) collected by the user calling the shooting component, automatic recognition processing can be performed based on the automatic frame selection algorithm, geometric modeling algorithm, and high-definition texture synthesis algorithm, and the results obtained after the above automatic recognition processing can be manually reviewed and corrected), thereby obtaining a 3D model and object texture corresponding to the person in the data set. Please refer to Fig.12c , Fig.12c It is a schematic diagram of a virtual model and object map provided in an embodiment of the present application.
[0300] 1.4 AI model training: A deep learning algorithm is used for model training, wherein the model structure of the AI model used in the embodiment of the present application can be, for example, a convolutional neural network (CNN) or a generative adversarial network (GAN). The present application does not specifically limit the model structure of the AI model; then, the CNN model or the GAN model is used to train the collected data set. Specifically, during the model training process, the model will learn how to extract features from the input head image to generate the corresponding virtual image.
[0301] 1.5 AI parameter adjustment: After the model training is completed, the model parameters need to be adjusted and optimized to ensure that the generated virtual image has sufficient realism and details. Therefore, the process of model parameter adjustment is usually a cyclic process, which usually requires multiple tests and modifications until the adjusted model meets the model convergence conditions, and then the model training is stopped.
[0302] 1.6 AI technology automatically generates virtual images: Using a trained AI model, input an image containing a head, and the model can automatically generate a corresponding virtual image. This process can be achieved through a specific software tool (integrating the trained AI model into the software tool), or directly using the AI model in the game engine to generate the corresponding virtual image.
[0303] In the above manner, the object in the image can be virtually modeled based on the trained AI model to obtain a virtual image of the object. It should be understood that the quality and realism of the generated virtual image depends largely on the quality and diversity of the training data set, as well as the precision and accuracy of the deep learning algorithm. Therefore, before generating a virtual image, it is necessary to fully investigate and evaluate the data set and algorithm to ensure that the final generated virtual image has sufficient quality and realism.
[0304] For further information, see Fig.12d , Fig.12d 1 is a schematic diagram of a process for performing skeleton binding processing on a virtual image provided by an embodiment of the present application. Fig.12d As shown, after obtaining the virtual image, the virtual image can be subjected to skeleton binding processing to obtain the skeleton features of the object. Among them, the so-called facial skeleton binding is a technology in computer graphics, which is used to combine the skeleton in the 3D face model with the facial expression animation. This technology aims to make the facial animation more realistic and natural, so that the facial expression of the virtual image can express emotions and tone more vividly. Facial skeleton binding is widely used in game development, virtual reality (VR), augmented reality (AR), movie special effects and other fields. It is one of the important technologies to achieve high-quality facial animation. In facial skeleton binding, you first need to establish a 3D model of the face image (such as Fig.12c The process involves creating a virtual image in the 3D model, and then marking the position and range of motion of the bones in the 3D model; then, combining these bones with facial expression animation to achieve fine control of facial expressions (for example, when simulating a smile, the mouth and eyes in the 3D model will change according to the set bone movement trajectory, thereby presenting a realistic smile effect); finally, generating the object's bone features based on the marked bone position, range of motion, and facial expression animation data.
[0305] In addition, in the above virtual modeling process, feature extraction can be performed on the virtual image to obtain model features, and feature extraction can be performed on the object texture to obtain texture features. In summary, bone features, model features, and texture features are collectively referred to as morphological features of the object.
[0306] S1103: Send the morphological feature to the search client, so that at least one feature tag of the object is displayed in the search client based on the morphological feature.
[0307] It should be noted that the feature tag is used to indicate the morphological features of at least one body part of the object. After receiving the morphological features returned by the server, the search client can display the corresponding morphological features in the interface based on the feature tag. The specific display method of the feature tag can be found in the relevant description in S202 of the above embodiment, and the embodiment of this application will not be repeated here.
[0308] S1104: Receive a media retrieval request sent by a retrieval client, where the media retrieval request carries at least one feature tag.
[0309] Among them, the feature tag carried in the media retrieval request refers to one or more feature tags selected by the user for media retrieval. Optionally, after receiving the media retrieval request, the server can obtain the identification of the retrieval client and verify the retrieval client based on the identification (any one or more of security verification and legality verification). If it is determined that the verification of the retrieval client passes, step S1104 is triggered; if the verification of the retrieval client fails, the media retrieval request is deleted. In this way, the identity of the retrieval client can be authenticated before the media retrieval to improve the security of the media retrieval process.
[0310] S1105: retrieve and obtain one or more media contents based on the feature tags, and return the one or more media contents to the search client.
[0311] In a possible implementation, one or more media contents are retrieved based on feature tags, and the specific process is as follows: first, based on the morphological features indicated by the feature tags, at least one matching model matching the virtual image is retrieved from a virtual model database, and the virtual model database includes: multiple virtual models obtained after virtual modeling of objects in multiple media contents to be retrieved; then, the media content associated with the matching model is determined as the retrieved media content. It should be noted that the specific process of virtual modeling of objects in the media content to be retrieved can be detailed with reference to the virtual modeling process of objects in the image in step S1102, and the embodiments of the present application will not be repeated here.
[0312] In specific implementation, based on the morphological features indicated by the feature tags, at least one matching model that matches the virtual image is retrieved from the virtual model database, which may include the following steps: first, the morphological features of each virtual model in the virtual model database are obtained; then, the feature similarity between the morphological features indicated by the feature tags and the morphological features of each virtual model in the virtual model database is calculated; finally, the virtual model corresponding to the feature similarity that meets the feature similarity threshold is determined as the matching model that matches the virtual image. The specific steps of obtaining the morphological features of any virtual model can be referred to in detail in the relevant steps of extracting the morphological features of the object in step S1102, and the embodiments of the present application will not be repeated here.
[0313] In summary, the steps involved in media retrieval are as follows:
[0314] ①Matching between virtual image and virtual model:
[0315] Specifically, in the embodiment of the present application, the technology in step S1102 can be used to generate a corresponding virtual model for each media content publishing object, thereby constructing a virtual model database, wherein the media content may include but is not limited to: vlog (video blog), plog (picture blog), short video, long video. Fig.13 , Fig.13 is a schematic diagram of a virtual image matching provided by an embodiment of the present application. Fig.13 As shown, when a user's image is input, a matching search operation of a virtual model can be performed, so that at least one matching model matching the virtual image of the object in the image can be retrieved. For example, when pictures of object 1 at various angles are input, virtual model 1 can be matched; and when pictures of object 2 at various angles are input, virtual model 2 can be matched. Optionally, the feature tags corresponding to the matching model (such as feature tags recognized by the image and feature tags added by the user) can be added to the feature tags of the input user image, so as to enrich the source of feature tags of the objects determined by the present application.
[0316] ② Calculate feature similarity:
[0317] Specifically, the morphological features of any virtual model include: any one or more of model features, mapping features, and skeletal features. Then, the process of calculating the feature similarity between the morphological features indicated by the feature tag and the morphological features of any virtual model in the virtual model database may include: first, obtaining a first similarity between the model features of the virtual image and the model features of any virtual model; then, obtaining a second similarity between the mapping features of the virtual image and the mapping features of any virtual model; next, obtaining a third similarity between the skeletal features of the virtual image and the skeletal features of any virtual model; finally, based on the first similarity, the second similarity, and the third similarity, calculating the feature similarity (for example, the first similarity, the second similarity, and the third similarity may be weighted averaged to obtain the feature similarity; for example, the maximum similarity may be selected from the first similarity, the second similarity, and the third similarity as the feature similarity).
[0318] It should be understood that in the calculation process of the first similarity, the second similarity, and the third similarity, the same similarity algorithm may be used, or different similarity algorithms may be used, wherein the similarity algorithm may include, but is not limited to, Euclidean distance, cosine similarity, and the like.
[0319] ③Find matching pictures similar to the object:
[0320] Specifically, according to the feature similarity obtained above, the face image of the virtual model corresponding to the feature similarity that meets the similarity threshold is used as a matching image similar to the object. This process can be sorted using a sorting algorithm (such as quick sort, heap sort, etc.), or a data structure such as a KD tree (K-dimensional tree, multidimensional binary tree) can be used to search and match images. It should be noted that when using AI technology to search for face images similar to the input image from the virtual model database, the quality and accuracy of the data need to be considered; in addition, due to the high dimensionality of the facial feature vector, high-performance computers and GPUs (Graphic Processing Units) are required to accelerate the calculation.
[0321] ④ Output the media content retrieved from the media:
[0322] Specifically, media retrieval results can be provided based on feature similarity and recommendation algorithms. When the face photo input by the user matches multiple face pictures in the database, the final media content can be further output based on the feature tags selected by the user in the front-end interface. For example, if the user wants to retrieve an object close to his nose or an object close to the wrinkles at the corners of his eyes, a video of a person matching the user's nose or a video of a person matching the wrinkles at the corners of his eyes can be output. Among them, the media content is retrieved from the blogger database, which refers to the media content published by bloggers in various fields (such as vlog bloggers, plog bloggers, beauty bloggers, skin care bloggers, entertainment bloggers, fashion bloggers, game bloggers, etc.). In the embodiment of the present application, for the blogger side, the blogger does not need to spend extra effort to match all relevant keywords; at the same time, it will not be limited by different language backgrounds; it can also increase the exposure rate of the media content (such as vlog videos) published by the blogger and increase media revenue.
[0323] In a possible implementation, if the media content is a video, the AI picture analysis technology can be used to analyze and process the media content to obtain video clips that match the feature tags, and then the corresponding video clips are extracted from the video based on the editing tool and sent to the retrieval client. Among them, the so-called AI picture analysis technology refers to the technology that uses artificial intelligence technology to analyze and identify objects, scenes, expressions and other information in the image. For example, AI picture analysis technology may include but is not limited to: image classification, target detection, semantic segmentation and other technologies. Specifically, AI picture analysis technology is usually implemented using the following technologies: image recognition technology (i.e., using deep learning algorithms to identify objects in images), video analysis technology (i.e., identifying dynamic events in videos), target detection technology (i.e., identifying targets in images or videos, such as people, vehicles, etc.), visual language technology (i.e., converting images into text descriptions based on deep learning algorithms), and computer vision technology (i.e., using image processing technology and deep learning algorithms to identify images). In this way, the user's search and browsing time can be reduced, helping users to accurately reach the video content they want to learn and the bloggers with matching features.
[0324] In the embodiment of the present application, the user is allowed to upload the facial image taken by the user in the video (e.g., short video / live broadcast) retrieval scenario, and the system matches the bloggers with relevant facial / skin features and other morphological features, and can also select the video content corresponding to the blogger based on the target question entered by the user, so that the user can more accurately find the media content (e.g., beauty video / skin care video / beauty products, etc.) that matches his facial features. On the one hand, for the user side, the threshold of media retrieval can be lowered, and the user does not need to learn too much relevant knowledge, and does not need additional learning during the retrieval process; and the user will not be limited by language restrictions and inefficiencies when searching; it can also reduce the user's retrieval and browsing time, and help users accurately reach the desired content and bloggers with matching features. On the other hand, for the blogger side, the blogger does not need to spend extra effort to match all relevant keywords; at the same time, it will not be limited by different language backgrounds; it can also increase the exposure rate of the media content (such as vlog videos) published by the blogger, and increase media revenue. On the other hand, for advertisers of beauty and skin care products, their products are also more likely to reach core users with corresponding needs, improving the accuracy of advertising.
[0325] The above describes in detail the method of the embodiment of the present application. In order to facilitate the better implementation of the above scheme of the embodiment of the present application, the device of the embodiment of the present application is provided below accordingly. Next, in combination with the media retrieval scheme provided in the above embodiment of the present application, the relevant device of the embodiment of the present application is introduced accordingly.
[0326] See also Fig.14 , Fig.14 Schematic diagram of the structure of a media retrieval device provided in an embodiment of the present application. Fig.14 As shown, the media retrieval device 1400 can be applied to the background server mentioned in the above-mentioned embodiment. Specifically, the media retrieval device 1400 can be a computer program (including program code) running in a computer device, for example, the media retrieval device 1400 is an application software; the media retrieval device 1400 can be used to execute the corresponding steps in the media retrieval method provided in the embodiment of the present application, and the media retrieval device 1400 can specifically include:
[0327] An acquisition unit 1401 is used to acquire an image containing an object;
[0328] A display unit 1402 is used to display at least one feature tag of the object, where the feature tag is used to indicate a morphological feature of at least one body part of the object;
[0329] The output unit 1403 is configured to output one or more media contents retrieved based on the feature tags.
[0330] In a possible implementation, the image includes at least one of the following:
[0331] The image obtained by calling the camera component to take a picture of the object;
[0332] Receive the image uploaded by the subject;
[0333] Call the camera component to shoot a video of the object to obtain a video, and extract the image from the video;
[0334] The image is acquired when an image shooting event is generated in the search page; the image shooting event includes any of the following: an event generated when a shooting search entry in the search page is triggered; an event generated when an image acquisition operation occurs in the operation area of the search page; the image acquisition operation includes any of the following: a single-click operation, a double-click operation, a long-press operation, a gesture operation, and a floating gesture;
[0335] The method of extracting an image from a video includes any of the following: randomly extracting an image frame from a video as an image; extracting a specified image frame from a video as an image; extracting an image frame that meets a preset image condition from a video as an image;
[0336] The preset image conditions include any one of the following: the image includes a specified body part, the shooting posture is a fixed posture, the shooting angle is a fixed angle, and the image clarity reaches a clarity threshold.
[0337] In a possible implementation, the morphological features of any body part of the object are indicated by one or more feature tags;
[0338] The display unit 1402 displays at least one feature tag of the object, including at least one of the following:
[0339] Displaying partial or complete feature labels of each body part of the object at the display position of each body part of the object in the image;
[0340] Displaying partial or complete feature labels of each body part of the object at the display position of each body part of the virtual image of the object, respectively, so that the body parts of the virtual image correspond to the body parts of the object one by one;
[0341] Displaying at least one characteristic label of the object in the label display area;
[0342] Displaying a portion of the feature labels of the object in the image, and displaying the remaining portion of the feature labels in the label display area;
[0343] Synchronously displaying at least one characteristic label of the object in the image and label display area;
[0344] Arrange and display at least one characteristic label of the object in the label display area;
[0345] Displaying at least one characteristic label of the object in a label display area according to body part clustering;
[0346] The label display area includes at least one of the following: an interface independent of the image display interface; a window independent of the image display interface; a fixed area defined in the image display interface; a floating layer suspended on the image display interface.
[0347] In a possible implementation, the media retrieval apparatus 1400 further includes a processing unit 1404, which is configured to perform the following operations:
[0348] In response to a zooming operation on a body part of the object, the displayed number of feature labels corresponding to each body part of the object is dynamically adjusted following the zooming operation.
[0349] In a possible implementation, the processing unit 1404 is further configured to perform the following operations:
[0350] In response to a circle operation on a body part of the subject, highlighting the circled body part;
[0351] Among them, the highlighted display includes any one or more of the following: enlarged display, flashing display, animated display, distinguishing the circled body parts and uncircled body parts in the display object by brightness, and distinguishing the circled body parts and uncircled body parts in the display object by color.
[0352] In a possible implementation, the processing unit 1404 is further configured to perform the following operations:
[0353] Following the highlighting of the circled body part, the display mode of the feature label of the circled body part is dynamically adjusted;
[0354] Among them, the dynamic adjustment includes any one or more of the following: dynamically adjusting the display quantity of the feature labels of the circled body parts, dynamically adjusting the display status of the feature labels of the circled body parts, and dynamically adjusting the display form of the feature labels of the circled body parts.
[0355] In a possible implementation, any feature tag supports being edited, and editing includes at least one of the following operations: modification, deletion, addition, and selection;
[0356] If the body part of the subject includes a face, the morphological features of the face indicated by the facial feature label include any one or more of the following: facial skin texture, skin color, organ features, facial morphology, and pore status;
[0357] If the body part of the object includes a head, the morphological features of the head indicated by the feature label of the head include any one or more of the following: head outline, cranial features, hair color type, and hair quality features;
[0358] If the body parts of the object include limbs, the morphological features of the limbs indicated by the feature labels of the limbs include any one or more of the following: limb position, limb outline, limb height, and body shape features.
[0359] In a possible implementation, the feature label is displayed in a display interface of the image, and an inspiration search control is provided in the display interface; the processing unit 1404 is further configured to perform the following operations:
[0360] Receiving a selection operation of at least one feature tag displayed in the display interface;
[0361] In response to a triggering operation on the inspiration retrieval control, one or more media contents are retrieved based on the selected feature tags;
[0362] The media content includes any one of the following: media content including some selected feature tags, and media content including all selected feature tags.
[0363] In a possible implementation, a directional search control is further provided in the display interface; after receiving a selection operation on at least one feature tag displayed in the display interface, the processing unit 1404 is further configured to perform the following operations:
[0364] After detecting that the directional search control is triggered, displaying the input panel;
[0365] In response to a text input operation detected in the input panel, obtaining a search category of the input;
[0366] Retrieve one or more media contents based on the selected feature tags and search categories;
[0367] The media content includes any of the following: media content including some of the selected feature tags and matching the search category, and media content including all of the selected feature tags and matching the search category.
[0368] In a possible implementation, the number of retrieved media contents is P, where P is a positive integer; and the outputting methods of the P media contents include at least one of the following:
[0369] Arrange and display P media contents;
[0370] Surround display of P media contents;
[0371] Clustering displays P media contents;
[0372] Switch to display P media contents;
[0373] The arrangement and display method includes at least one of the following: arranging and displaying in random order, arranging and displaying in order of the release time of the P media contents, arranging and displaying in order of the length of the media duration of the P media contents, and arranging and displaying in order of the feature similarity between the blogger and the object in the P media contents;
[0374] The surround display method includes at least one of the following: taking the media content with the greatest feature similarity to the image as the center point, and the other media content in the P media content except the center point as the points on the circumference; taking the media content with the latest release time as the center point, and the other media content in the P media content except the center point as the points on the circumference; taking the media content with the longest media duration as the center point, and the other media content in the P media content except the center point as the points on the circumference;
[0375] The cluster display method includes at least one of the following: cluster display according to the type of the P media contents, cluster display according to the source of the P media contents, cluster display according to the media duration of the P media contents, and cluster display according to the release time of the P media contents;
[0376] The switching display method includes at least one of the following: switching and displaying P media contents according to a specified number, switching and displaying P media contents according to a specified duration, switching and displaying P media contents according to type, and switching and displaying P media contents according to source.
[0377] In a possible implementation, the retrieved media content is P videos, where P is a positive integer; the output unit 1403 outputs one or more media content retrieved based on the feature tag, and is used to perform the following operations:
[0378] Displaying P videos retrieved based on feature tags in the same display interface;
[0379] Play P videos in a preset manner;
[0380] Among them, the preset method includes any of the following: playing P videos in sequence according to the arrangement order of each video, playing P videos in sequence according to the video length of each video, playing P videos in sequence according to the release time of each video, and playing P videos in sequence at a fixed time interval.
[0381] In a possible implementation, the retrieved media content is P videos, where P is a positive integer; any video has one or more video segments that match the feature tag;
[0382] The output unit 1403 outputs one or more media contents retrieved based on the feature tag, including at least one of the following:
[0383] Output one or more video clips in P videos that match the feature labels;
[0384] Output P videos, and highlight one or more corresponding video clips in each video;
[0385] The output mode of each video segment in the P videos includes any one of the following: automatically playing each video segment, randomly playing each video segment, playing each video segment according to a fixed duration, and playing each video segment according to a selected feature tag;
[0386] The highlighting includes any of the following: highlighting a progress bar corresponding to a video clip, magnifying and displaying the screen content corresponding to the video clip, animating and displaying the screen content corresponding to the video clip, and distinguishing the color or brightness of the screen content of the video clip.
[0387] In a possible implementation, the processing unit 1404 is further configured to perform the following operations:
[0388] During the playback of the video clip, a feature tag matching the video clip is displayed;
[0389] In response to a trigger operation of the feature tag, outputting a media result indicated by the trigger operation;
[0390] Among them, the media results include any of the following: video images that match the feature tags; retrieved historical media content that matches the feature tags; any one or more of the upcoming live videos, conferences, forums, and lectures that match the feature tags; and the advertiser's main page that matches the feature tags.
[0391] In a possible implementation, the media content includes a makeup and hair video or a dressing video; the processing unit 1404 is further configured to perform the following operations:
[0392] During the playback of the makeup and hair video or the outfit video, in response to an object interaction event, a preview screen is displayed;
[0393] Among them, the preview screen is used to present the video screen after the object in the makeup and hair video or the dressing video is replaced with the object in the image; or, the preview screen is used to present the video screen after the object in the makeup and hair video or the dressing video is replaced with the virtual image of the object in the image.
[0394] In a possible implementation, the processing unit 1404 is further configured to perform the following operations:
[0395] Displaying a historical search interface, the historical search interface includes one or more historical search records, and a historical search record is used to record any one or more of the following: feature tags of objects identified during a historical search process, and feature tags selected for retrieval;
[0396] In response to a selection operation on a target historical search record, one or more feature tags in the target historical search record are displayed.
[0397] In a possible implementation, the processing unit 1404 is further configured to perform the following operations:
[0398] In response to any one of the historical search records being selected, outputting one or more historical media contents retrieved corresponding to the historical search record;
[0399] Among them, historical search records support any one or more operations of being forwarded, shared, edited, and downloaded.
[0400] In an embodiment of the present application, an image containing an object can be obtained, and at least one feature tag of the object can be displayed. The feature tag is used to indicate the morphological features of at least one body part of the object. Then, one or more media contents retrieved based on the feature tag can be output. It can be seen that, on the one hand, when performing media retrieval, the present application can complete the retrieval of media content by providing an image. Specifically, the system can analyze the morphological features of each body part of the object in the image, and provide corresponding feature tags for user retrieval, so that the user can customize the selection of corresponding feature tags to retrieve matching media content. There is no need for the user to manually enter keywords and other content for media retrieval, which can lower the retrieval threshold and improve the universality of media retrieval. On the other hand, in the process of media retrieval, the media content is matched with the morphological features of the user's body parts, avoiding retrieval errors caused by the user's inaccurate understanding of his or her own morphological features, thereby improving the accuracy of media retrieval.
[0401] See also Fig.15 , Fig.15 is a schematic diagram of the structure of another media retrieval device provided in an embodiment of the present application. Fig.15 As shown, the media retrieval device 1500 can be applied to the background server mentioned in the above-mentioned embodiment. Specifically, the media retrieval device 1500 can be a computer program (including program code) running in a computer device, for example, the media retrieval device 1500 is an application software; the media retrieval device 1500 can be used to execute the corresponding steps in the media retrieval method provided in the embodiment of the present application, and the media retrieval device 1500 can specifically include:
[0402] The receiving unit 1501 is used to receive an image recognition request sent by a search client, wherein the image recognition request carries an image including an object;
[0403] The processing unit 1502 is used to perform recognition processing on the image to obtain the morphological characteristics of at least one body part of the object;
[0404] A sending unit 1503, configured to send the morphological feature to a search client, so that at least one feature tag of the object is displayed in the search client based on the morphological feature;
[0405] The receiving unit 1501 is further configured to receive a media retrieval request sent by a retrieval client, wherein the media retrieval request carries at least one feature tag;
[0406] The processing unit 1502 is further configured to retrieve one or more media contents based on the selected feature tags, and return the one or more media contents to the search client, wherein the media contents include bloggers matching the selected feature tags.
[0407] In a possible implementation, the processing unit 1502 performs recognition processing on the image to obtain a morphological feature of at least one body part of the object, and performs the following operations:
[0408] Performing virtual modeling processing on the object in the image to obtain a virtual image;
[0409] Performing multi-dimensional feature analysis processing on the virtual image to obtain morphological features of at least one body part of the object, the feature analysis processing including: bone binding processing;
[0410] The morphological features include any one or more of model features, texture features, and bone features.
[0411] In a possible implementation, the feature tag is used to indicate a morphological feature of at least one body part of the object; the processing unit 1502 retrieves one or more media content based on the feature tag, and performs the following operations:
[0412] Based on the morphological features indicated by the feature tags, at least one matching model matching the virtual image is retrieved from a virtual model database, wherein the virtual model database includes: a plurality of virtual models obtained by performing virtual modeling processing on objects in a plurality of media contents to be retrieved;
[0413] The media content associated with the matching model is determined as the retrieved media content.
[0414] In a possible implementation, the processing unit 1502 retrieves at least one matching model that matches the virtual image from the virtual model database based on the morphological feature indicated by the feature tag, and performs the following operations:
[0415] Acquire the morphological features of each virtual model in the virtual model database;
[0416] Calculating feature similarity between the morphological feature indicated by the feature label and the morphological features of each virtual model in the virtual model database;
[0417] The virtual model corresponding to the feature similarity that meets the feature similarity threshold is determined as the matching model that matches the virtual image.
[0418] In a possible implementation, the morphological features of any virtual model include: any one or more of model features, mapping features, and skeleton features; the process of the processing unit 1502 calculating the feature similarity between the morphological features indicated by the feature tag and the morphological features of any virtual model in the virtual model database specifically includes:
[0419] Obtaining a first similarity between a model feature of the virtual image and a model feature of any virtual model;
[0420] Obtaining a second similarity between the texture feature of the virtual image and the texture feature of any virtual model;
[0421] Obtaining a third similarity between the skeleton features of the virtual image and the skeleton features of any virtual model;
[0422] Based on the first similarity, the second similarity, and the third similarity, the feature similarity is calculated.
[0423] In the embodiment of the present application, the user is allowed to upload the facial image taken by the user in the video (e.g., short video / live broadcast) retrieval scenario, and the system matches the bloggers with relevant facial / skin features and other morphological features, and can also select the video content corresponding to the blogger based on the target question entered by the user, so that the user can more accurately find the media content (e.g., beauty video / skin care video / beauty products, etc.) that matches his facial features. On the one hand, for the user side, the threshold of media retrieval can be lowered, and the user does not need to learn too much relevant knowledge, and does not need additional learning during the retrieval process; and the user will not be limited by language restrictions and inefficiencies when searching; it can also reduce the user's retrieval and browsing time, and help users accurately reach the desired content and bloggers with matching features. On the other hand, for the blogger side, the blogger does not need to spend extra effort to match all relevant keywords; at the same time, it will not be limited by different language backgrounds; it can also increase the exposure rate of the media content (such as vlog videos) published by the blogger, and increase media revenue. On the other hand, for advertisers of beauty and skin care products, their products are also more likely to reach core users with corresponding needs, improving the accuracy of advertising.
[0424] See also Fig.16 , Fig.16 16 is a schematic diagram of the structure of a computer device provided in an embodiment of the present application. The computer device 1600 is used to execute the steps performed by the terminal device or the server in the aforementioned method embodiment. The computer device 1600 includes: one or more processors 1601; one or more input devices 1602, one or more output devices 1603 and a memory 1604. The above-mentioned processors 1601, input devices 1602, output devices 1603 and memory 1604 are connected via a bus 1605. Specifically, the memory 1604 is used to store a computer program, and the computer program includes program instructions. The processor 1601 and the output device 1603 call the program instructions stored in the memory 1604 to perform the following operations:
[0425] Processor 1601, configured to acquire an image containing an object;
[0426] An output device 1603, used to display at least one feature tag of the object, the feature tag being used to indicate a morphological feature of at least one body part of the object;
[0427] The output device 1603 is further used to output the retrieved one or more media contents according to the selected feature tags, wherein the media contents include bloggers matching the selected feature tags.
[0428] In a possible implementation, the image includes at least one of the following:
[0429] The image obtained by calling the camera component to take a picture of the object;
[0430] Receive the image uploaded by the subject;
[0431] Call the camera component to shoot a video of the object to obtain a video, and extract the image from the video;
[0432] The image is acquired when an image shooting event is generated in the search page; the image shooting event includes any of the following: an event generated when a shooting search entry in the search page is triggered; an event generated when an image acquisition operation occurs in the operation area of the search page; the image acquisition operation includes any of the following: a single-click operation, a double-click operation, a long-press operation, a gesture operation, and a floating gesture;
[0433] The method of extracting an image from a video includes any of the following: randomly extracting an image frame from a video as an image; extracting a specified image frame from a video as an image; extracting an image frame that meets a preset image condition from a video as an image;
[0434] The preset image conditions include any one of the following: the image includes a specified body part, the shooting posture is a fixed posture, the shooting angle is a fixed angle, and the image clarity reaches a clarity threshold.
[0435] In a possible implementation, the morphological features of any body part of the object are indicated by one or more feature tags;
[0436] The output device 1603 displays at least one characteristic tag of the object, including at least one of the following:
[0437] Displaying partial or complete feature labels of each body part of the object at the display position of each body part of the object in the image;
[0438] Displaying partial or complete feature labels of each body part of the object at the display position of each body part of the virtual image of the object, respectively, so that the body parts of the virtual image correspond to the body parts of the object one by one;
[0439] Displaying at least one characteristic label of the object in the label display area;
[0440] Displaying a portion of the feature labels of the object in the image, and displaying the remaining portion of the feature labels in the label display area;
[0441] Synchronously displaying at least one characteristic label of the object in the image and label display area;
[0442] Arrange and display at least one characteristic label of the object in the label display area;
[0443] Displaying at least one characteristic label of the object in a label display area according to body part clustering;
[0444] The label display area includes at least one of the following: an interface independent of the image display interface; a window independent of the image display interface; a fixed area defined in the image display interface; a floating layer suspended on the image display interface.
[0445] In a possible implementation, the processor 1601 is further configured to perform the following operations:
[0446] In response to a zooming operation on a body part of the object, the displayed number of feature labels corresponding to each body part of the object is dynamically adjusted following the zooming operation.
[0447] In a possible implementation, the processor 1601 is further configured to perform the following operations:
[0448] In response to a circle operation on a body part of the subject, highlighting the circled body part;
[0449] Among them, the highlighted display includes any one or more of the following: enlarged display, flashing display, animated display, distinguishing the circled body parts and uncircled body parts in the display object by brightness, and distinguishing the circled body parts and uncircled body parts in the display object by color.
[0450] In a possible implementation, the processor 1601 is further configured to perform the following operations:
[0451] Following the highlighting of the circled body part, the display mode of the feature label of the circled body part is dynamically adjusted;
[0452] Among them, the dynamic adjustment includes any one or more of the following: dynamically adjusting the display quantity of the feature labels of the circled body parts, dynamically adjusting the display status of the feature labels of the circled body parts, and dynamically adjusting the display form of the feature labels of the circled body parts.
[0453] In a possible implementation, any feature tag supports being edited, and editing includes at least one of the following operations: modification, deletion, addition, and selection;
[0454] If the body part of the subject includes a face, the morphological features of the face indicated by the facial feature label include any one or more of the following: facial skin texture, skin color, organ features, facial morphology, and pore status;
[0455] If the body part of the object includes a head, the morphological features of the head indicated by the feature label of the head include any one or more of the following: head outline, cranial features, hair color type, and hair quality features;
[0456] If the body parts of the object include limbs, the morphological features of the limbs indicated by the feature labels of the limbs include any one or more of the following: limb position, limb outline, limb height, and body shape features.
[0457] In a possible implementation, the feature tag is displayed in a display interface of the image, and an inspiration search control is provided in the display interface; the processor 1601 is further configured to perform the following operations:
[0458] Receiving a selection operation of at least one feature tag displayed in the display interface;
[0459] In response to a triggering operation on the inspiration retrieval control, one or more media contents are retrieved based on the selected feature tags;
[0460] The media content includes any one of the following: media content including some selected feature tags, and media content including all selected feature tags.
[0461] In a possible implementation, a directional search control is further provided in the display interface; after receiving a selection operation on at least one feature tag displayed in the display interface, the processor 1601 is further configured to perform the following operations:
[0462] After detecting that the directional search control is triggered, displaying the input panel;
[0463] In response to a text input operation detected in the input panel, obtaining a search category of the input;
[0464] Retrieve one or more media contents based on the selected feature tags and search categories;
[0465] The media content includes any of the following: media content including some of the selected feature tags and matching the search category, and media content including all of the selected feature tags and matching the search category.
[0466] In a possible implementation, the number of retrieved media contents is P, where P is a positive integer; and the outputting methods of the P media contents include at least one of the following:
[0467] Arrange and display P media contents;
[0468] Surround display of P media contents;
[0469] Clustering displays P media contents;
[0470] Switch to display P media contents;
[0471] The arrangement and display method includes at least one of the following: arranging and displaying in random order, arranging and displaying in order of the release time of the P media contents, arranging and displaying in order of the length of the media duration of the P media contents, and arranging and displaying in order of the feature similarity between the blogger and the object in the P media contents;
[0472] The surround display method includes at least one of the following: taking the media content with the greatest feature similarity to the image as the center point, and the other media content in the P media content except the center point as the points on the circumference; taking the media content with the latest release time as the center point, and the other media content in the P media content except the center point as the points on the circumference; taking the media content with the longest media duration as the center point, and the other media content in the P media content except the center point as the points on the circumference;
[0473] The cluster display method includes at least one of the following: cluster display according to the type of the P media contents, cluster display according to the source of the P media contents, cluster display according to the media duration of the P media contents, and cluster display according to the release time of the P media contents;
[0474] The switching display method includes at least one of the following: switching and displaying P media contents according to a specified number, switching and displaying P media contents according to a specified duration, switching and displaying P media contents according to type, and switching and displaying P media contents according to source.
[0475] In a possible implementation, the retrieved media content is P videos, where P is a positive integer; the output device 1603 outputs one or more media content retrieved based on the feature tag, and is used to perform the following operations:
[0476] Displaying P videos retrieved based on feature tags in the same display interface;
[0477] Play P videos in a preset manner;
[0478] Among them, the preset method includes any of the following: playing P videos in sequence according to the arrangement order of each video, playing P videos in sequence according to the video length of each video, playing P videos in sequence according to the release time of each video, and playing P videos in sequence at a fixed time interval.
[0479] In a possible implementation, the retrieved media content is P videos, where P is a positive integer; any video has one or more video segments that match the feature tag;
[0480] The output device 1603 outputs one or more media contents retrieved based on the feature tag, including at least one of the following:
[0481] Output one or more video clips in P videos that match the feature labels;
[0482] Output P videos, and highlight one or more corresponding video clips in each video;
[0483] The output mode of each video segment in the P videos includes any one of the following: automatically playing each video segment, randomly playing each video segment, playing each video segment according to a fixed duration, and playing each video segment according to a selected feature tag;
[0484] The highlighting includes any of the following: highlighting a progress bar corresponding to a video clip, magnifying and displaying the screen content corresponding to the video clip, animating and displaying the screen content corresponding to the video clip, and distinguishing the color or brightness of the screen content of the video clip.
[0485] In a possible implementation, the processor 1601 is further configured to perform the following operations:
[0486] During the playback of the video clip, a feature tag matching the video clip is displayed;
[0487] In response to a trigger operation of the feature tag, outputting a media result indicated by the trigger operation;
[0488] Among them, the media results include any of the following: video images that match the feature tags; retrieved historical media content that matches the feature tags; any one or more of the upcoming live videos, conferences, forums, and lectures that match the feature tags; and the advertiser's main page that matches the feature tags.
[0489] In a possible implementation, the media content includes a makeup and hair video or a dressing video; the processor 1601 is further configured to perform the following operations:
[0490] During the playback of the makeup and hair video or the outfit video, in response to an object interaction event, a preview screen is displayed;
[0491] Among them, the preview screen is used to present the video screen after the object in the makeup and hair video or the dressing video is replaced with the object in the image; or, the preview screen is used to present the video screen after the object in the makeup and hair video or the dressing video is replaced with the virtual image of the object in the image.
[0492] In a possible implementation, the processor 1601 is further configured to perform the following operations:
[0493] Displaying a historical search interface, the historical search interface includes one or more historical search records, and a historical search record is used to record any one or more of the following: feature tags of objects identified during a historical search process, and feature tags selected for retrieval;
[0494] In response to a selection operation on a target historical search record, one or more feature tags in the target historical search record are displayed.
[0495] In a possible implementation, the processor 1601 is further configured to perform the following operations:
[0496] In response to any one of the historical search records being selected, outputting one or more historical media contents retrieved corresponding to the historical search record;
[0497] Among them, historical search records support any one or more operations of being forwarded, shared, edited, and downloaded.
[0498] Specifically, the processor 1601 is further configured to perform the following operations:
[0499] receiving an image recognition request sent by a search client, wherein the image recognition request carries an image including an object;
[0500] Performing recognition processing on the image to obtain morphological features of at least one body part of the subject;
[0501] sending the morphological feature to a search client so that at least one feature tag of the object is displayed in the search client based on the morphological feature;
[0502] Receiving a media retrieval request sent by a retrieval client, where the media retrieval request carries at least one feature tag;
[0503] One or more media contents are retrieved based on the selected feature tags, and the one or more media contents are returned to the search client, wherein the media contents include bloggers matching the selected feature tags.
[0504] In a possible implementation, the processor 1601 performs recognition processing on the image to obtain a morphological feature of at least one body part of the object, and performs the following operations:
[0505] Performing virtual modeling processing on the object in the image to obtain a virtual image;
[0506] Performing multi-dimensional feature analysis processing on the virtual image to obtain morphological features of at least one body part of the object, the feature analysis processing including: bone binding processing;
[0507] The morphological features include any one or more of model features, texture features, and bone features.
[0508] In a possible implementation, the feature tag is used to indicate a morphological feature of at least one body part of the object; the processor 1601 retrieves one or more media content based on the feature tag, and performs the following operations:
[0509] Based on the morphological features indicated by the feature tags, at least one matching model matching the virtual image is retrieved from a virtual model database, wherein the virtual model database includes: a plurality of virtual models obtained by performing virtual modeling processing on objects in a plurality of media contents to be retrieved;
[0510] The media content associated with the matching model is determined as the retrieved media content.
[0511] In a possible implementation, the processor 1601 retrieves at least one matching model that matches the virtual image from the virtual model database based on the morphological feature indicated by the feature tag, and performs the following operations:
[0512] Acquire the morphological features of each virtual model in the virtual model database;
[0513] Calculating feature similarity between the morphological feature indicated by the feature label and the morphological features of each virtual model in the virtual model database;
[0514] The virtual model corresponding to the feature similarity that meets the feature similarity threshold is determined as the matching model that matches the virtual image.
[0515] In a possible implementation, the morphological features of any virtual model include: any one or more of model features, mapping features, and skeleton features; the process of the processor 1601 calculating the feature similarity between the morphological features indicated by the feature tag and the morphological features of any virtual model in the virtual model database specifically includes:
[0516] Obtaining a first similarity between a model feature of the virtual image and a model feature of any virtual model;
[0517] Obtaining a second similarity between the texture feature of the virtual image and the texture feature of any virtual model;
[0518] Obtaining a third similarity between the skeleton features of the virtual image and the skeleton features of any virtual model;
[0519] Based on the first similarity, the second similarity, and the third similarity, the feature similarity is calculated.
[0520] Based on the same inventive concept, the principles and beneficial effects of the terminal provided in the embodiment of the present application for solving the problem are similar to the principles and beneficial effects of the media retrieval method in the method embodiment of the present application for solving the problem. Please refer to the principles and beneficial effects of the implementation of the method. For the sake of concise description, they will not be repeated here.
[0521] The embodiment of the present application also provides a computer program product or a computer program, which includes a computer instruction stored in a computer-readable storage medium. The processor of the terminal reads the computer instruction from the computer-readable storage medium, and the processor executes the computer instruction, so that the terminal executes the above-mentioned media retrieval method.
[0522] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present invention is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable devices. The computer instructions can be stored in a computer-readable storage medium or transmitted through a computer-readable storage medium. The computer instructions can be transmitted from a website site, a computer, a server or a data center to another website site, a computer, a server or a data center by wired (e.g., coaxial cable, optical fiber, digital line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer-readable storage medium can be any available medium that a computer can access or a data processing device such as a server, a data center, etc. that includes one or more available media integrated. The available medium can be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive (SSD)), etc.
[0523] The above is only a specific implementation of the present application, but the protection scope of the present application is not limited thereto. Any technical object familiar with the technical field can be easily changed or replaced within the technical scope disclosed by the present invention, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application should be based on the protection scope of the claims.
Claims
1. A media retrieval method, It is characterized in that include: Get the image containing the object; displaying at least one feature tag of the object, wherein the feature tag is used to indicate a morphological feature of at least one body part of the object; According to the selected feature tag, the retrieved one or more media contents are outputted, wherein the media contents include bloggers matching the selected feature tag.
2. The method according to claim 1, It is characterized in that The image includes at least one of the following: An image obtained by calling a camera component to take a picture of the object; receiving an image uploaded by the subject; Calling a camera component to shoot a video of the object to obtain a video, and extracting an image from the video; The image is acquired when an image shooting event is generated in the search page; the image shooting event includes any one of the following: an event generated when a shooting search entry in the search page is triggered; an event generated when an image acquisition operation occurs in the operation area of the search page; the image acquisition operation includes any one of the following: a single-click operation, a double-click operation, a long-press operation, a gesture operation, and a floating gesture; The method of extracting the image from the video includes any one of the following: randomly extracting an image frame from the video as the image; extracting a specified image frame from the video as the image; extracting an image frame that meets a preset image condition from the video as the image; The preset image conditions include any one of the following: the image frame includes a specified body part, the shooting posture is a fixed posture, the shooting angle is a fixed angle, and the image clarity reaches a clarity threshold.
3. The method according to claim 1, It is characterized in that The morphological features of any body part of the object are indicated by one or more feature tags; The displaying of at least one feature tag of the object includes at least one of the following: Respectively displaying partial or complete feature labels of each body part of the object at the display position of each body part of the object in the image; Displaying partial or complete feature labels of each body part of the object at the display position of each body part of the virtual image of the object, respectively, the body parts of the virtual image corresponding to the body parts of the object one by one; Displaying at least one characteristic tag of the object in a tag display area; Displaying a portion of the feature labels of the object in the image, and displaying the remaining portion of the feature labels in the label display area; Synchronously displaying at least one characteristic tag of the object in the image and the tag display area; Arrange and display at least one characteristic label of the object in the label display area; Displaying at least one characteristic label of the object in the label display area according to body part clustering; The label display area includes at least one of the following: an interface independent of the display interface of the image; a window independent of the display interface of the image; a fixed area in the display interface of the image; A floating layer suspended on the display interface of the image.
4. The method according to claim 1, It is characterized in that The method further comprises: In response to a zooming operation on a body part of the object, the displayed number of feature labels corresponding to each body part of the object is dynamically adjusted following the zooming operation.
5. The method according to claim 1, It is characterized in that The method further comprises: In response to a circle operation on a body part of the subject, highlighting the circled body part; Among them, the highlighted display includes any one or more of the following: enlarged display, flashing display, animated display, displaying the circled body parts and uncircled body parts of the object by distinguishing by brightness, and displaying the circled body parts and uncircled body parts of the object by distinguishing by color.
6. The method according to claim 5, It is characterized in that The method further comprises: Following the highlighting of the circled body part, dynamically adjusting the display mode of the feature label of the circled body part; The dynamic adjustment includes any one or more of the following: dynamically adjusting the display quantity of the feature labels of the circled body parts, dynamically adjusting the display status of the feature labels of the circled body parts, and dynamically adjusting the display form of the feature labels of the circled body parts.
7. The method according to claim 1, It is characterized in that Any feature tag supports being edited, and the editing includes at least one of the following operations: modify, delete, add, and select; If the body part of the object includes a face, the morphological features of the face indicated by the feature label of the face include any one or more of the following: facial skin texture, skin color, organ features, facial morphology, and pore status; If the body part of the object includes a head, the morphological features of the head indicated by the feature label of the head include any one or more of the following: head outline, cranial features, hair color type, and hair quality features; If the body parts of the object include limbs, the morphological features of the limbs indicated by the feature tags of the limbs include any one or more of the following: limb position, limb outline, limb height, and body shape features.
8. The method according to claim 1, It is characterized in that The feature tag is displayed in a display interface of the image, and an inspiration search control is provided in the display interface; the method further includes: Receiving a selection operation of at least one feature tag displayed in the display interface; In response to a triggering operation on the inspiration retrieval control, one or more media contents are retrieved based on the selected feature tags; The media content includes any one of the following: media content including some selected feature tags, and media content including all selected feature tags.
9. The method according to claim 8, It is characterized in that The display interface is also provided with a directional search control; after receiving a selection operation of at least one feature tag displayed in the display interface, the method further includes: After detecting that the directional search control is triggered, displaying an input panel; In response to a text input operation detected in the input panel, obtaining a search category of the input; Retrieve one or more media contents based on the selected feature tags and the search category; The media content includes any one of the following: media content including some selected feature tags and matching the search category, and media content including all selected feature tags and matching the search category.
10. The method according to claim 1, It is characterized in that The number of the retrieved media contents is P, where P is a positive integer; the output mode of the P media contents includes at least one of the following: Arrange and display P media contents; Surround display of the P media contents; Clustering and displaying the P media contents; Switching to display the P media contents; The arrangement and display method includes at least one of the following: arranging and displaying in random order, arranging and displaying in order of the release time of the P media contents, arranging and displaying in order of the length of the media duration of the P media contents, and arranging and displaying in order of the feature similarity between the blogger and the object in the P media contents; The surround display method includes at least one of the following: taking the media content with the greatest feature similarity to the image as the center point, and the other media content of the P media content except the center point as points on the circumference; taking the media content with the latest release time as the center point, and the other media content of the P media content except the center point as points on the circumference; taking the media content with the longest media duration as the center point, and the other media content of the P media content except the center point as points on the circumference; The cluster display method includes at least one of the following: cluster display according to the type of the P media contents, cluster display according to the source of the P media contents, cluster display according to the media duration of the P media contents, and cluster display according to the release time of the P media contents; The switching display method includes at least one of the following: switching display of the P media contents according to a specified number, switching display of the P media contents according to a specified duration, switching display of the P media contents according to type, and switching display of the P media contents according to source.
11. The method according to claim 1, It is characterized in that The retrieved media content is P videos, where P is a positive integer; the output is one or more media contents retrieved based on the feature tag, including: Displaying P videos retrieved based on the feature tags in the same display interface; Play the P videos in a preset manner; Among them, the preset method includes any one of the following: playing the P videos in sequence according to the arrangement order of each video, playing the P videos in sequence according to the video length of each video, playing the P videos in sequence according to the release time of each video, and playing the P videos in sequence at fixed time intervals.
12. The method of claim 1, It is characterized in that The retrieved media content is P videos, where P is a positive integer; any video has one or more video clips that match the feature tag; The output is based on the one or more media contents retrieved by the feature tag, including at least one of the following: Output one or more video clips in the P videos that match the feature tags; Output P videos, and highlight one or more corresponding video clips in each video; The output mode of each video segment in the P videos includes any one of the following: automatically playing each video segment, randomly playing each video segment, playing each video segment according to a fixed duration, and playing each video segment according to a selected feature tag; The highlighting includes any one of the following: highlighting a progress bar corresponding to the video clip, magnifying and displaying the screen content corresponding to the video clip, animating and displaying the screen content corresponding to the video clip, and distinguishing and displaying the color or brightness of the screen content of the video clip.
13. The method according to claim 12, It is characterized in that The method further comprises: During the playback of the video clip, displaying a feature tag matching the video clip; In response to a trigger operation of the feature tag, outputting a media result indicated by the trigger operation; Among them, the media results include any one of the following: video images matching the feature tag; retrieved historical media content matching the feature tag; any one or more of the upcoming live videos, conferences, forums, and lectures matching the feature tag; and the advertiser's main page matching the feature tag.
14. The method of claim 1, It is characterized in that The media content includes a makeup and hair video or a dressing video; the method further includes: During the playback of the makeup and hair video or the outfit video, in response to an object interaction event, displaying a preview screen; Among them, the preview screen is used to present the video screen after the object in the makeup and hair video or the dressing video is replaced with the object in the image; or, the preview screen is used to present the video screen after the object in the makeup and hair video or the dressing video is replaced with the virtual image of the object in the image.
15. The method of claim 1, It is characterized in that The method further comprises: Displaying a historical search interface, wherein the historical search interface includes one or more historical search records, wherein a historical search record is used to record any one or more of the following: a feature tag of the object identified during a historical search process, and a feature tag selected for retrieval; In response to a selection operation on a target history search record, one or more feature tags in the target history search record are displayed.
16. The method of claim 15, It is characterized in that The method further comprises: In response to any historical search record being selected, outputting one or more historical media contents retrieved corresponding to the historical search record; The historical search record supports any one or more operations of being forwarded, shared, edited, and downloaded.
17. A media retrieval method, It is characterized in that include: Receiving an image recognition request sent by a search client, wherein the image recognition request carries an image including an object; Performing recognition processing on the image to obtain morphological features of at least one body part of the object; Sending the morphological feature to the search client, so that at least one feature tag of the object is displayed in the search client based on the morphological feature; Receiving a media retrieval request sent by the retrieval client, wherein the media retrieval request carries at least one feature tag; One or more media contents are retrieved based on the selected feature tags, and the one or more media contents are returned to the search client, wherein the media contents include bloggers matching the selected feature tags.
18. The method of claim 17, It is characterized in that The performing recognition processing on the image to obtain the morphological features of at least one body part of the object includes: Performing virtual modeling processing on the object in the image to obtain a virtual image; Performing multi-dimensional feature analysis processing on the virtual image to obtain morphological features of at least one body part of the object, the feature analysis processing including: bone binding processing; The morphological features include any one or more of model features, texture features, and skeleton features.
19. The method of claim 18, It is characterized in that The feature tag is used to indicate a morphological feature of at least one body part of the object; and the retrieving and obtaining one or more media contents based on the feature tag includes: Based on the morphological features indicated by the feature tags, at least one matching model matching the virtual image is retrieved from a virtual model database, wherein the virtual model database includes: a plurality of virtual models obtained by performing virtual modeling processing on objects in a plurality of media contents to be retrieved; The media content associated with the matching model is determined as the retrieved media content.
20. The method of claim 19, It is characterized in that The retrieving at least one matching model matching the virtual image from a virtual model database based on the morphological feature indicated by the feature tag comprises: Acquire the morphological features of each virtual model in the virtual model database; Calculating feature similarity between the morphological feature indicated by the feature tag and the morphological features of each virtual model in the virtual model database; The virtual model corresponding to the feature similarity that meets the feature similarity threshold is determined as the matching model that matches the virtual image.
21. The method of claim 20, It is characterized in that The morphological features of any virtual model include: any one or more of model features, texture features, and skeleton features; The process of calculating the feature similarity between the morphological feature indicated by the feature tag and the morphological feature of any virtual model in the virtual model database includes: Acquire a first similarity between a model feature of the virtual image and a model feature of any virtual model; Obtaining a second similarity between the texture feature of the virtual image and the texture feature of any one of the virtual models; Obtaining a third similarity between the skeleton features of the virtual image and the skeleton features of any one of the virtual models; Based on the first similarity, the second similarity, and the third similarity, feature similarity is calculated.
22. A media retrieval device, It is characterized in that include: An acquisition unit, used for acquiring an image containing an object; A display unit, configured to display at least one feature tag of the object, wherein the feature tag is used to indicate a morphological feature of at least one body part of the object; The output unit is used to output the retrieved one or more media contents according to the selected feature tags, wherein the media contents include bloggers matching the selected feature tags.
23. A media retrieval device, It is characterized in that include: A receiving unit, configured to receive an image recognition request sent by a search client, wherein the image recognition request carries an image including an object; a processing unit, configured to perform recognition processing on the image to obtain a morphological feature of at least one body part of the object; a sending unit, configured to send the morphological feature to the search client, so that at least one feature tag of the object is displayed in the search client based on the morphological feature; The receiving unit is further configured to receive a media retrieval request sent by the retrieval client, wherein the media retrieval request carries at least one feature tag; The processing unit is further configured to retrieve one or more media contents based on the selected feature tags, and return the one or more media contents to the search client, wherein the media contents include bloggers matching the selected feature tags.
24. A computer device, It is characterized in that include: a processor adapted to execute a computer program; A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by the processor, the media retrieval method as described in any one of claims 1-16 or 17-21 is implemented.
25. A computer-readable storage medium, It is characterized in that The computer-readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor and executing the media retrieval method according to any one of claims 1-16 or 17-21.
26. A computer program product, It is characterized in that The computer program product comprises computer instructions, and when the computer instructions are executed by a processor, the media retrieval method according to any one of claims 1-16 or 17-21 is implemented.
