An image retrieval method and related apparatus

Abstract

Embodiments of the present application disclose an image retrieval method and related device, in order to make the retrieval image returned to the initiator better fit the emotional expression brought by the game event, event recognition can be performed on the to-be-retrieved information first, a target game event corresponding to the to-be-retrieved information is determined, and then a target emotional type corresponding to the target game event is determined, the target emotional type is the emotion brought to the player object by the target game event under normal circumstances. By determining the retrieval image corresponding to the to-be-retrieved information according to the target emotional type, the retrieval image can fully fit the emotional experience that can be triggered by the to-be-retrieved information, thereby expanding the image retrieval dimension and improving the richness of the retrieval image while ensuring the fitting degree of the retrieval image.

Description

Technical Field

[0001] The present application relates to the technical field, and in particular to an image retrieval method and related devices. Background Art

[0002] With the continuous development of network technology, the ways people communicate online are becoming increasingly diverse. Using images to express and communicate has become an important part of online communication. For example, people use a variety of emoticons to spice up their conversations.

[0003] To provide appropriate images to those who need to communicate information, related technologies can retrieve corresponding images based on the information provided by the subject. However, these image retrieval methods can only be based on the degree of match between the semantics of the information being retrieved and the semantics of the image itself. For example, the degree of match between the image being retrieved and images in a database, or the degree of match between the text being retrieved and the text contained in the image, can be used to retrieve the image.

[0004] However, the image retrieval method in the related art is relatively simple, and it is difficult to provide rich and accurate image retrieval results for the objects that need image retrieval, and the image retrieval effect is poor. Summary of the Invention

[0005] In order to solve the above technical problems, the present application provides an image retrieval method, which can perform image retrieval based on the emotional dimension corresponding to the things expressed by the information to be retrieved, thereby improving the matching degree and richness of the retrieved images.

[0006] The embodiments of this application disclose the following technical solutions:

[0007] In a first aspect, an embodiment of the present application discloses an image retrieval method, the method comprising:

[0008] Obtaining information to be retrieved provided by an initiator of the image retrieval, wherein the information to be retrieved is used to express a game event, wherein the game event is an event triggered by a player object in the game;

[0009] Performing event recognition on the information to be retrieved to determine a target game event corresponding to the information to be retrieved;

[0010] Determining a target emotion type corresponding to the target game event;

[0011] Determining an image corresponding to the target emotion type as a first retrieval image;

[0012] Determining, based on the first search image, a search image corresponding to the information to be retrieved;

[0013] presenting the search image to the initiator.

[0014] In a second aspect, the embodiments of the present application disclose an image search device, which comprises a first acquisition unit, a first identification unit, a first determination unit, a second determination unit, a third determination unit and a first presentation unit:

[0015] The first acquisition unit is configured to acquire search information provided by an initiator of image search, wherein the search information is used to express a game event triggered by a player object in a game.

[0016] The first identification unit is configured to perform event identification on the search information, and determine a target game event corresponding to the search information.

[0017] The first determination unit is configured to determine a target emotion type corresponding to the target game event.

[0018] The second determination unit is configured to determine an image corresponding to the target emotion type as a first search image.

[0019] The third determination unit is configured to determine, according to the first search image, a search image corresponding to the search information.

[0020] The first presentation unit is configured to present the search image to the initiator.

[0021] In a possible implementation, the search information is a game picture image, and the first acquisition unit is specifically configured to:

[0022] acquire a search video provided by the initiator;

[0023] perform video frame extraction on the search video to obtain a video frame image corresponding to the search video;

[0024] take the video frame image as the game picture image;

[0025] The first identification unit is specifically configured to:

[0026] perform information extraction on a preset image position in the game picture image to determine to-be-identified information corresponding to the game picture image, wherein the to-be-identified information comprises any one or a combination of multiple kinds of to-be-identified text and to-be-identified images;

[0027] determine a game event corresponding to the to-be-identified information and the game event information as a target game event corresponding to the search information, wherein the game event information is used to identify a game event.

[0028] The device further comprises a first adding unit, a replacing unit and a second displaying unit.

[0029] The first adding unit is configured to add the search image in the video frame image to obtain a processed video frame image corresponding to the video frame image.

[0030] The replacing unit is configured to replace the video frame image in the to-be-searched video with the processed video frame image to generate a processed video corresponding to the to-be-searched video.

[0031] The second displaying unit is configured to display the processed video to the initiator.

[0032] In a possible implementation, the second determining unit is specifically configured to:

[0033] An image including a target event text corresponding to the target game event and corresponding to the target emotion type is determined as the first search image, a text keyword corresponding to the image is extracted from text content included in the image, and the target event text is used to identify the target game event.

[0034] In a possible implementation, the device further comprises a fourth determining unit and a fifth determining unit.

[0035] The fourth determining unit is configured to determine whether the first search image meets a preset image quantity.

[0036] The fifth determining unit is configured to, in response to the first search image not meeting the preset image quantity, determine, as a second search image, an image including a target event text corresponding to the target game event in a corresponding text keyword or an image corresponding to the target emotion type, except the first search image.

[0037] The third determining unit is specifically configured to:

[0038] Determine a search image corresponding to the to-be-searched information according to the first search image and the second search image.

[0039] In a possible implementation, the device further comprises a sixth determining unit and a seventh determining unit.

[0040] The sixth determining unit is configured to determine, based on semantic similarity, a synonymous extended emotion type corresponding to the target emotion type and a synonymous extended event text corresponding to the target event text, a semantic similarity between the synonymous extended emotion type and the target emotion type being greater than a first preset threshold, and a semantic similarity between the synonymous extended event text and the target event text being greater than a second preset threshold.

[0041] The seventh determining unit is configured to determine, in addition to the first search image and the second search image, an image whose corresponding text keyword includes the synonymous expanded event text, or an image corresponding to the synonymous expanded emotion type as a third search image;

[0042] The third determining unit is specifically configured to:

[0043] A search image corresponding to the information to be retrieved is determined according to the first search image, the second search image, and the third search image.

[0044] In a possible implementation manner, the apparatus further includes an eighth determining unit:

[0045] The eighth determination unit is used to determine, as a fourth retrieval image, an image other than the first retrieval image, the second retrieval image, and the third retrieval image, an image whose text similarity between the text content included in the image and any text in the target text set is greater than a third preset threshold, the target text set including the target event text, the synonymous extended event text, the target emotion type, and the synonymous extended emotion type, and the arbitrary text is any one of the target event text, the synonymous extended event text, the target emotion type, and the synonymous extended emotion type;

[0046] The third determining unit is specifically configured to:

[0047] A search image corresponding to the information to be retrieved is determined according to the first search image, the second search image, the third search image, and the fourth search image.

[0048] In a possible implementation, the apparatus further includes a ninth determining unit:

[0049] The ninth determining unit is configured to determine, as a fifth retrieval image, an image other than the first retrieval image, the second retrieval image, the third retrieval image, and the fourth retrieval image, wherein the semantic similarity between the text content included in the image and any text in the target text set is greater than a fourth preset threshold;

[0050] The third determining unit is specifically configured to:

[0051] A search image corresponding to the information to be retrieved is determined according to the first search image, the second search image, the third search image, the fourth search image, and the fifth search image.

[0052] In a possible implementation manner, the third determining unit is specifically configured to:

[0053] determine a first image quantity corresponding to the first retrieval image and a second image quantity corresponding to the second retrieval image based on a preset retrieval image quantity, a first preset proportion corresponding to the first retrieval image, and a second preset proportion corresponding to the second retrieval image, the first preset proportion being used to identify a proportion of the first retrieval image in the retrieval image, the second preset proportion being used to identify a proportion of the second retrieval image in the retrieval image, the first preset proportion being greater than the second preset proportion, and the preset retrieval image quantity being a maximum quantity corresponding to the retrieval image;

[0054] randomly select a first image corresponding to the first image quantity from the first retrieval image, and randomly select a second image corresponding to the second image quantity from the second retrieval image;

[0055] determine the first image and the second image as retrieval images corresponding to the to-be-retrieved information.

[0056] In a possible implementation, the apparatus further includes an ordering unit:

[0057] The ordering unit is configured to order the retrieval images corresponding to the to-be-retrieved information in a sequence from large to small based on the preset proportions.

[0058] The first display unit is specifically configured to:

[0059] display, to the initiator, images in the retrieval images that are located in a top N position in the image ordering based on the image ordering of the retrieval images, N being less than the preset retrieval image quantity, and an ordering of the first retrieval image in the image ordering being located before an ordering of the second retrieval image.

[0060] In a possible implementation, the apparatus further includes a second identifying unit, an extracting unit, a third identifying unit, a tenth determining unit, and an eleventh determining unit:

[0061] The second identifying unit is configured to perform emotion identification on a to-be-analyzed image by using an image emotion identification model, and determine a first emotion type corresponding to the to-be-analyzed image.

[0062] The extracting unit is configured to extract image text included in the to-be-analyzed image.

[0063] The third identifying unit is configured to perform emotion identification on the image text by using a text emotion identification model, and determine a second emotion type corresponding to the image text.

[0064] The tenth determining unit is configured to: if the first sentiment type and the second sentiment type correspond to a same superordinate sentiment type, taking both the first sentiment type and the second sentiment type as the sentiment type corresponding to the image to be analyzed, wherein the superordinate sentiment type includes a positive sentiment, a negative sentiment and a neutral sentiment.

[0065] The eleventh determining unit is configured to: if the first sentiment type and the second sentiment type correspond to different superordinate sentiment types, determining the first sentiment type as the sentiment type corresponding to the image to be analyzed.

[0066] In a possible implementation, the apparatus further includes a twelfth determining unit:

[0067] The twelfth determining unit is configured to determine a target image retrieval precision corresponding to the initiator, wherein the target image retrieval precision is used to identify an image retrieval precision requirement corresponding to the initiator.

[0068] The first retrieval image, the second retrieval image, the third retrieval image, the fourth retrieval image and the fifth retrieval image each have a corresponding image retrieval precision, and the third determining unit is specifically configured to:

[0069] determine, as the retrieval image corresponding to the retrieval information, an image corresponding to the target image retrieval precision from the first retrieval image, the second retrieval image, the third retrieval image, the fourth retrieval image and the fifth retrieval image.

[0070] In a possible implementation, the twelfth determining unit is specifically configured to:

[0071] obtain a precision selection operation corresponding to the initiator, wherein the precision selection operation is used to determine the target image retrieval precision from a plurality of pending image retrieval precisions;

[0072] or,

[0073] obtain image application scenario information corresponding to the initiator, wherein the image application scenario information is used to identify an image application scenario corresponding to the initiator, and the image application scenario is an application scenario of a retrieval image corresponding to the retrieval information to be retrieved;

[0074] determine the target image retrieval precision corresponding to the initiator according to the image application scenario information.

[0075] In a possible implementation, the apparatus further includes a thirteenth determining unit:

[0076] The thirteenth determining unit is configured to determine a game type corresponding to the target game event.

[0077] The second determining unit is specifically configured to:

[0078] An image corresponding to the target emotion type and the game type is determined as the first retrieval image.

[0079] In a possible implementation, the first acquiring unit is specifically configured to:

[0080] Obtaining the information to be retrieved input by the initiator of the image retrieval in the chat interface;

[0081] The device further includes a second acquiring unit and a sending unit:

[0082] The second acquiring unit is configured to acquire an image selection operation performed by the initiator on a target search image in the search images;

[0083] The sending unit is used to send the target retrieval image in the chat interface.

[0084] In a possible implementation, the apparatus further includes a third acquiring unit, a generating unit, a second adding unit, and a returning unit:

[0085] The third obtaining unit is used to obtain the game account information corresponding to the initiator of the image retrieval;

[0086] The generating unit is configured to generate initial game event display information corresponding to the target game event based on the game account information and the target game event, wherein the initial game event display information is used to display the target game event triggered by the initiator in the game corresponding to the game account information;

[0087] The second adding unit is configured to add the search image to the initial game event display information to generate game event display information corresponding to the information to be retrieved;

[0088] The returning unit is configured to return the game event display information to the initiator.

[0089] In a possible implementation manner, the first determining unit is specifically configured to:

[0090] According to the mapping relationship between game events and emotion types, determining the target emotion type corresponding to the target game event;

[0091] or,

[0092] Inputting the target game event into a game event emotion recognition model to determine a target emotion type corresponding to the target game event;

[0093] The game event emotion recognition model is trained in the following way:

[0094] obtain a sample game event set, the sample game event set comprising a plurality of sample game events, the sample game events having corresponding sample sentiment types;

[0095] train the game event sentiment recognition model by taking the sample game events as training samples and the sample sentiment types corresponding to the sample game events as training labels.

[0096] In a third aspect, an embodiment of the present application discloses a computer device, the computer device comprising a processor and a memory:

[0097] The memory is configured to store program code and transmit the program code to the processor.

[0098] The processor is configured to execute the image retrieval method according to the instructions in the program code.

[0099] In a fourth aspect, an embodiment of the present application discloses a computer readable storage medium configured to store a computer program, the computer program being configured to execute the image retrieval method according to any one of the first aspect.

[0100] In a fourth aspect, an embodiment of the present application discloses a computer program product comprising instructions which, when executed on a computer, cause the computer to execute the image retrieval method according to any one of the first aspect.

[0101] As can be seen from the above technical solutions, when performing image retrieval, the present application can first obtain the to-be-retrieved information provided by the initiator of image retrieval, the to-be-retrieved information being used to express a game event, the game event being an event triggered by a player object in a game. In order to make the retrieved image returned to the initiator better fit the emotional expression brought by the game event, the to-be-retrieved information can be subjected to event recognition first to determine the target game event corresponding to the to-be-retrieved information, and then the target sentiment type corresponding to the target game event is determined, the target sentiment type being the emotion usually brought by the target game event to the player object. By determining the retrieved image corresponding to the to-be-retrieved information according to the target sentiment type, the retrieved image can be made to fully fit the emotional experience that can be triggered by the to-be-retrieved information, thereby expanding the image retrieval dimension and improving the richness of the retrieved image while ensuring the fitting degree of the retrieved image. After the retrieved image is displayed to the initiator, the initiator can flexibly use the retrieved image to enrich the emotional expression of the content related to the target game event on the target game event, so that the acceptor who browses these contents can more vividly experience the emotion when the target game event is triggered. BRIEF DESCRIPTION OF DRAWINGS

[0102] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the accompanying drawings needed to be used in the embodiments or prior art description will be briefly introduced as follows. Obviously, the accompanying drawings in the following description only only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor on the basis of these drawings.

[0103] Figure 1 A schematic diagram of an image retrieval method in a practical application scenario provided by an embodiment of the present application;

[0104] Figure 2 A flowchart of an image retrieval method provided by an embodiment of the present application;

[0105] Figure 3 A schematic diagram of an image retrieval method provided by an embodiment of the present application;

[0106] Figure 4 A schematic diagram of an image retrieval method provided by an embodiment of the present application;

[0107] Figure 5 A schematic diagram of an image retrieval method provided by an embodiment of the present application;

[0108] Figure 6 A schematic diagram of an image retrieval method provided by an embodiment of the present application;

[0109] Figure 7 A schematic diagram of an image retrieval method provided by an embodiment of the present application;

[0110] Figure 8 A schematic diagram of an image retrieval method provided by an embodiment of the present application;

[0111] Figure 9 A structural block diagram of an image retrieval device provided by an embodiment of the present application;

[0112] Figure 10 A structural diagram of a terminal provided by an embodiment of the present application;

[0113] Figure 11 A structural diagram of a server provided by an embodiment of the present application. DETAILED DESCRIPTION

[0114] The embodiments of the present application will be described below with reference to the accompanying drawings.

[0115] Enriching information through images is a common method of expressing information. For example, when creating content related to gaming events, creators often add emojis and other images that match the gaming events. Related technologies facilitate content creation by providing image retrieval capabilities, allowing creators to retrieve images corresponding to the gaming events depicted in the content.

[0116] However, in related art, image retrieval for game events can only be based on the semantics of the game event itself. The retrieved images are all images that are semantically close to the game event. For example, for the game event "killing other game objects," the images retrieved in related art are all images with a high degree of image semantic similarity to the scene of "killing other game objects," or images with a high degree of textual semantic similarity to the text of "killing other game objects." This image retrieval method can only return images with a high degree of semantic similarity to the game event. The image retrieval dimension is relatively single, lacking the emotional expression brought about by the game event, and it is difficult to meet the information expression needs of the creator.

[0117] It's understandable that game events often trigger the emotions of the player object that triggered the game event. For example, when a player object kills a game object controlled by another player object, they may feel happy or excited. When a game object controlled by another player object kills a game object controlled by another player object, they may feel sad or angry. Game events typically convey a relatively uniform emotion to the player object. Therefore, using images that match the corresponding emotions can enrich the representation of game events to a certain extent.

[0118] Based on this, in order to solve the above technical problems, the present application provides an image retrieval method, in which the processing device can first identify the game event expressed by the retrieved information, and then determine the emotion type corresponding to the game event, and return the retrieved image to the initiator of the image retrieval based on the emotion type, thereby enriching the image retrieval dimension and enabling the returned retrieved image to emotionally enhance the expression of the game event.

[0119] It is understood that this method can be applied to a processing device capable of performing image retrieval, such as a terminal device or server with image retrieval capabilities. This method can be executed independently by a terminal device or server, or it can be applied to a network scenario where a terminal device and a server communicate, with the terminal device and server working together to execute the method. The terminal device can be a computer, mobile phone, or other device. The server can be understood to be an application server or a web server. In actual deployment, the server can be a standalone server or a clustered server.

[0120] In order to facilitate understanding of the technical solutions provided by the embodiments of the present application, next, an image retrieval method provided by the embodiments of the present application will be introduced in combination with an actual application scenario.

[0121] Referring to Figure 1 , Figure 1 A flowchart of an image retrieval method provided by the embodiments of the present application, the method comprises:

[0122] The initiator of image retrieval can input the to-be-retrieved information through the terminal device 101, wherein the to-be-retrieved information can be a to-be-retrieved image, a to-be-retrieved text and a to-be-retrieved video, etc., and these to-be-retrieved information are all used to express game events. In this actual application scenario, the to-be-retrieved information is a game picture image, and the image retrieval server 101 can perform event identification on the game picture image to determine the target game event corresponding to the game picture image, and then determine the retrieval image corresponding to the to-be-retrieved information according to the target game event. Figure 1 As can be seen, in the game picture image, the game character of player A has killed the game character of player B, and the game picture image has the game broadcast information of “you have killed player B”, so the target game event corresponding to the game picture image is killing, and the server 101 can determine that the target emotion type corresponding to the target game event is “happy”, and then determine the retrieval image corresponding to the to-be-retrieved information according to the target emotion type. Figure 1 As can be seen, the three retrieval images determined by the server 101 are retrieval images with a smile or with the text “hahahahah” expressing happiness. The server 101 can return the retrieval image to the terminal device 101 for display to the initiator, so that the initiator knows the image retrieval result.

[0123] Since the server 101 performs image retrieval based on the emotion type corresponding to the target game event, the retrieved retrieval image is more consistent with the emotional expression corresponding to the target game event, so that the initiator can use the retrieval image to more fully express the target game event in terms of emotion, and thus can significantly drive the emotion of the object who knows the target game event.

[0124] Next, an image retrieval method provided by the embodiments of the present application will be introduced in combination with the accompanying drawings.

[0125] Referring to Figure 2 , Figure 2 A flowchart of an image retrieval method provided by the embodiments of the present application, the method comprises:

[0126] S201: Obtain the to-be-retrieved information provided by the initiator of image retrieval.

[0127] The initiator refers to an object initiating image retrieval, for example, a game event video producer, a game event article author, etc. The to-be-retrieved information is used to express a game event, which is an event triggered by a player object in a game, for example, can include "killing other game objects", "being killed by other game objects", "continuous killing of game objects", etc. The game object is an object that can appear in the game, for example, can be a game object controlled by another player object or a non-controllable game object, etc.

[0128] The information type of the to-be-retrieved information can include multiple types, for example, can include image information, video information, text information, etc. The image information can be a game screen image, the video information can be a game screen video, and the text information can be a text directly describing a game event, etc. The processing device can provide an input interface for the initiator to input the to-be-retrieved information, so that the initiator can input the to-be-retrieved information for the processing device to acquire.

[0129] S202: Event recognition is performed on the to-be-retrieved information to determine a target game event corresponding to the to-be-retrieved information.

[0130] It can be understood that in the game, the emotion triggered by the game event is relatively fixed, for example, when a player object kills other game objects, the emotion is usually happy and excited, and when the game object controlled by the player object is killed by other player objects, the emotion is usually sad and angry. Therefore, the processing device can first identify the game event corresponding to the to-be-retrieved information, and then match the corresponding emotion type based on the game event.

[0131] The processing device can first identify the game event expressed by the to-be-retrieved information to determine a target game event corresponding to the to-be-retrieved information. The target game event can be any game event triggered by a player object in the game.

[0132] It can be understood that the event recognition of the to-be-retrieved information is actually one of the technical points of the present application different from related technologies in image retrieval. In related technologies, since the emotion type corresponding to the to-be-retrieved information does not need to be analyzed, the game event associated with the to-be-retrieved information is not actually concerned, but the retrieval image is directly determined based on the similarity of the image content or the similarity of the image text between the to-be-retrieved information and the image. In the embodiments of the present application, image retrieval is performed based on the emotion type, and the emotion type is the emotion type matched with the game event expressed by the to-be-retrieved information. Therefore, the present application needs to first identify the game event expressed by the to-be-retrieved information.

[0133] For example, assuming that the inputted information to be retrieved by the initiator of image retrieval is the text "five kills", in the related art, when image retrieval is performed, whether "five kills" is a game event is not analyzed, but images containing the text "five kills" in the text content of the images are directly returned as the retrieved images, that is, only the matching degree on the text or the semantic similarity is concerned. However, the emotional type brought to the player object by the game event "five kills" (referring to killing five game objects in succession) is usually "happy", "joyful" and the like, and the semantic similarity and the text similarity between "five kills" and the emotional types "happy" and "joyful" are not high, so the image retrieval technology in the related art does not return images related to the emotional type "happy", resulting in low richness of image retrieval, and the initiator of image retrieval cannot fully express the emotion brought by the game event through the retrieved images.

[0134] S203: Determine a target emotional type corresponding to the target game event.

[0135] As mentioned above, the emotion brought by the game event in the game is relatively fixed, that is, the emotions generated by most player objects when triggering the same game event are relatively similar, so the processing device can take the target emotional type corresponding to the target game event as the emotional type on which this image retrieval is based, so as to obtain retrieved images that can fit the emotional resonance of the object creating or browsing the content related to the target game event.

[0136] S204: Determine the images corresponding to the target emotional type as the first retrieved images.

[0137] The processing device can obtain the retrieved images matching the target emotional type as the retrieved images corresponding to the to-be-retrieved information according to the target emotional type from various image acquisition channels such as an image database or a network. Here, it is not limited to taking only the images corresponding to the target emotional type as the retrieved images corresponding to the to-be-retrieved information, but image retrieval can also be performed in combination with other retrieval methods, which will be introduced below. This step ensures that the retrieved images corresponding to the to-be-retrieved information determined have retrieved images that can reflect the target emotional type, for example, the images reflecting the emotional types "happy" and "excited" are included in the retrieved images corresponding to the to-be-retrieved information "five kills".

[0138] In a possible implementation, when the retrieved images are determined according to the target emotional type, the processing device can determine the images corresponding to the target emotional type as the first retrieved images. That is, the images have a corresponding emotional type, which can be determined based on the image content and the like of the images.

[0139] S205: Determine the retrieved images corresponding to the to-be-retrieved information according to the first retrieved images.

[0140] The processing device can determine the search image corresponding to the to-be-searched information according to the first search image. For example, the first search image can be directly used as the search image corresponding to the to-be-searched information, or part of the first search image can be extracted as the search image corresponding to the to-be-searched information.

[0141] S205: displaying the search image to the initiator.

[0142] Since the search images can match the target game event in the emotional type, the processing device can display the search image searched by the above steps to the initiator, so that the initiator can use the search image to enrich the emotional expression of the target game event, so as to better express the emotion of the initiator when creating the content related to the target game object, and can more effectively arouse the emotion of the viewer who browses the content.

[0143] The processing device can display the search image to the initiator in various ways. For example, the search image can be in the form of a thumbnail file, and the initiator needs to open the thumbnail file to see the image content. Alternatively, the search image can be directly displayed in the form of image content, which is not limited here.

[0144] As can be seen from the above technical solution, in the image search, the processing device can first acquire the to-be-searched information provided by the initiator of the image search. The to-be-searched information is used to express a game event, and the game event is an event triggered by a player object in a game. In order to make the search image returned to the initiator better match the emotional expression brought by the game event, the to-be-searched information is first subjected to event identification to determine a target game event corresponding to the to-be-searched information, and then a target emotional type corresponding to the target game event is determined. The target emotional type is the emotion usually brought by the target game event to the player object. By determining the search image corresponding to the to-be-searched information according to the target emotional type, the search image can fully match the emotional experience that can be triggered by the to-be-searched information, thereby expanding the image search dimension and improving the richness of the search image while ensuring the matching degree of the search image. After the search image is displayed to the initiator, the initiator can flexibly use the search image to enrich the emotional expression of the content related to the target game event on the target game event, so that the receiver who browses the content can more vividly experience the emotion when the target game event is triggered.

[0145] In addition to the image retrieval based on the emotion type, it can be understood that different retrieval images corresponding to the same emotion type can have different biases for the game type. For example, multiple images expressing the same emotion type of "happy", some images can have shooting elements, and the image is more biased towards shooting games; some images can have various character action elements, and the image is more biased towards action games. Based on this, in a possible implementation, in order to make the retrieved images more suitable for the target game event, thereby highlighting the emotional expression for the target game event, the processing device can determine the game type corresponding to the target game event, which is the type corresponding to the game that can trigger the target game event.

[0146] Then, when performing image retrieval, the processing device can analyze the game type corresponding to each image based on the elements contained in the image, and then determine the image corresponding to the target emotion type and the game type as the first retrieval image, so that the first retrieval image can have a more accurate matching degree with the information to be retrieved in the game dimension and the emotion dimension. For example, when the target game event is "several target heads", the target game event usually corresponds to a shooting type game, and the emotion type is "happy" or other positive emotions, and the processing device can determine the image with shooting elements and expressing happy as the first retrieval image.

[0147] Among them, the event recognition manner for the game event can include multiple manners, and the processing device can have different manners for event recognition for different types of information to be retrieved. Next, different event recognition manners will be introduced.

[0148] In a possible implementation, the information to be retrieved is a game screen image. It can be understood that when a game event occurs, specific text information or image information will usually be displayed at a specific position in the game screen. For example, as shown in Figure 3 , in the game screen displayed Figure 3 , after the player object A controls the game character to kill the game character B controlled by the player object B, the game broadcast information "You killed player B" will be displayed in the upper center of the game screen.

[0149] Based on this, the processing device can determine the game event corresponding to the game screen image by recognizing the information at a specific position of the game screen image. The processing device can extract information from a preset image position in the game screen image to determine the to-be-recognized information corresponding to the game screen image, the to-be-recognized information including any one or a combination of multiple of to-be-recognized text and to-be-recognized images, and the preset image position being a position in the game that can appear information for identifying a game event. For example, in Figure 3In the preset position can be a position above the center of the game picture image, the to-be-identified information can be to-be-identified text "you killed player B", and in Figure 4 As shown in the game picture, when the game character controlled by the player object A kills the game character of the player object B, a knife-shaped kill icon is used in the game broadcast information to express the game event "you killed player B", and the to-be-identified information can be the kill icon.

[0150] The processing device can determine the game event information corresponding to each game event based on the analysis of the game content. The game event information identifies the game event, for example, can include specific text or specific images corresponding to the game event, etc. For example, in Figure 3 Figure 4 In the "kill" game event corresponds to the specific text "kill", and the specific image is the kill icon. The processing device can determine whether the game event information corresponding to each game event has game event information matching the to-be-identified information. If it matches, it can be determined that the game event has occurred. Based on this, the processing device can determine the game event corresponding to the game event information matching the to-be-identified information as the target game event corresponding to the to-be-retrieved information.

[0151] The processing device can obtain the game picture image provided by the initiator in multiple ways. For example, the initiator can directly provide a single or multiple game picture images to the processing device, or provide a game picture video, and the processing device extracts video frames in the game picture video as game picture images for event identification.

[0152] In a possible implementation, since the purpose of the initiator performing image retrieval on the game event is usually to more vividly and fully express the game event by retrieving images, and the way of expressing the game event by images is usually to add images to the game event related content, in order to bring a more efficient image retrieval experience to the initiator, the processing device can directly add the retrieval image corresponding to the game picture image to the game picture image after determining the retrieval image, and provide the initiator with game event related content after image processing.

[0153] ​For example, when the information provided by the initiator is a video to be retrieved, the processing device can first obtain the video to be retrieved provided by the initiator, which is a video including game screen images. The processing device can perform video frame extraction on the video to be retrieved to obtain a video frame image corresponding to the video to be retrieved. Then, the processing device can take the video frame image as a game screen image and perform the above-mentioned step of extracting information from a preset image position in the game screen image. After determining the search image corresponding to the game screen image in this way, the processing device can add the search image to the video frame image to obtain a processed video frame image corresponding to the video frame image, and then replace the video frame image in the video to be retrieved with the processed video frame image to generate a processed video corresponding to the video to be retrieved. That is, the processed video is a video obtained by adding a search image to part or all of the video frames in the original video to be retrieved. Since the search image can better express the emotion corresponding to the game event, the processed video can better match the game event contained in the video to be retrieved in terms of emotional expression, so that the creator of the video can more vividly express the emotional experience when triggering the game event, and the viewer of the video can more fully experience the emotion when triggering the game event.

[0154] When the processing device displays the search image to the initiator, in addition to displaying the search image, the processing device can also display the processed video to the initiator, thereby providing an automatic content generation method for the initiator, which can vividly express the game event in the emotional dimension, enriching the image search function and the image search experience of the initiator. Of course, in addition to displaying the video to the initiator, the processing device can also directly display the game screen image with the added search image to the initiator. Referring to Figure 5 , Figure 5 The effect of adding a search image to a game screen image corresponding to a game event of "killing" and an emotional type of "happy" is shown. That is, the player object is usually happy when successfully killing other game characters in the game. At this time, the search image retrieved by the processing device is a search image corresponding to "happy", and by adding the search image to the game screen image, the emotion of the player object A when killing other game characters can be more vividly expressed.

[0155] In one possible implementation, in addition to expressing game events through game screen images, the initiator of image retrieval can also express game events directly through text. For example, when performing image retrieval, the text to be retrieved can be input to describe the game event. It can be understood that, similar to the above-mentioned recognition of game screen images, when text is recognized from the game screen image, the corresponding game event can be determined by judging whether the text in the game event information matches the text in the game screen image. Similarly, when the information to be retrieved is text to be retrieved, the processing device can judge the game event corresponding to the text to be retrieved by the event text corresponding to each game event, and the event text is used to identify the corresponding game event. For example, the event text corresponding to the game event "killing other game objects" can be "kill", "single kill (referring to independently completing a kill of other game objects)", etc.

[0156] In this implementation, the information to be retrieved is the text to be retrieved, and the processing device can determine whether the text to be retrieved contains event text corresponding to a pending game event. The pending game event can be any game event, and the event text corresponding to the pending game event is used to identify the pending game event.

[0157] In response to the fact that the text to be retrieved contains event text corresponding to the pending game event, it means that the text to be retrieved has a high probability of being the text expressing the pending game event. At this time, the processing device can determine the pending game event as the target game event corresponding to the information to be retrieved.

[0158] As mentioned above, in the game, the emotional experience brought about by triggering the same game event is relatively uniform. Based on this, in one possible implementation method, the processing device can pre-determine the mapping relationship between game events and emotion types, such as shown in the following table, where "killing" refers to controlling a game object to kill other game objects in the game, "being killed" refers to the controlled game object being killed by other game objects, and "strange death" refers to the death of the controlled game object due to triggering a low-probability game event in the game, etc.

[0159] Game Events Emotional Type Kill Happy Killed Sadness, anger mysterious death Surprise, fear … …

[0160] After identifying the target game event corresponding to the information to be retrieved through event recognition, the processing device can determine the target emotion type corresponding to the target game event based on the mapping relationship between game events and emotion types. This can effectively determine the emotion type that matches the target game event, improving image retrieval efficiency. This emotion type determination method is particularly suitable for gaming scenarios. For other scenarios, the processing device can also perform emotion type analysis through other methods such as semantic analysis, which is not limited here.

[0161] In addition to determining the corresponding emotion type based on the mapping relationship, the processing device can also perform emotion analysis in various ways. For example, in one possible implementation, the processing device can train a game event emotion recognition model, which is used to determine the emotion type corresponding to the game event.

[0162] The game event emotion recognition model can be trained in the following way:

[0163] A sample game event set is obtained, where the sample game event set includes multiple sample game events. The sample game events have corresponding sample emotion types, and the sample emotion types are accurate emotion types corresponding to the sample game events.

[0164] The processing device can use the sample game event as a training sample and the sample emotion type corresponding to the sample game event as a training label to train a game event emotion recognition model. Specifically, the processing device can input the sample game event into the initial model, obtain the undetermined emotion type output by the initial model, and adjust the model parameters based on the difference between the undetermined emotion type and the sample emotion type, so that the initial model learns how to analyze a more accurate emotion type based on the game event.

[0165] The processing device may input the target game event into a game event emotion recognition model to determine a target emotion type corresponding to the target game event.

[0166] In order to perform image retrieval based on emotion type, the processing device may first determine the emotion type corresponding to each image. It is understandable that the image content and text content in the image can reflect the emotion expressed by the image to a certain extent. For example, Figure 5 In the emotional image shown, the image content is a laughing face, and the text content is "Hahahaha." Both can reflect the emotion of "happiness" to a certain extent. Based on this, in one possible implementation, when analyzing the emotion type corresponding to an image in order to accurately retrieve the corresponding image based on the emotion type, the processing device can, on the one hand, perform emotion recognition on the image to be analyzed using an image emotion recognition model to determine the first emotion type corresponding to the image to be analyzed. The image emotion recognition model is used to determine the emotion type corresponding to the image based on the image content. The image to be analyzed can be any image that requires emotion type analysis.

[0167] On the other hand, the processing device can extract image text included in the image to be analyzed, and then perform sentiment recognition on the image text through a text sentiment recognition model, to determine a second sentiment type corresponding to the image text, the text sentiment recognition model being used for sentiment type determination on text. The processing device can comprehensively analyze the image content and the text content of the image according to the first sentiment type and the second sentiment type, to determine a sentiment type corresponding to the image to be analyzed, thereby improving the accuracy of image sentiment type analysis.

[0168] It can be understood that when expressing sentiment through an image, the sentiment is mainly embodied through image content in the image. Therefore, in a possible implementation, when the first sentiment type and the second sentiment type are comprehensively analyzed to determine the sentiment type corresponding to the image to be analyzed, the processing device can take the first sentiment type corresponding to the image content as a higher-priority sentiment type determination basis.

[0169] The processing device can determine whether the first sentiment type and the second sentiment type correspond to the same upper-level sentiment type, the upper-level sentiment type including positive sentiment, negative sentiment, and no sentiment, and the sentiment type being a lower-level branch of the upper-level sentiment type. The positive sentiment is an emotion that can bring positive feelings, for example, can include happiness, joy, excitement, etc. The negative sentiment is an emotion that can bring negative feelings, for example, can include anger, sadness, fear, etc. The no sentiment is other emotions other than the positive sentiment and the negative sentiment, and has no explicit sentiment tendency.

[0170] If the first sentiment type and the second sentiment type correspond to the same upper-level sentiment type, it indicates that the first sentiment type and the second sentiment type are relatively similar, i.e., the image content and the text content of the image to be analyzed are relatively close in sentiment expression. At this time, in order to expand the comprehensiveness of image sentiment type analysis, the processing device can take both the first sentiment type and the second sentiment type as the sentiment type corresponding to the image to be analyzed, i.e., when searching based on a target sentiment type, as long as the image to be analyzed has the target sentiment type in multiple sentiment types corresponding to the image to be analyzed, the image to be analyzed can be determined as the first search image.

[0171] If the first sentiment type and the second sentiment type correspond to different upper-level sentiment types, it indicates that the first sentiment type and the second sentiment type are quite different. In order to ensure the accuracy of sentiment type analysis, the processing device can determine the first sentiment type as the sentiment type corresponding to the image to be analyzed, and discard the second sentiment type, so that the sentiment type of the image can be determined based on the main dimension of the sentiment type of the image content, avoiding the case that the sentiment expression of the image and the required sentiment type are quite different.

[0172] As mentioned above, this application does not limit the use of images corresponding to the target emotion type as retrieval images for the information to be retrieved. Image retrieval can also be combined with other retrieval methods. In order to expand the richness of retrieval images and provide the initiator with more comprehensive and high-quality image retrieval results, the processing device can also combine multiple image retrieval methods with different matching degrees and dimensions to determine the retrieval image corresponding to the information to be retrieved. This will be described in detail below.

[0173] In one possible implementation, in order to obtain a retrieval image that is more closely matched with the information to be retrieved, the processing device may not only require that the emotion type corresponding to the image matches the emotion type corresponding to the game event, but may also require that the text content expressed in the image matches the game event.

[0174] In addition to determining whether the emotion type corresponding to the image matches the target emotion type, the processing device can also determine whether the text keywords corresponding to the image include the target event text corresponding to the target game event, wherein the text keywords corresponding to the image are extracted from the text content included in the image and are used to identify the key information expressed by the text content in the image, and the target event text is used to identify the target game event.

[0175] If the keywords corresponding to the image include the target event information, it means that the content expressed by the image is most likely related to the target game event. Based on this, the processing device can include the target event text corresponding to the target game event in the corresponding text keywords, and determine the image corresponding to the target emotional type as the first retrieval image, so that the first retrieval image can match the information to be retrieved in both the game event dimension and the emotional type dimension, and obtain a retrieval image with a higher matching degree.

[0176] For example, Figure 6 As shown, the information to be retrieved is a game screen image, the corresponding game event is "kill", the event text corresponding to the game event is "kill", and the corresponding emotion type is "happy". The keywords of the determined retrieval image include "kill". On the one hand, the retrieval image can express the "happy" emotion through the smiling face in the image content. On the other hand, the text content of the image includes the event text "kill", which matches the game event. Therefore, it can take into account the full expression of the game event and the embodiment of the emotions related to the game event.

[0177] It is understandable that when performing image retrieval, the higher the image retrieval precision, the fewer the number of retrieved images. Therefore, in one possible implementation, to increase the number and diversity of retrieved images, the processing device may combine image retrieval methods with multiple precisions to jointly determine the retrieval images.

[0178] For example, in a possible implementation, the processing device can determine, as the second search image, an image that only needs to match one of the text content keyword and the emotion type, and that has a higher matching degree with the to-be-searched information. For example, in Figure 1 The search image searched in the middle is a search image that matches the to-be-searched information in the emotion type, but has a smaller relevance to the game event in the content.

[0179] The processing device can determine the search image corresponding to the to-be-searched information according to the first search image and the second search image, so as to enrich the search image content by using an image that has a higher matching degree in any dimension of the content and the emotion, and enable the initiator to have more search images to choose from.

[0180] It can be understood that some different texts have similar semantic expressions. In the present application, the emotions expressed by different emotion types can be similar, and the game events expressed by different event texts can also be similar. For example, the emotions expressed by the two emotion types of "happy" and "joy" are similar, and the emotions expressed by the two emotion types of "angry" and "angry" are also similar. In terms of game events, "killed" and "mysterious death" are similar, and both are the death of a game object controlled by a player object, only the reason for the death is different.

[0181] Based on this, in a possible implementation, to further expand the recall degree of image search, the processing device can determine, based on semantic similarity, a synonymous extended emotion type corresponding to the target emotion type and a synonymous extended event text corresponding to the target event text, the semantic similarity between the synonymous extended emotion type and the target emotion type being greater than a first preset threshold, and the semantic similarity between the synonymous extended event text and the target event text being greater than a second preset threshold. Thus, by using the synonymous extended emotion type and the synonymous extended event text, the processing device can also search for a search image that is more matched with the to-be-searched information, and improve the richness of the search image without excessively reducing the search accuracy.

[0182] The processing device can determine, in addition to the above-mentioned first retrieval image and second retrieval image, an image whose corresponding text keywords include the synonymous extended event text, or an image corresponding to the synonymous extended emotional type as a third retrieval image. The processing device can determine the retrieval image corresponding to the information to be retrieved based on the first retrieval image, the second retrieval image and the third retrieval image. In this way, the processing device can provide the initiator with some retrieval images that are not directly targeted at the target game event, but are relatively similar to the target game event, and are relatively similar in the emotions expressed by the target game event, thereby enriching the image retrieval results. Since the extended information is obtained based on semantic similarity, this retrieval method not only retains the efficiency of the text matching retrieval method, but also integrates the matching range brought by semantic matching to a certain extent.

[0183] As mentioned above, the text keywords corresponding to an image reflect the key content within the textual content of the image. Matching solely based on keywords improves image retrieval accuracy to a certain extent and enhances the textual content match between the retrieved image and the target game event. However, textual content other than keywords may also contain content related to the game event. This content may not be key content within the image's textual content and therefore not be extracted as keywords. In other words, the match between this textual content and the game event is low. Furthermore, as mentioned above, textual content can also reflect the emotional information associated with the image. When analyzing the emotional type associated with textual content, the analysis is primarily based on semantics, which may result in multiple textual expressions being categorized as belonging to the same emotional type. For example, the textual content "happy" and "happy" may be categorized as belonging to the emotional type "happy." Therefore, the textual content may not contain text directly related to the corresponding emotional type, but may contain other text with semantically similar emotional types.

[0184] Based on this, in order to further expand the retrieval strength and recall more abundant retrieval images, the processing device can directly perform text matching in the text content corresponding to the image through the target event text, synonymous expanded event text, target emotion type and synonymous expanded emotion type to retrieve images that are more closely matched with the target game event in terms of game events or emotional expressions.

[0185] The processing device can determine, as the fourth retrieval image, an image in addition to the first retrieval image, the second retrieval image and the third retrieval image, an image in which the text similarity between the text content included in the image and any text in the target text set is greater than a third preset threshold, and the target text set includes target event text, synonymous expanded event text, target emotion type and synonymous expanded emotion type, and the arbitrary text is any one of the target event text, synonymous expanded event text, target emotion type and synonymous expanded emotion type.

[0186] The processing device can determine the search image corresponding to the to-be-searched information according to the first search image, the second search image, the third search image, and the fourth search image. It should be emphasized that the above matching processes are all based on the matching degree of the text, and do not need to analyze the semantics of the text, so the matching process is relatively simple, and the initiator can be provided with search images efficiently.

[0187] In addition to matching directly from the text, in order to further improve the richness and quantity of image search, the processing device can also search images based on semantic dimensions. As mentioned above, the text content of the image can reflect the game event corresponding to the image, and can also reflect the emotion expressed by the image. Therefore, in a possible implementation, the processing device can determine, as a fifth search image, an image in which the semantic similarity between the text content included in the image and any text in the target text set is greater than a fourth preset threshold, in addition to the first search image, the second search image, the third search image, and the fourth search image. That is, the fifth search image is relatively close to the target game event or the emotion type in the game event dimension or the emotion type dimension, and thus can be determined to have a certain matching degree with the to-be-searched information. Therefore, the processing device can determine the search image corresponding to the to-be-searched information according to the first search image, the second search image, the third search image, the fourth search image, and the fifth search image, and provide more image search results from the two dimensions of text matching degree and semantic matching degree.

[0188] In addition, the initiator's demand for image search accuracy can be diverse. For example, when the search image is used for chatting with other objects, the accuracy requirement of the image matching with the game event can be low, and the initiator can need more diverse images to enrich the interest of the chat. At this time, the processing device can perform image search with low accuracy, so as to feed back more diverse image search results to the initiator. When the search image is used to describe the target game event, the image needs to have high emotional fit with the game event, so as to bring the viewer of the target game event a more immersive emotional experience. At this time, the processing device can perform image search with high accuracy to return more suitable search images.

[0189] Based on this, in a possible implementation, the processing device can flexibly provide search images to the initiator based on different image search accuracies.

[0190] The processing device can determine the target image retrieval accuracy corresponding to the initiator, and the target image retrieval accuracy is used to identify the image retrieval accuracy requirement corresponding to the initiator. The above-mentioned first retrieval image, second retrieval image, third retrieval image, fourth retrieval image and fifth retrieval image respectively have corresponding image retrieval accuracy. The processing device can determine the image corresponding to the target image retrieval accuracy among the first retrieval image, second retrieval image, third retrieval image, fourth retrieval image and fifth retrieval image as the retrieval image corresponding to the retrieval information. It can be seen from the above content that since text matching is more direct than semantic matching, the matching accuracy of the retrieval images is, from high to low, the first retrieval image, the second retrieval image, the third retrieval image, the fourth retrieval image and the fifth retrieval image.

[0191] Of course, the processing device can determine the initiator's accuracy requirements in a variety of ways. In one possible implementation, the processing device can allow the initiator to select a retrieval accuracy. The processing device can provide the initiator with a variety of available pending image retrieval accuracies. The initiator can then initiate an accuracy selection operation based on their desired target image retrieval accuracy. This accuracy selection operation is used to determine the image retrieval accuracy from the multiple pending image retrieval accuracies. The processing device can obtain the initiator's corresponding accuracy selection operation and thereby determine the image retrieval accuracy required by the initiator for the search.

[0192] Alternatively, the processing device may independently determine the image retrieval accuracy required by the initiator based on the image application scenario corresponding to the initiator. The processing device may obtain image application scenario information corresponding to the initiator, which is used to identify the image application scenario corresponding to the initiator. The image application scenario is the application scenario of the retrieval image corresponding to the information to be retrieved. For example, the image application scenario information may identify the scenario in which the initiator inputs the information to be retrieved, such as a chat scenario or a video editing scenario.

[0193] Based on the image application scenario information, the processing device can determine the target image retrieval accuracy corresponding to the initiator, thereby ensuring that the retrieval images provided to the initiator are more closely aligned with the application scenario of the retrieval images. Furthermore, by providing retrieval images that match the initiator's accuracy requirements, the number of retrieval images can be appropriately controlled, thereby saving image retrieval time and avoiding the reduction of image retrieval efficiency by retrieving too many images per retrieval.

[0194] It can be understood that the five matching images described above gradually decrease in matching accuracy, and thus the number of images retrieved usually gradually increases. In actual cases, the initiator can not need too many image retrieval results, and thus, in a possible implementation, the processing device can determine whether the number of images retrieved by the retrieval method with higher accuracy meets the number requirement of image retrieval, and if not, combine the retrieval method with lower accuracy to enrich the diversity of image retrieval.

[0195] Taking the first retrieval image and the second retrieval image as examples, after determining the first retrieval image, the processing device can determine whether the first retrieval image meets a preset image number, which is a number for determining whether the number of first retrieval images is sufficient, and can be set according to different scenarios and different needs. If the first retrieval image meets the preset image number, it means that sufficient first retrieval images have been retrieved, and the image retrieval needs of the initiator can be met by the first retrieval image alone. At this time, the processing device can determine the retrieval image corresponding to the to-be-retrieved information for the initiator by the first retrieval image, without retrieving the second retrieval image, so as to provide the initiator with sufficient and high-matching-accuracy retrieval images.

[0196] In response to the first retrieval image not meeting the preset image number, it means that the number of first retrieval images with high matching accuracy obtained by this retrieval method is insufficient. At this time, the processing device can obtain the second retrieval image by the retrieval method of the second retrieval image, that is, perform the step of determining the image corresponding to the target event text of the target game event or the image corresponding to the target emotion type in the text keyword corresponding to the first retrieval image as the second retrieval image, so as to enrich the number and diversity of retrieval images by images with slightly lower matching accuracy. The third, fourth and fifth retrieval images are the same, and after sufficient retrieval images are obtained by the retrieval method with higher matching accuracy, retrieval of images with lower matching accuracy can not be performed; when sufficient retrieval images cannot be obtained, retrieval of images with lower matching accuracy is supplemented.

[0197] As described above, the matching accuracy of retrieval images obtained by different retrieval methods is different, and in general, the higher the matching accuracy of retrieval images, the higher the demand of the initiator for the retrieval image, and the higher the application degree of the retrieval image to the related content of the game event. Based on this, in a possible implementation, in order to meet the image needs of the initiator and provide the initiator with a high-quality image retrieval experience, the processing device can first obtain retrieval images retrieved by multiple retrieval methods when determining the retrieval image corresponding to the to-be-retrieved information, and then extract different numbers of retrieval images from the retrieval images corresponding to the multiple retrieval methods as the retrieval image corresponding to the to-be-retrieved information based on the matching accuracy of the retrieval method, wherein the higher the matching accuracy of the retrieval method, the more the number of images.

[0198] Taking the first search image and the second search image as examples, it can be known from the above content that the matching degree of the first search image with the to-be-searched information is higher than the matching degree of the second search image with the to-be-searched information, and therefore, the processing device can set a preset search image quantity, a first preset proportion corresponding to the first search image, and a second preset proportion corresponding to the second search image, the first preset proportion being used to identify a proportion of the first search image in the search image, the second preset proportion being used to identify a proportion of the second search image in the search image, and the preset search image quantity being a maximum quantity corresponding to the search image.

[0199] The processing device can determine a first image quantity corresponding to the first search image and a second image quantity corresponding to the second search image based on the preset search image quantity, the first preset proportion corresponding to the first search image, and the second preset proportion corresponding to the second search image, the first preset proportion being greater than the second preset proportion. For example, the first image quantity can be a product of the first preset proportion and the preset search image quantity, and the second image quantity can be a product of the second preset proportion and the preset search image quantity.

[0200] The processing device can randomly select, from the first search image, a first image corresponding to the first image quantity, and randomly select, from the second search image, a second image corresponding to the second image quantity, and then determine the first image and the second image as the search image corresponding to the to-be-searched information, so that the initiator can obtain more search images with higher matching accuracy. The search modes of the third search image, the fourth search image, and the fifth search image are the same, for example, the sum of the preset proportions of the first search image, the second search image, the third search image, the fourth search image, and the fifth search image is 1, and the preset proportions decrease in turn. After the processing device obtains these search images, the processing device can first perform the random selection processing, and then display the search images to the initiator.

[0201] Referring to Figure 7 , the preset search image quantity can be 20, the first preset proportion can be 50%, and the second preset proportion can be 20%, that is, the first image quantity is 10, and the second image quantity is 4. The processing device can randomly select 10 first search images from the first search image to join the search image corresponding to the to-be-searched information, and randomly select 4 second search images from the second search image to join the search image corresponding to the to-be-searched information.

[0202] In addition to assigning different weights to images with different matching accuracies in the determination of the search image, the processing device can also set different display priorities for search images with different matching accuracies when displaying.

[0203] In one possible implementation, the processing device may sort the search images corresponding to the pending search information in descending order based on a preset percentage, for example, placing the first search image before the second search image. Based on the image ranking corresponding to the search images, the processing device may display the top N images in the search images to the initiator, where N is less than the preset number of search images. This allows the initiator to preferentially view images with higher matching accuracy, further improving the initiator's image search experience.

[0204] As mentioned above, the search image may be a search image used in a chat scenario. Specifically, in one possible implementation, the information to be searched may be information entered by the initiator in the chat interface, such as information entered in the chat input box or information already sent to the chat information display interface. The processing device may obtain the information to be searched entered by the initiator of the image search in the chat interface.

[0205] When displaying the searched image, the processing device can provide the initiator with a function to quickly send the searched image in the chat. If the initiator wishes to send a searched image, they can directly select an image from the searched image, such as by clicking on it. The processing device can then receive the initiator's image selection operation for a target searched image in the searched images and then send the target searched image in the chat interface. The target searched image can be any image in the searched images.

[0206] Since the search image can fit the emotion caused by the corresponding game event in the information to be retrieved, in addition to providing the search image to the initiator for use by the initiator, the processing device can also automatically generate some content related to the game event and provide it to the initiator.

[0207] For example, when an initiator performs an image search using information to be retrieved related to a game event, they likely intend to use the retrieved image to describe the game event. Based on this, in one possible implementation, the processing device can obtain the game account information corresponding to the initiator of the image search. This game account information can identify the initiator's identity information in the game, and can be used to find the initiator's game content in the game.

[0208] The processing device can obtain information related to the target game event in the game based on the game account information and the target game event, thereby generating initial game event display information corresponding to the target game event. The initial game event display information is used to display the target game event triggered by the initiator in the game corresponding to the game account information. The initial game event display information can include multiple information types, such as text type information, image type information, and video type information.

[0209] For example, when the target game event is "kill", the processing device can obtain information related to the game event of "kill" from the game content corresponding to the game account information of the initiator in the game based on the game account information, such as a game video of the initiator killing in the game. In this way, the processing device can generate information for displaying the target game event more quickly. It can be understood that the initiator may have multiple triggers in the game for the same target game event, that is, the initial game event can be used to display one or more target game events triggered by the initiator in the game.

[0210] The processing device can add a search image in the initial game event display information to generate game event display information corresponding to the search information, so that the search image can enrich the display of the initial game event display information in the emotional dimension for the target game event. The processing device can return the game event display information to the initiator, so as to provide the initiator with automatically generated game event related information, so that the initiator can further enrich the content creation through the information.

[0211] In order to facilitate understanding of the technical solutions provided by the embodiments of the present application, next, an image search method provided by the present application will be introduced in combination with an actual application scenario. In the actual application scenario, the searched image can be an expression package image.

[0212] Referring to Figure 8 , Figure 8 is a schematic diagram of an image search method provided by an embodiment of the present application in an actual application scenario. The initiator can input the search information through the terminal device, for example, the initiator can provide the video address of the search video, the text to be searched, etc. The video address of the search video and the text are written into the corresponding fields of the script, and then the script is started. The server can search the image based on the search information, and send the searched image to an online service, and finally feed back the searched image to the initiator through the terminal device through the Hyper Text Transfer Protocol (HTTP) protocol.

[0213] Firstly, the processing device can perform emotion analysis on the image content and the text included in the image. In the actual application scenario, the processing device can use a deep convolutional neural network (CNN) model to train and learn the emotional type of the image. The emotional type can include: no emotion, happy, sad, surprised, frightened, disgusted, and angry, etc. The trained CNN model can be used as an image emotion recognition model to determine the emotional type corresponding to the image content.

[0214] Then, the processing device can process the text content included in the image. First, the processing device can obtain keywords corresponding to the text content by using a keyword extraction technique, and then obtain a BERT (Bidirectional Encoder Representations from Transformers) model with general Chinese semantics by using a transformer-based bidirectional encoder representation pre-trained on a large amount of Chinese corpus. Then, the model is trained using a general text corpus with sentiment types to learn the sentiment tendency in the text. The sentiment types can include: no emotion, happy, sad, surprised, scared, disgusted, and angry, and the sentiment types of the image are one-to-one corresponding. Finally, the model can be fine-tuned by manually annotating the text content in the image with sentiment types to obtain a final text sentiment recognition model.

[0215] In determining the sentiment type corresponding to the image, the image content corresponding to the sentiment type can be determined by the image sentiment recognition model, and the text content corresponding to the sentiment type can be analyzed by the text sentiment recognition model. The text corresponding to the sentiment type that matches the image sentiment (i.e., corresponding to the same upper-level sentiment type) is retained.

[0216] Thus, the image information obtained by the processing device can include the following parts: ‘text content’ (original text in the image); ‘keywords’ (keyword list); ‘sentiment type’ (sentiment type 1) (sentiment type 2). The sentiment type 1 is the sentiment type corresponding to the image content, and the sentiment type 2 is the sentiment type corresponding to the text content. The processing device can store the image in the online storage to obtain the stored uniform resource locator (URL) address, and store the image information in the database (e.g., Hbase database) for image retrieval.

[0217] The image information is as follows:

[0218] data {‘label’: label information, ‘image_id’: ‘image code’, ‘image_url’: ‘image storage address’}.

[0219] After obtaining the to-be-retrieved information, the processing device can identify the game event corresponding to the to-be-retrieved information by image recognition, text recognition, etc. Then, based on the mapping relationship between the game event and the sentiment type, the sentiment type corresponding to the game event is determined, and the hierarchical image retrieval is performed based on the event text of the game event and the sentiment type.

[0220] As shown in the figure, first, the processing device performs text-based image retrieval, wherein the processing device first performs direct matching retrieval, and based on the obtained sentiment type and event text, uses the distributed full-text retrieval framework (Elasticsearch, referred to as ES) technology to directly search in the 'keyword' and'sentiment type' fields in the Hbase database for images matching the event text and sentiment type as first retrieval images to join the data pool. If the first retrieval image meets the number n1, the subsequent step of determining the retrieval image is performed, and if it does not meet the number n1, the image matching the event text or the sentiment type is taken as the second retrieval image.

[0221] If the second retrieval image meets the number n2, the image retrieval in the data pool is ended; if it does not meet the number n2, the processing device can perform first expansion word retrieval, and first uses a game-side word vector extraction (word2vec) model trained using a game event corpus. The model can extract the word vectors of the event text and the sentiment type of the game event. Since the word vector can represent the semantics of the text, the semantically similar synonymous expansion words can be determined based on the similarity of the word vectors. The synonymous expansion words include synonymous expansion event texts corresponding to the event text and synonymous expansion sentiment types corresponding to the sentiment type.

[0222] In the first expansion word retrieval, the processing device uses the ES technology to search in the 'keyword' and'sentiment' fields in the Hbase database for images matching the event text or the sentiment type based on the event text, the sentiment type, and the synonymous expansion words as third retrieval images.

[0223] Similarly, if the third retrieval image meets the number n3, the image classification retrieval is ended and the subsequent step is performed; if it does not meet the number n3, second expansion word retrieval is performed. The processing device can use the text similarity retrieval (Facebook AI Similarity Search, referred to as Faiss) technology to search in the 'content' field of Hbase for fourth retrieval images with high text similarity based on the event text, the sentiment type, and the synonymous expansion words.

[0224] If the number of the fourth retrieval images meets the number n4, the image classification retrieval is ended and the subsequent step is performed; if it does not meet the number n4, semantic matching-based retrieval is performed. The processing device can train a game field BERT-based semantic matching model based on a large amount of relevant game corpus information collected, and then calculate the semantic similarity between the 'content' text and the sentiment type, the event text, and the synonymous expansion words based on the model to obtain fifth retrieval images to join the data pool.

[0225] After obtaining the data pool, the processing device can randomly select a corresponding number of search images from the data pool as search images corresponding to the information to be retrieved, based on the preset proportions of the number of search methods. At the same time, the processing device can use the preset proportions as the image weights corresponding to each search method, sort the images based on the preset proportions, and display the top n images in the sorted order to the initiator.

[0226] Based on the image retrieval method provided in the above embodiment, the present application also provides an image retrieval device, see Figure 9 , Figure 9 This is a structural block diagram of an image retrieval device provided in an embodiment of the present application. The device includes a first acquisition unit 901, a first recognition unit 902, a first determination unit 903, a second determination unit 904, a third determination unit 905, and a first display unit 906:

[0227] The first acquisition unit 901 is used to acquire information to be retrieved provided by an initiator of image retrieval, wherein the information to be retrieved is used to express a game event, which is an event triggered by a player object in the game;

[0228] The first identification unit 902 is configured to perform event identification on the information to be retrieved, and determine a target game event corresponding to the information to be retrieved;

[0229] The first determining unit 903 is configured to determine a target emotion type corresponding to the target game event;

[0230] The second determining unit 904 is configured to determine the image corresponding to the target emotion type as the first retrieval image;

[0231] The third determining unit 905 is configured to determine a search image corresponding to the information to be retrieved based on the first search image;

[0232] The first display unit 906 is configured to display the search image to the initiator.

[0233] In a possible implementation, the information to be retrieved is a game screen image, and the first acquiring unit 901 is specifically configured to:

[0234] Obtain the video to be retrieved provided by the initiator;

[0235] Extracting video frames from the video to be retrieved to obtain video frame images corresponding to the video to be retrieved;

[0236] Using the video frame image as the game screen image;

[0237] The first identification unit 902 is specifically configured to:

[0238] Extracting information from a preset image position in the game screen image to determine information to be identified corresponding to the game screen image, wherein the information to be identified includes any one or more combinations of text to be identified and images to be identified;

[0239] The game event whose corresponding game event information matches the information to be identified is determined as the target game event corresponding to the information to be retrieved, and the game event information is used to identify the game event.

[0240] The device further comprises a first adding unit, a replacing unit and a second display unit:

[0241] The first adding unit is configured to add the search image to the video frame image to obtain a processed video frame image corresponding to the video frame image;

[0242] The replacing unit is configured to replace the video frame image in the video to be retrieved with the processed video frame image to generate a processed video corresponding to the video to be retrieved;

[0243] The second display unit is used to display the processed video to the initiator.

[0244] In a possible implementation manner, the second determining unit is specifically configured to:

[0245] The corresponding text keywords include the target event text corresponding to the target game event, and the image corresponding to the target emotion type is determined as the first retrieval image, the text keywords corresponding to the image are extracted from the text content included in the image, and the target event text is used to identify the target game event.

[0246] In a possible implementation, the apparatus further includes a fourth determining unit and a fifth determining unit:

[0247] The fourth determining unit is configured to determine whether the first search image meets a preset number of images;

[0248] The fifth determining unit is configured to, in response to the first search image not meeting the preset number of images, determine, in addition to the first search image, an image whose corresponding text keyword includes a target event text corresponding to the target game event, or an image corresponding to the target emotion type as a second search image;

[0249] The third determining unit 905 is specifically configured to:

[0250] A search image corresponding to the information to be retrieved is determined according to the first search image and the second search image.

[0251] In a possible implementation, the apparatus further includes a sixth determining unit and a seventh determining unit:

[0252] The sixth determining unit is configured to determine, based on semantic similarity, a synonymous extended emotion type corresponding to the target emotion type and a synonymous extended event text corresponding to the target event text, wherein the semantic similarity between the synonymous extended emotion type and the target emotion type is greater than a first preset threshold, and the semantic similarity between the synonymous extended event text and the target event text is greater than a second preset threshold;

[0253] The seventh determining unit is configured to determine, in addition to the first search image and the second search image, an image whose corresponding text keyword includes the synonymous expanded event text, or an image corresponding to the synonymous expanded emotion type as a third search image;

[0254] The third determining unit 905 is specifically configured to:

[0255] A search image corresponding to the information to be retrieved is determined according to the first search image, the second search image, and the third search image.

[0256] In a possible implementation manner, the apparatus further includes an eighth determining unit:

[0257] The eighth determination unit is used to determine, as a fourth retrieval image, an image other than the first retrieval image, the second retrieval image, and the third retrieval image, an image whose text similarity between the text content included in the image and any text in the target text set is greater than a third preset threshold, the target text set including the target event text, the synonymous extended event text, the target emotion type, and the synonymous extended emotion type, and the arbitrary text is any one of the target event text, the synonymous extended event text, the target emotion type, and the synonymous extended emotion type;

[0258] The third determining unit 905 is specifically configured to:

[0259] A search image corresponding to the information to be retrieved is determined according to the first search image, the second search image, the third search image, and the fourth search image.

[0260] In a possible implementation, the apparatus further includes a ninth determining unit:

[0261] The ninth determining unit is configured to determine, as a fifth retrieval image, an image other than the first retrieval image, the second retrieval image, the third retrieval image, and the fourth retrieval image, wherein the semantic similarity between the text content included in the image and any text in the target text set is greater than a fourth preset threshold;

[0262] The third determination unit 905 is specifically configured to:

[0263] According to the first search image, the second search image, the third search image, the fourth search image, and the fifth search image, determine the search image corresponding to the to-be-searched information.

[0264] In a possible implementation, the third determination unit 905 is specifically configured to:

[0265] Based on a preset search image quantity, a first preset proportion corresponding to the first search image, and a second preset proportion corresponding to the second search image, determine a first image quantity corresponding to the first search image and a second image quantity corresponding to the second search image, the first preset proportion is used to identify the proportion of the first search image in the search image, the second preset proportion is used to identify the proportion of the second search image in the search image, the first preset proportion is greater than the second preset proportion, and the preset search image quantity is a maximum quantity corresponding to the search image;

[0266] Randomly select a first image corresponding to the first image quantity from the first search image, and randomly select a second image corresponding to the second image quantity from the second search image;

[0267] Determine the first image and the second image as the search image corresponding to the to-be-searched information.

[0268] In a possible implementation, the apparatus further includes a sorting unit:

[0269] The sorting unit is configured to sort the search image corresponding to the to-be-searched information in an order from large to small based on a preset proportion;

[0270] The first display unit 906 is specifically configured to:

[0271] Based on the image sorting corresponding to the search image, display the images in the search image with the top N positions in the sorting to the initiator, N is less than the preset search image quantity, and the sorting of the first search image is before the sorting of the second search image in the image sorting.

[0272] In a possible implementation, the apparatus further includes a second identification unit, an extraction unit, a third identification unit, a tenth determination unit, and an eleventh determination unit:

[0273] The second identification unit is configured to perform emotion identification on a to-be-analyzed image through an image emotion identification model, and determine a first emotion type corresponding to the to-be-analyzed image.

[0274] The extraction unit is configured to extract image text included in the image to be analyzed;

[0275] The third identification unit is configured to perform sentiment identification on the image text by using a text sentiment identification model to determine a second sentiment type corresponding to the image text;

[0276] The tenth determination unit is configured to, if the first sentiment type and the second sentiment type correspond to a same superordinate sentiment type, take both the first sentiment type and the second sentiment type as a sentiment type corresponding to the image to be analyzed, and the superordinate sentiment type includes positive sentiment, negative sentiment, and no sentiment.

[0277] The eleventh determination unit is configured to, if the first sentiment type and the second sentiment type correspond to different superordinate sentiment types, determine the first sentiment type as a sentiment type corresponding to the image to be analyzed.

[0278] In a possible implementation, the apparatus further includes a twelfth determination unit:

[0279] The twelfth determination unit is configured to determine a target image retrieval precision corresponding to the initiator, and the target image retrieval precision is used to identify an image retrieval precision requirement corresponding to the initiator.

[0280] The first retrieval image, the second retrieval image, the third retrieval image, the fourth retrieval image, and the fifth retrieval image each have a corresponding image retrieval precision, and the third determination unit 905 is specifically configured to:

[0281] Determine, as a retrieval image corresponding to the retrieval information, an image corresponding to the target image retrieval precision from among the first retrieval image, the second retrieval image, the third retrieval image, the fourth retrieval image, and the fifth retrieval image.

[0282] In a possible implementation, the twelfth determination unit is specifically configured to:

[0283] Obtain a precision selection operation corresponding to the initiator, and the precision selection operation is used to determine the target image retrieval precision from among multiple pending image retrieval precisions.

[0284] Or,

[0285] Obtain image application scenario information corresponding to the initiator, and the image application scenario information is used to identify an image application scenario corresponding to the initiator, and the image application scenario is an application scenario of a retrieval image corresponding to the retrieval information to be retrieved.

[0286] According to the image application scenario information, a target image retrieval precision corresponding to the initiator is determined.

[0287] In a possible implementation, the apparatus further includes a thirteenth determination unit:

[0288] The thirteenth determination unit is configured to determine a game type corresponding to the target game event.

[0289] The second determination unit is specifically configured to:

[0290] An image corresponding to the target sentiment type and the game type is determined as the first retrieval image.

[0291] In a possible implementation, the first acquisition unit 901 is specifically configured to:

[0292] Acquire information to be retrieved input by an initiator of image retrieval in a chat interface.

[0293] The apparatus further includes a second acquisition unit and a sending unit:

[0294] The second acquisition unit is configured to acquire an image selection operation of the initiator on a target retrieval image in the retrieval image.

[0295] The sending unit is configured to send the target retrieval image in the chat interface.

[0296] In a possible implementation, the apparatus further includes a third acquisition unit, a generation unit, a second addition unit, and a return unit:

[0297] The third acquisition unit is configured to acquire game account information corresponding to an initiator of image retrieval.

[0298] The generation unit is configured to generate initial game event display information corresponding to the target game event according to the game account information and the target game event, the initial game event display information being used to display the target game event triggered by the initiator in a game corresponding to the game account information.

[0299] The second addition unit is configured to add the retrieval image in the initial game event display information, and generate game event display information corresponding to the information to be retrieved.

[0300] The return unit is configured to return the game event display information to the initiator.

[0301] In a possible implementation, the first determination unit 903 is specifically configured to:

[0302] According to the mapping relationship between the game event and the emotion type, a target emotion type corresponding to the target game event is determined according to the target game event;

[0303] Or,

[0304] The target game event is input into a game event emotion recognition model, and a target emotion type corresponding to the target game event is determined;

[0305] The game event emotion recognition model is obtained by training in the following manner:

[0306] A sample game event set is obtained, the sample game event set including a plurality of sample game events, the sample game events having corresponding sample emotion types;

[0307] The sample game events are used as training samples, and the sample emotion types corresponding to the sample game events are used as training labels, so as to train the game event emotion recognition model.

[0308] The embodiments of the present application also provide a computer device, which will be introduced below in combination with the drawings. Referring to FIG. 1, Figure 10 As shown in FIG. 1, the embodiments of the present application provide a device, which can also be a terminal device. The terminal device can be any intelligent terminal including a mobile phone, a tablet computer, a personal digital assistant (PDA), a point of sales (POS), a vehicle-mounted computer, etc. Taking the terminal device as a mobile phone as an example,

[0309] Figure 10 As shown in FIG. 2, which is a block diagram of part of the structure of a mobile phone related to the terminal device provided by the embodiments of the present application. Referring to FIG. 2, Figure 10 The mobile phone includes a radio frequency (RF) circuit 710, a memory 720, an input unit 730, a display unit 740, a sensor 750, an audio circuit 760, a wireless fidelity (WiFi) module 770, a processor 780, and a power supply 790, etc. Those skilled in the art can understand that the structure of the mobile phone shown in FIG. 2 does not constitute a limitation on the mobile phone, and the mobile phone can include more or fewer components than those shown in the figure, or combine certain components, or have a different arrangement of components. Figure 10

[0310] The embodiments of the present application will be described below in combination with Figure 10 The various constituent components of the mobile phone will be described in detail as follows:

[0311] ​The RF circuit 710 can be used for receiving and sending signals in the process of information or communication, in particular, receiving the downlink information from the base station and processing by the processor 780; in addition, sending the uplink data to the base station. Generally, the RF circuit 710 includes but is not limited to an antenna, at least one amplifier, a transceiver, a coupler, a low noise amplifier (LNA), a duplexer, etc. In addition, the RF circuit 710 can also communicate with the network and other devices through wireless communication. The above-mentioned wireless communication can use any communication standard or protocol, including but not limited to global system for mobile communication (GSM), general packet radio service (GPRS), code division multiple access (CDMA), wideband code division multiple access (WCDMA), long term evolution (LTE), email, short messaging service (SMS), etc.

[0312] The memory 720 can be used to store software programs and modules, and the processor 780 can execute various function applications and data processing of the mobile phone by running the software programs and modules stored in the memory 720. The memory 720 can mainly include a program storage area and a data storage area, wherein the program storage area can store an operating system, at least one application program required by a function (such as a sound playing function, an image playing function, etc.), etc.; the data storage area can store data created according to the use of the mobile phone (such as audio data, a phone book, etc.), etc. In addition, the memory 720 can include a high-speed random access memory, and can also include a non-volatile memory, for example, at least one magnetic disk storage device, a flash memory device, or other volatile solid-state memory device.

[0313] The input unit 730 can be used to receive input digital or character information, and to generate key signal input related to the object setting and function control of the mobile phone. Specifically, the input unit 730 may include a touch panel 731 and other input devices 732. The touch panel 731, also known as a touch screen, can collect touch operations of an object on or near it (such as operations performed by an object using a finger, stylus, or any other suitable object or accessory on or near the touch panel 731) and drive the corresponding connection device according to a pre-set program. Optionally, the touch panel 731 may include two parts: a touch detection device and a touch controller. Among them, the touch detection device detects the touch direction of the object and detects the signal caused by the touch operation, and transmits the signal to the touch controller; the touch controller receives the touch information from the touch detection device and converts it into touch point coordinates, which are then sent to the processor 780. It can also receive commands sent by the processor 780 and execute them. In addition, the touch panel 731 can be implemented using various types such as resistive, capacitive, infrared, and surface acoustic wave. In addition to the touch panel 731, the input unit 730 may further include other input devices 732. Specifically, the other input devices 732 may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control keys, switch keys, etc.), a trackball, a mouse, and a joystick.

[0314] The display unit 740 can be used to display information input by the object or information provided to the object and various menus of the mobile phone. The display unit 740 may include a display panel 741. Optionally, the display panel 741 may be configured in the form of a liquid crystal display (LCD), an organic light-emitting diode (OLED), etc. Further, the touch panel 731 may cover the display panel 741. When the touch panel 731 detects a touch operation on or near it, it is transmitted to the processor 780 to determine the type of touch event. Subsequently, the processor 780 provides corresponding visual output on the display panel 741 according to the type of touch event. Although in Figure 10 In the embodiment, the touch panel 731 and the display panel 741 are used as two independent components to realize the input and output functions of the mobile phone, but in some embodiments, the touch panel 731 and the display panel 741 can be integrated to realize the input and output functions of the mobile phone.

[0315] The phone can also include at least one sensor 750, such as an optical sensor, a motion sensor, and other sensors. Specifically, the optical sensor can include an ambient light sensor to adjust the brightness of the display panel 741 according to the brightness of ambient light, and a proximity sensor to turn off the display panel 741 and / or the backlight when the phone is moved to the ear. As one of the motion sensors, the accelerometer sensor can detect the magnitude of acceleration in each direction (usually three axes), and when at rest, the magnitude and direction of gravity, which can be used for applications such as identifying the phone posture (such as switching between landscape and portrait screens, related games, magnetometer posture calibration), vibration recognition related functions (such as pedometers, taps), and the like. As for other sensors that the phone can also be configured, such as a gyroscope, a barometer, a hygrometer, a thermometer, an infrared sensor, and the like, will not be described here.

[0316] The audio circuit 760, the speaker 761, and the microphone 762 can provide an audio interface between the object and the phone. The audio circuit 760 can convert the received audio data into an electrical signal and transmit it to the speaker 761, which converts it into a sound signal for output. On the other hand, the microphone 762 converts the collected sound signal into an electrical signal, which is received by the audio circuit 760 and converted into audio data. The audio data is then output to the processor 780 for processing, and then transmitted to another phone via the RF circuit 710, or output to the memory 720 for further processing.

[0317] WiFi is a short-range wireless transmission technology. The WiFi module 770 can help the object to send and receive emails, browse web pages, and access streaming media, etc. It provides the object with wireless broadband Internet access. Although Figure 10 The WiFi module 770 is shown, but it is understood that it does not belong to the necessary components of the phone, and can be omitted as needed without changing the essence of the invention.

[0318] The processor 780 is the control center of the phone, which connects all parts of the phone through various interfaces and lines, executes various functions of the phone and processes data by running or executing software programs and / or modules stored in the memory 720, and calling data stored in the memory 720, thereby overall detecting the phone. Optionally, the processor 780 can include one or more processing units; preferably, the processor 780 can integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, the object interface, and the application program, etc., and the modem processor mainly processes wireless communication. It can be understood that the above-mentioned modem processor can also not be integrated into the processor 780.

[0319] The mobile phone also includes a power supply 790 (such as a battery) for supplying power to various components. Preferably, the power supply can be logically connected to the processor 780 through a power management system, thereby managing charging, discharging, and power consumption management functions through the power management system.

[0320] Although not shown, the mobile phone may also include a camera, a Bluetooth module, etc., which will not be described in detail here.

[0321] In this embodiment, the processor 780 included in the terminal device further has the following functions:

[0322] Obtaining information to be retrieved provided by an initiator of the image retrieval, wherein the information to be retrieved is used to express a game event, wherein the game event is an event triggered by a player object in the game;

[0323] Performing event recognition on the information to be retrieved to determine a target game event corresponding to the information to be retrieved;

[0324] Determining a target emotion type corresponding to the target game event;

[0325] Determining a search image corresponding to the information to be retrieved according to the target emotion type;

[0326] The retrieved image is presented to the initiator.

[0327] This application embodiment also provides a server, see Figure 11 As shown, Figure 11 The structural diagram of the server 800 provided in the embodiment of the present application, the server 800 may have relatively large differences due to different configurations or performances, and may include one or more central processing units (CPUs) 822 (for example, one or more processors) and memories 832, and one or more storage media 830 (for example, one or more mass storage devices) for storing application programs 842 or data 844. Among them, the memories 832 and the storage media 830 can be temporary storage or persistent storage. The program stored in the storage medium 830 may include one or more modules (not shown in the figure), and each module may include a series of instruction operations on the server. Furthermore, the central processing unit 822 can be configured to communicate with the storage medium 830 to execute a series of instruction operations in the storage medium 830 on the server 800.

[0328] The server 800 may also include one or more power supplies 826, one or more wired or wireless network interfaces 850, one or more input and output interfaces 858, and / or one or more operating systems 841, such as Windows Server 2000. TM, Mac OS X TM , Unix TM ,Linux TM , FreeBSD TM etc.

[0329] The steps performed by the server in the above embodiment can be based on Figure 11 The server structure shown.

[0330] An embodiment of the present application further provides a computer-readable storage medium for storing a computer program, wherein the computer program is used to execute any one of the image retrieval methods described in the aforementioned embodiments.

[0331] An embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the image retrieval method described in any one of the above embodiments.

[0332] Those skilled in the art will understand that all or part of the steps of implementing the above-mentioned method embodiment can be completed by hardware related to program instructions, and the above-mentioned program can be stored in a computer-readable storage medium. When the program is executed, it executes the steps of the above-mentioned method embodiment; and the above-mentioned storage medium can be at least one of the following media: read-only memory (English: read-only memory, abbreviated: ROM), RAM, magnetic disk or optical disk, etc., various media that can store program codes.

[0333] It should be noted that the various embodiments in this specification are described in a progressive manner, and the same or similar parts between the various embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and system embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments. The device and system embodiments described above are merely schematic, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules can be selected according to actual needs to achieve the purpose of the scheme of this embodiment. A person of ordinary skill in the art can understand and implement it without expending creative work.

[0334] The above merely provides one specific embodiment of the present application, but the protection scope of the present application is not limited thereto, any changes or replacements within the technical scope disclosed by the present application, which can be easily thought by any person skilled in the art, should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. An image retrieval method, characterized in that: The method comprises: Obtaining information to be retrieved provided by an initiator of image retrieval, wherein the information to be retrieved includes image information, the image information includes a game screen image, and the image information of the information to be retrieved is used to express a game event, wherein the game event is an event triggered by a player object in the game; Performing event recognition on the information to be retrieved to determine a target game event corresponding to the information to be retrieved; Determining a target emotion type corresponding to the target game event; Determining an image corresponding to the target emotion type as a first retrieval image; Determining, based on the first search image, a search image corresponding to the information to be retrieved; The search image added to the game screen image is presented to the initiator.

2. The method according to claim 1, characterized in that The information to be retrieved provided by the initiator of the image retrieval includes: Obtain the video to be retrieved provided by the initiator; Extracting video frames from the video to be retrieved to obtain video frame images corresponding to the video to be retrieved; Using the video frame image as the game screen image; The performing event identification on the information to be retrieved and determining a target game event corresponding to the information to be retrieved includes: Extracting information from a preset image position in the game screen image to determine information to be identified corresponding to the game screen image, wherein the information to be identified includes any one or more combinations of text to be identified and images to be identified; Determining a game event whose corresponding game event information matches the information to be identified as a target game event corresponding to the information to be retrieved, wherein the game event information is used to identify the game event; The method further comprises: Adding the search image to the video frame image to obtain a processed video frame image corresponding to the video frame image; Replacing the video frame image in the video to be retrieved with the processed video frame image to generate a processed video corresponding to the video to be retrieved; The processed video is presented to the initiator.

3. The method according to claim 1, characterized in that The step of determining the image corresponding to the target emotion type as the first retrieval image includes: The corresponding text keywords include the target event text corresponding to the target game event, and the image corresponding to the target emotion type is determined as the first retrieval image, the text keywords corresponding to the image are extracted from the text content included in the image, and the target event text is used to identify the target game event.

4. The method according to claim 3, characterized in that The method further comprises: determining whether the first search image satisfies a preset number of images; In response to the first search image not meeting the preset number of images, determining, in addition to the first search image, an image whose corresponding text keyword includes a target event text corresponding to the target game event, or an image corresponding to the target emotion type as a second search image; The step of determining, based on the first search image, a search image corresponding to the information to be searched includes: A search image corresponding to the information to be retrieved is determined according to the first search image and the second search image.

5. The method according to claim 4, characterized in that The method further comprises: Based on semantic similarity, determining a synonymous extended emotion type corresponding to the target emotion type and a synonymous extended event text corresponding to the target event text, wherein the semantic similarity between the synonymous extended emotion type and the target emotion type is greater than a first preset threshold, and the semantic similarity between the synonymous extended event text and the target event text is greater than a second preset threshold; Determining, in addition to the first search image and the second search image, an image whose corresponding text keyword includes the synonymous expanded event text, or an image corresponding to the synonymous expanded emotion type as a third search image; The determining, based on the first search image and the second search image, a search image corresponding to the information to be retrieved includes: A search image corresponding to the information to be retrieved is determined according to the first search image, the second search image, and the third search image.

6. The method according to claim 5, characterized in that The method further comprises: Determine, as a fourth retrieval image, an image, other than the first retrieval image, the second retrieval image, and the third retrieval image, an image whose text similarity between the text content included in the image and any text in the target text set is greater than a third preset threshold, wherein the target text set includes the target event text, the synonymous expanded event text, the target emotion type, and the synonymous expanded emotion type, and the arbitrary text is any one of the target event text, the synonymous expanded event text, the target emotion type, and the synonymous expanded emotion type; The determining, based on the first search image, the second search image, and the third search image, a search image corresponding to the information to be retrieved includes: A search image corresponding to the information to be retrieved is determined according to the first search image, the second search image, the third search image, and the fourth search image.

7. The method according to claim 6, characterized in that The method further comprises: determining, as a fifth retrieval image, an image other than the first retrieval image, the second retrieval image, the third retrieval image, and the fourth retrieval image, an image whose text content has a semantic similarity with any text in the target text set greater than a fourth preset threshold; The determining, based on the first search image, the second search image, the third search image, and the fourth search image, of a search image corresponding to the information to be retrieved includes: A search image corresponding to the information to be retrieved is determined according to the first search image, the second search image, the third search image, the fourth search image, and the fifth search image.

8. The method according to claim 4, characterized in that The determining, based on the first search image and the second search image, a search image corresponding to the information to be retrieved includes: Determining the number of first images corresponding to the first search image and the number of second images corresponding to the second search image based on a preset number of search images, a first preset proportion corresponding to the first search image, and a second preset proportion corresponding to the second search image, wherein the first preset proportion is used to identify the proportion of the first search image in the search images, and the second preset proportion is used to identify the proportion of the second search image in the search images, the first preset proportion is greater than the second preset proportion, and the preset number of search images is the maximum number corresponding to the search images; randomly selecting a number of first images corresponding to the first number of images from the first search images, and randomly selecting a number of second images corresponding to the second number of images from the second search images; The first image and the second image are determined as search images corresponding to the information to be retrieved.

9. The method according to claim 8, characterized in that The method further comprises: Sorting the search images corresponding to the information to be retrieved in descending order based on preset proportions; The presenting the retrieved image to the initiator includes: Based on the image ranking corresponding to the search image, the images ranked in the top N positions in the search image are displayed to the initiator, where N is less than the preset number of search images, and the ranking of the first search image in the image ranking is before the ranking of the second search image.

10. The method according to claim 1, characterized in that The method further comprises: Performing emotion recognition on the image to be analyzed by using an image emotion recognition model to determine a first emotion type corresponding to the image to be analyzed; extracting image text included in the image to be analyzed; Performing emotion recognition on the image text using a text emotion recognition model to determine a second emotion type corresponding to the image text; If the first emotion type and the second emotion type correspond to the same upper-level emotion type, both the first emotion type and the second emotion type are used as the emotion type corresponding to the image to be analyzed, and the upper-level emotion type includes positive emotion, negative emotion, and no emotion; If the first emotion type and the second emotion type correspond to different upper-level emotion types, the first emotion type is determined as the emotion type corresponding to the image to be analyzed.

11. The method according to claim 7, characterized in that The method further comprises: Determining a target image retrieval accuracy corresponding to the initiator, where the target image retrieval accuracy is used to identify an image retrieval accuracy requirement corresponding to the initiator; The first retrieval image, the second retrieval image, the third retrieval image, the fourth retrieval image, and the fifth retrieval image respectively have corresponding image retrieval accuracies, and determining the retrieval image corresponding to the information to be retrieved based on the first retrieval image, the second retrieval image, the third retrieval image, the fourth retrieval image, and the fifth retrieval image includes: An image corresponding to the target image retrieval accuracy among the first retrieval image, the second retrieval image, the third retrieval image, the fourth retrieval image, and the fifth retrieval image is determined as a retrieval image corresponding to the retrieval information.

12. The method according to claim 11, characterized in that Determining the target image retrieval accuracy corresponding to the initiator includes: Obtaining a precision selection operation corresponding to the initiator, wherein the precision selection operation is used to determine the target image retrieval precision from a plurality of pending image retrieval precisions; or, Acquire image application scenario information corresponding to the initiator, where the image application scenario information is used to identify the image application scenario corresponding to the initiator, and the image application scenario is an application scenario of the retrieval image corresponding to the information to be retrieved; The target image retrieval accuracy corresponding to the initiator is determined according to the image application scenario information.

13. The method according to claim 1, wherein The method further comprises: Determining the game type corresponding to the target game event; The step of determining the image corresponding to the target emotion type as the first retrieval image includes: An image corresponding to the target emotion type and the game type is determined as the first retrieval image.

14. The method according to claim 1, wherein The information to be retrieved provided by the initiator of the image retrieval includes: Obtaining the information to be retrieved input by the initiator of the image retrieval in the chat interface; The method further comprises: Acquire an image selection operation of the initiator on a target search image in the search images; Send the target retrieval image in the chat interface.

15. The method according to claim 14, characterized in that The method further comprises: Obtaining the game account information corresponding to the initiator of the image retrieval; generating, based on the game account information and the target game event, initial game event display information corresponding to the target game event, wherein the initial game event display information is used to display the target game event triggered by the initiator in the game corresponding to the game account information; Add the search image to the initial game event display information to generate game event display information corresponding to the information to be retrieved; The game event display information is returned to the initiator.

16. The method according to claim 1, wherein Determining the target emotion type corresponding to the target game event includes: Determining a target emotion type corresponding to the target game event based on a mapping relationship between game events and emotion types; or, Inputting the target game event into a game event emotion recognition model to determine a target emotion type corresponding to the target game event; The game event emotion recognition model is trained in the following way: Acquire a sample game event set, wherein the sample game event set includes a plurality of sample game events, and the sample game events have corresponding sample emotion types; The sample game events are used as training samples, and the sample emotion types corresponding to the sample game events are used as training labels, and the game event emotion recognition model is obtained through training.

17. An image retrieval device, characterized in that: The device includes a first acquisition unit, a first identification unit, a first determination unit, a second determination unit, a third determination unit and a first display unit: The first acquisition unit is configured to acquire information to be retrieved provided by an initiator of image retrieval, wherein the information to be retrieved includes image information, which includes a game screen image, and the image information of the information to be retrieved is used to express a game event, which is an event triggered by a player object in the game; The first identification unit is configured to perform event identification on the information to be retrieved, and determine a target game event corresponding to the information to be retrieved; The first determining unit is configured to determine a target emotion type corresponding to the target game event; The second determining unit is configured to determine an image corresponding to the target emotion type as a first retrieval image; The third determining unit is configured to determine a search image corresponding to the information to be retrieved based on the first search image; The first display unit is used to display the search image added to the game screen image to the initiator.

18. The device according to claim 17, characterized in that The first acquiring unit is specifically configured to: Obtain the video to be retrieved provided by the initiator; Extracting video frames from the video to be retrieved to obtain video frame images corresponding to the video to be retrieved; Using the video frame image as the game screen image; The first identification unit is specifically configured to: Extracting information from a preset image position in the game screen image to determine information to be identified corresponding to the game screen image, wherein the information to be identified includes any one or more combinations of text to be identified and images to be identified; determining a game event whose corresponding game event information matches the information to be identified as a target game event corresponding to the information to be retrieved, wherein the game event information identifies the game event; The device further comprises a first adding unit, a replacing unit and a second display unit: The first adding unit is configured to add the search image to the video frame image to obtain a processed video frame image corresponding to the video frame image; The replacing unit is configured to replace the video frame image in the video to be retrieved with the processed video frame image to generate a processed video corresponding to the video to be retrieved; The second display unit is used to display the processed video to the initiator.

19. The device according to claim 17, characterized in that The second determining unit is specifically configured to: The corresponding text keywords include the target event text corresponding to the target game event, and the image corresponding to the target emotion type is determined as the first retrieval image, the text keywords corresponding to the image are extracted from the text content included in the image, and the target event text is used to identify the target game event.

20. The device according to claim 19, characterized in that The apparatus further includes a fourth determining unit and a fifth determining unit: The fourth determining unit is configured to determine whether the first search image meets a preset number of images; The fifth determining unit is configured to, in response to the first search image not meeting the preset number of images, determine, in addition to the first search image, an image whose corresponding text keyword includes a target event text corresponding to the target game event, or an image corresponding to the target emotion type as a second search image; The third determining unit is specifically configured to: A search image corresponding to the information to be retrieved is determined according to the first search image and the second search image.

21. The device according to claim 20, characterized in that The apparatus further includes a sixth determining unit and a seventh determining unit: The sixth determining unit is configured to determine, based on semantic similarity, a synonymous extended emotion type corresponding to the target emotion type and a synonymous extended event text corresponding to the target event text, wherein the semantic similarity between the synonymous extended emotion type and the target emotion type is greater than a first preset threshold, and the semantic similarity between the synonymous extended event text and the target event text is greater than a second preset threshold; The seventh determining unit is configured to determine, in addition to the first search image and the second search image, an image whose corresponding text keyword includes the synonymous expanded event text, or an image corresponding to the synonymous expanded emotion type as a third search image; The third determining unit is specifically configured to: A search image corresponding to the information to be retrieved is determined according to the first search image, the second search image, and the third search image.

22. The device according to claim 21, characterized in that The apparatus further includes an eighth determining unit: The eighth determination unit is used to determine, as a fourth retrieval image, an image other than the first retrieval image, the second retrieval image, and the third retrieval image, an image whose text similarity between the text content included in the image and any text in the target text set is greater than a third preset threshold, the target text set including the target event text, the synonymous extended event text, the target emotion type, and the synonymous extended emotion type, and the arbitrary text is any one of the target event text, the synonymous extended event text, the target emotion type, and the synonymous extended emotion type; The third determining unit is specifically configured to: A search image corresponding to the information to be retrieved is determined according to the first search image, the second search image, the third search image, and the fourth search image.

23. The device according to claim 22, characterized in that The apparatus further includes a ninth determining unit: The ninth determining unit is configured to determine, as a fifth retrieval image, an image other than the first retrieval image, the second retrieval image, the third retrieval image, and the fourth retrieval image, wherein the semantic similarity between the text content included in the image and any text in the target text set is greater than a fourth preset threshold; The third determining unit is specifically configured to: A search image corresponding to the information to be retrieved is determined according to the first search image, the second search image, the third search image, the fourth search image, and the fifth search image.

24. The device according to claim 20, characterized in that The third determining unit is specifically configured to: Determining the number of first images corresponding to the first search image and the number of second images corresponding to the second search image based on a preset number of search images, a first preset proportion corresponding to the first search image, and a second preset proportion corresponding to the second search image, wherein the first preset proportion is used to identify the proportion of the first search image in the search images, and the second preset proportion is used to identify the proportion of the second search image in the search images, the first preset proportion is greater than the second preset proportion, and the preset number of search images is the maximum number corresponding to the search images; randomly selecting a number of first images corresponding to the first number of images from the first search images, and randomly selecting a number of second images corresponding to the second number of images from the second search images; The first image and the second image are determined as search images corresponding to the information to be retrieved.

25. The device according to claim 24, characterized in that The device further comprises a sorting unit: The sorting unit is configured to sort the search images corresponding to the information to be retrieved in descending order based on a preset proportion; The first display unit is specifically used for: Based on the image ranking corresponding to the search image, the images ranked in the top N positions in the search image are displayed to the initiator, where N is less than the preset number of search images, and the ranking of the first search image in the image ranking is before the ranking of the second search image.

26. The device according to claim 17, characterized in that The apparatus further includes a second recognition unit, an extraction unit, a third recognition unit, a tenth determination unit, and an eleventh determination unit: The second recognition unit is configured to perform emotion recognition on the image to be analyzed by using an image emotion recognition model, and determine a first emotion type corresponding to the image to be analyzed; The extraction unit is used to extract the image text included in the image to be analyzed; The third recognition unit is configured to perform emotion recognition on the image text using a text emotion recognition model to determine a second emotion type corresponding to the image text; the tenth determining unit is configured to use the first emotion type and the second emotion type as emotion types corresponding to the image to be analyzed if the first emotion type and the second emotion type correspond to the same upper-level emotion type, where the upper-level emotion type includes positive emotion, negative emotion, and no emotion; The eleventh determining unit is configured to determine the first emotion type as the emotion type corresponding to the image to be analyzed if the first emotion type and the second emotion type correspond to different upper-level emotion types.

27. The device according to claim 23, characterized in that The apparatus further includes a twelfth determining unit: The twelfth determining unit is configured to determine a target image retrieval accuracy corresponding to the initiator, where the target image retrieval accuracy is used to identify an image retrieval accuracy requirement corresponding to the initiator; The first retrieval image, the second retrieval image, the third retrieval image, the fourth retrieval image, and the fifth retrieval image respectively have corresponding image retrieval accuracies, and the third determining unit 905 is specifically configured to: An image corresponding to the target image retrieval accuracy among the first retrieval image, the second retrieval image, the third retrieval image, the fourth retrieval image, and the fifth retrieval image is determined as a retrieval image corresponding to the retrieval information.

28. The device according to claim 27, characterized in that The twelfth determining unit is specifically configured to: Obtaining a precision selection operation corresponding to the initiator, wherein the precision selection operation is used to determine the target image retrieval precision from a plurality of pending image retrieval precisions; or, Acquire image application scenario information corresponding to the initiator, where the image application scenario information is used to identify the image application scenario corresponding to the initiator, and the image application scenario is an application scenario of the retrieval image corresponding to the information to be retrieved; The target image retrieval accuracy corresponding to the initiator is determined according to the image application scenario information.

29. The device according to claim 17, wherein The apparatus further includes a thirteenth determining unit: The thirteenth determining unit is used to determine the game type corresponding to the target game event; The second determining unit is specifically configured to: An image corresponding to the target emotion type and the game type is determined as the first retrieval image.

30. The device according to claim 17, wherein The first acquiring unit is specifically configured to: Obtaining the information to be retrieved input by the initiator of the image retrieval in the chat interface; The device further includes a second acquiring unit and a sending unit: The second acquiring unit is configured to acquire an image selection operation performed by the initiator on a target search image in the search images; The sending unit is used to send the target retrieval image in the chat interface.

31. The device according to claim 30, characterized in that The device further includes a third acquiring unit, a generating unit, a second adding unit and a returning unit: The third obtaining unit is used to obtain the game account information corresponding to the initiator of the image retrieval; The generating unit is configured to generate initial game event display information corresponding to the target game event based on the game account information and the target game event, wherein the initial game event display information is used to display the target game event triggered by the initiator in the game corresponding to the game account information; The second adding unit is configured to add the search image to the initial game event display information to generate game event display information corresponding to the information to be retrieved; The returning unit is configured to return the game event display information to the initiator.

32. The device according to claim 17, wherein The first determining unit is specifically configured to: Determining a target emotion type corresponding to the target game event based on a mapping relationship between game events and emotion types; or, Inputting the target game event into a game event emotion recognition model to determine a target emotion type corresponding to the target game event; The game event emotion recognition model is trained in the following way: Acquire a sample game event set, wherein the sample game event set includes a plurality of sample game events, and the sample game events have corresponding sample emotion types; The sample game events are used as training samples, and the sample emotion types corresponding to the sample game events are used as training labels, and the game event emotion recognition model is obtained through training.

33. A computer device, characterized in that: The computer device includes a processor and a memory: The memory is used to store program code and transmit the program code to the processor; The processor is configured to execute the image retrieval method according to any one of claims 1 to 16 according to instructions in the program code.

34. A computer-readable storage medium, characterized in that The computer-readable storage medium is used to store a computer program, and the computer program is used to execute the image retrieval method according to any one of claims 1 to 16.

35. A computer program product comprising instructions, which, when executed on a computer, enables the computer to execute the image retrieval method according to any one of claims 1 to 16.

Images