Method and equipment for acquiring enhanced multimedia information

Abstract

The invention aims to provide a method and equipment for acquiring enhanced multimedia information, and the method specifically comprises the following steps: acquiring a multimedia query request input by a user, the multimedia query request comprising corresponding query multimedia information; in response to the multimedia query request, querying and determining one or more pieces of corresponding matched multimedia information in a local database according to the query multimedia information; a corresponding multimedia generation request is sent to a corresponding network device, and the multimedia generation request comprises the query multimedia information and the one or more pieces of matched multimedia information; and receiving enhanced multimedia information returned by the network equipment based on the multimedia generation request. The resource of the user equipment side is reasonably utilized, the cloud side resource is saved, the data source matching is more flexible, the database matching is more fit, and the user experience is improved.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of communication, and in particular to a technology for obtaining enhanced multimedia information. BACKGROUND

[0002] Retrieval-Augmented Generation (RAG) is a technology that combines information retrieval and multimedia content generation, such as text generation technology, etc. It is widely used in the field of Natural Language Processing (NLP), especially in the construction of large language models. The core idea of this technology is to use the information provided by external data sources to assist in generating more accurate and rich content during text generation. The advantage of retrieval-augmented generation technology is that it can combine large-scale external knowledge to generate more rich and accurate text. This makes the model perform better when dealing with complex problems or tasks that require a lot of background knowledge. At the same time, this technology also helps to reduce the amount of data required for model training, as it can use external data sources to make up for the lack of model knowledge. SUMMARY

[0003] One object of the present application is to provide a device for obtaining enhanced multimedia information.

[0004] According to one aspect of the present application, a method for obtaining enhanced multimedia information is provided, applied to a user device, wherein the method comprises:

[0005] Obtaining a multimedia query request input by a user, wherein the multimedia query request includes corresponding query multimedia information;

[0006] In response to the multimedia query request, determining one or more matching multimedia information corresponding to the query multimedia information in the local database;

[0007] Sending a corresponding multimedia generation request to a corresponding network device, wherein the multimedia generation request includes the query multimedia information and the one or more matching multimedia information;

[0008] Receiving enhanced multimedia information returned by the network device based on the multimedia generation request, wherein the enhanced multimedia information includes output multimedia content determined by inputting the query multimedia information and the one or more matching multimedia information into a corresponding matching model.

[0009] According to another aspect of the present application, a method for obtaining enhanced multimedia information is provided, applied to a network device, wherein the method comprises:

[0010] receiving a multimedia generation request sent by a user device, wherein the multimedia generation request comprises query multimedia information and one or more matched multimedia information corresponding to the query multimedia information, and the one or more matched multimedia information is determined by the user device according to the query multimedia information in a local database;

[0011] inputting the query multimedia information and the one or more matched multimedia information into a matching model, and determining output multimedia content of the matching model as corresponding enhanced multimedia information;

[0012] returning the enhanced multimedia information to the user device.

[0013] According to an aspect of the present application, a method for obtaining enhanced multimedia information is provided, wherein the method comprises:

[0014] a user device obtains a multimedia query request input by a user, wherein the multimedia query request comprises corresponding query multimedia information; in response to the multimedia query request, one or more matched multimedia information corresponding to the query multimedia information is determined according to the query multimedia information in a local database; and a multimedia generation request is sent to a corresponding network device, wherein the multimedia generation request comprises the query multimedia information and the one or more matched multimedia information;

[0015] the network device receives the multimedia generation request sent by the user device, inputs the query multimedia information and the one or more matched multimedia information into a matching model, determines output multimedia content of the matching model as corresponding enhanced multimedia information, and returns the enhanced multimedia information to the user device;

[0016] the user device receives enhanced multimedia information returned by the network device based on the multimedia generation request, wherein the enhanced multimedia information comprises output multimedia content determined by inputting the query multimedia information and the one or more matched multimedia information into a matching model.

[0017] According to an aspect of the present application, a user device for obtaining enhanced multimedia information is provided, wherein the user device comprises:

[0018] a first module configured to obtain a multimedia query request input by a user, wherein the multimedia query request comprises corresponding query multimedia information;

[0019] a second module configured to, in response to the multimedia query request, determine one or more matched multimedia information corresponding to the query multimedia information according to the query multimedia information in a local database;

[0020] a third module configured to send a corresponding multimedia generation request to a corresponding network device, wherein the multimedia generation request comprises the query multimedia information and the one or more matched multimedia information;

[0021] a fourth module configured to receive enhanced multimedia information returned by the network device based on the multimedia generation request, wherein the enhanced multimedia information comprises output multimedia content determined by inputting the query multimedia information and the one or more matched multimedia information into a corresponding matching model.

[0022] According to another aspect of the present application, there is provided a network device for obtaining enhanced multimedia information, wherein the network device comprises:

[0023] a first module configured to receive a multimedia generation request sent by a user device, wherein the multimedia generation request comprises query multimedia information and one or more matched multimedia information corresponding to the query multimedia information, the one or more matched multimedia information being determined by the user device in a local database according to the query multimedia information;

[0024] a second module configured to input the query multimedia information and the one or more matched multimedia information into a corresponding matching model, and determine output multimedia content of the matching model as corresponding enhanced multimedia information;

[0025] a third module configured to return the enhanced multimedia information to the user device.

[0026] According to an aspect of the present application, there is provided a computer device, wherein the device comprises:

[0027] a processor; and

[0028] a memory arranged to store computer executable instructions which, when executed, cause the processor to perform the steps of any of the above methods.

[0029] According to an aspect of the present application, there is provided a computer readable storage medium having stored thereon computer programs / instructions which, when executed, cause a system to perform the steps of any of the above methods.

[0030] According to an aspect of the present application, there is provided a computer program product comprising computer programs / instructions which, when executed by a processor, implement the steps of any of the above methods.

[0031] Compared with the prior art, the application determines one or more matching multimedia information by matching in the local database of the user equipment, and generates enhanced multimedia information at the network equipment end based on the query multimedia information and the one or more matching multimedia information, reasonably utilizes the resources of the user equipment end, saves the cloud resources, and makes the matching data source more flexible and the database matching more suitable, thereby improving the user experience. BRIEF DESCRIPTION OF DRAWINGS

[0032] Other features, objects, and advantages of the application will become more apparent from the following detailed description of non-limiting embodiments, read in conjunction with the accompanying drawings:

[0033] Figure 1 A flow chart of a system method for obtaining enhanced multimedia information is shown according to one embodiment of the application;

[0034] Figure 2 A flow chart of a method for obtaining enhanced multimedia information is shown according to one embodiment of the application;

[0035] Figure 3 A flow chart of a method for obtaining enhanced multimedia information is shown according to another embodiment of the application;

[0036] Figure 4 A device structure diagram of a user equipment is shown according to one embodiment of the application;

[0037] Figure 5 A device structure diagram of a network equipment is shown according to one embodiment of the application;

[0038] Figure 6 An exemplary system that can be used for implementing the various embodiments described herein is shown.

[0039] The same or similar reference numerals in the drawings represent the same or similar components. DETAILED DESCRIPTION

[0040] The application will be further described in detail below with reference to the accompanying drawings.

[0041] In one typical configuration of the application, the terminal, the device of the service network and the trusted party each include one or more processors (for example, a central processing unit (CPU), an input / output interface, a network interface and a memory.

[0042] Memory can include non-persistent memory and / or volatile memory, random access memory (RAM), and / or non-volatile memory, e.g., read only memory (ROM), EPROM, and / or the like. Memory is an example of computer readable media.

[0043] Computer readable media includes permanent and non-permanent, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PCM), programmable random access memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read only memory (ROM), electrically erasable programmable read only memory (EEPROM), flash memory or other memory technology, compact disc read only memory (CD-ROM), digital versatile discs (DVDs) or other optical storage, magnetic cassettes, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device.

[0044] The device referred to in the present application includes, but is not limited to, a user device, a network device, or a device formed by integrating a user device and a network device through a network. The user device includes, but is not limited to, any kind of mobile electronic product capable of human-computer interaction (for example, human-computer interaction through a touch panel), such as a smart phone, a tablet computer, etc. The mobile electronic product can adopt any operating system, such as an Android operating system, an iOS operating system, etc. The network device includes an electronic device capable of automatically performing numerical calculation and information processing according to pre-set or stored instructions. The hardware of the network device includes, but is not limited to, a microprocessor, an application specific integrated circuit (ASIC), a programmable logic device (PLD), a field programmable gate array (FPGA), a digital signal processor (DSP), an embedded device, etc. The network device includes, but is not limited to, a computer, a network host, a single network server, a plurality of network servers, or a cloud formed by a plurality of servers. The cloud is formed by a large number of computers or network servers based on cloud computing. Cloud computing is a kind of distributed computing, which is a virtual supercomputer formed by a group of loosely coupled computer clusters. The network includes, but is not limited to, the Internet, a wide area network, a metropolitan area network, a local area network, a VPN network, a wireless ad hoc network, etc. Preferably, the device can also be a program running on the user device, the network device, or a device formed by integrating a user device and a network device, a network device, a touch terminal, or a device formed by integrating a network device and a touch terminal through a network.

[0045] Of course, those skilled in the art should understand that the above device is only an example, and other existing or future devices, such as devices that can be applicable to the present application, should also be included in the protection scope of the present application, and are hereby incorporated by reference.

[0046] In the description of the present application, the meaning of "a plurality of" is two or more, unless otherwise explicitly specified.

[0047] Figure 1 A system method for acquiring enhanced multimedia information is shown according to an aspect of the present application. The method is applied to a system formed by a user device 100 and a network device 200. The method includes:

[0048] The user equipment acquires a multimedia query request input by a user, wherein the multimedia query request comprises corresponding query multimedia information; in response to the multimedia query request, one or more pieces of matching multimedia information corresponding to the query multimedia information are determined by querying a local database according to the query multimedia information; a corresponding multimedia generation request is sent to a corresponding network equipment, wherein the multimedia generation request comprises the query multimedia information and the one or more pieces of matching multimedia information;

[0049] The network equipment receives the corresponding multimedia generation request sent by the user equipment, inputs the query multimedia information and the one or more pieces of matching multimedia information into a corresponding matching model, determines output multimedia content of the matching model as corresponding enhanced multimedia information, and returns the enhanced multimedia information to the user equipment;

[0050] The user equipment receives the enhanced multimedia information returned by the network equipment based on the multimedia generation request, wherein the enhanced multimedia information comprises output multimedia content determined by inputting the query multimedia information and the one or more pieces of matching multimedia information into a corresponding matching model.

[0051] The multimedia information comprises, but is not limited to, one or a combination of text, image, audio or video, etc. In this application, text is mainly taken as an example to explain the following embodiments, and those skilled in the art should understand that the following embodiments are also applicable to other various multimedia information and their combination corresponding scenarios.

[0052] Specifically, as Figure 2 A method for acquiring enhanced multimedia information is shown, which is applied to user equipment 100, and the method comprises steps S101, S102, S103 and S104. In step S101, a multimedia query request input by a user is acquired, wherein the multimedia query request comprises corresponding query multimedia information; in step S102, in response to the multimedia query request, one or more pieces of matching multimedia information corresponding to the query multimedia information are determined by querying a local database according to the query multimedia information; in step S103, a corresponding multimedia generation request is sent to a corresponding network equipment, wherein the multimedia generation request comprises the query multimedia information and the one or more pieces of matching multimedia information; in step S104, enhanced multimedia information returned by the network equipment based on the multimedia generation request is received, wherein the enhanced multimedia information comprises output multimedia content determined by inputting the query multimedia information and the one or more pieces of matching multimedia information into a corresponding matching model.

[0053] In step S101, a multimedia query request input by a user is acquired, wherein the multimedia query request comprises corresponding query multimedia information. For example, the user holds a user device (e.g., a node in a local area network or a mobile phone, etc.), and the user device can acquire the multimedia query request input by the user. For example, the user device comprises an input device (e.g., a keyboard, a mouse, a touch screen, etc.) for collecting input operations of the user. The user can click a corresponding page through the input device and input multimedia information expected to be queried in a multimedia query page, so as to generate the corresponding multimedia query request. In this case, the multimedia query request can be generated by the user operating the user device, or the user device can receive input operations of the user operating other user devices, or the user device directly receives the multimedia query request generated by other user devices based on input operations of the user, etc. In some cases, the multimedia query request can be issued by a network device. The multimedia query request is generated by other user devices based on corresponding user operations and sent to the network device. Further, the network device stores a type description field of a database type of a local database of the user device (e.g., the device identifier information of the user device and the type description field have a mapping relationship), and determines that the type description field of the local database of the user device matches the multimedia field of the query multimedia information based on the query multimedia information in the multimedia query request. Then, the network device issues the multimedia query request to the user device, etc.

[0054] In step S102, in response to the multimedia query request, one or more pieces of matching multimedia information corresponding to the query multimedia information are determined in the local database. For example, the user device locally or in a local area network stores a corresponding local database, and the local database is used to store related text segments, etc. In some cases, the database type of the local database can be different types of multimedia libraries based on different settings, such as programming, movies, or entertainment classified according to vocabulary types, or text, images, audio, or video classified according to data types, etc. The local database comprises a multimedia database constructed based on commonly used multimedia and / or private multimedia of the user / current local area network. The number of the local database can be one or more. After the user device acquires the corresponding text query request, the local database is called and the fields contained in the query text information are matched with the text segments in the local database, so that one or more pieces of matching text information related to the query text information can be determined. For example, the matching is performed based on semantic matching, vector similarity matching of fields and text segments, etc. The matching text information is used to indicate knowledge segments related to the query text information in the database, such as similarity greater than or equal to a similarity threshold, or the cosine distance of fields and knowledge segments in a vector space is less than or equal to a preset distance threshold, etc.

[0055] In step S103, a corresponding multimedia generation request is sent to a corresponding network device, wherein the multimedia generation request includes the query multimedia information and the one or more matching multimedia information. For example, after the user device obtains the one or more matching multimedia information, a corresponding multimedia generation request is generated according to the corresponding query multimedia information and the one or more matching multimedia information, and the multimedia generation request is sent to the network device to obtain the final output multimedia content using cloud resources. Specifically, the user device sends the corresponding multimedia generation request to the network device, and the network device end establishes a corresponding matching model (such as a large language model or other comprehensive model, etc.), inputs the corresponding query multimedia information and the one or more matching multimedia information into the matching model, and obtains the output multimedia content output by the matching model. The matching model is used for training according to the demand, such as a model for enhancing a single type of multimedia information trained according to any single type of multimedia information, such as a large language model, or a multi-modal model for enhancing multiple types of multimedia information trained according to multiple types of multimedia information, etc.

[0056] In step S104, enhanced multimedia information returned by the network device based on the multimedia generation request is received, wherein the enhanced multimedia information includes output multimedia content determined by inputting the query multimedia information and the one or more matching multimedia information into a corresponding matching model. For example, the matching model refers to a deep learning model trained using a large amount of multimedia data, which can generate enhanced multimedia content or understand the meaning of multimedia content. The matching model can process tasks of multiple multimedia contents, such as multimedia classification, question and answer, dialogue, etc. The network device inputs the corresponding query multimedia information and the one or more matching multimedia information into the matching model, and obtains the output multimedia content output by the matching model, so as to take the output multimedia content as the final enhanced multimedia information of this multimedia query, and returns the enhanced multimedia information to the user device. If the multimedia query request is generated by other user devices based on the input operation of other users, the network device will also return the enhanced multimedia information to the other user devices, such as enhanced text information, enhanced image information, enhanced audio information or enhanced video information in the current scene / context, etc. In some cases, the user device can also update the corresponding local database based on the enhanced multimedia information and the corresponding multimedia query request.

[0057] In some embodiments, the step S102 comprises a sub-step S1021 (not shown) and a sub-step S1022 (not shown); in the step S1021, one or more query field information in the query multimedia information is extracted in response to the multimedia query request; in the step S1022, the corresponding matching multimedia information is determined according to the one or more query field information in the local database. For example, after the user equipment obtains the corresponding query multimedia information, the query multimedia information is field extracted, so that specific information is taken out from the query multimedia information as a new field, and the specific method includes but is not limited to regular extraction, JSON extraction, key-value pair extraction, specific function extraction, etc. The user equipment matches the extracted query field information with the knowledge fragments in the local database, so as to determine the corresponding matching multimedia information, such as determining the knowledge fragments in the local database that match each query field information, so as to obtain the knowledge fragments as the corresponding matching multimedia information, etc. The number of the local databases can be one or more. If the number of the local databases is one, the user equipment directly matches the query field information with the local database to determine the corresponding matching multimedia information. In some embodiments, the number of the local databases is multiple; wherein in the step S1022, the corresponding matching multimedia information is determined according to the one or more query field information in the multiple local databases. For example, if the number of the local databases is multiple, selective matching can be performed, such as first selecting the databases, then matching based on the selected databases, or first matching all the databases, and then selecting the matching multimedia information based on the matching multimedia information, so as to determine a proper number of matching multimedia information (for example, a preset number of matching multimedia information is determined according to the similarity / vector distance ranking, a preset number of matching multimedia information with higher similarity / near vector distance is taken as the final matching multimedia information, etc.), and too many matching multimedia information will cause resource waste, and too many matching multimedia information will reduce the accuracy of the multimedia enhancement process, such as not fitting the context of the corresponding scene, etc.

[0058] In some embodiments, in step S1022, a target local database for query matching is determined from the plurality of local databases; and corresponding matching multimedia information is determined according to the query field information in the target local database. For example, the number of local databases of the user equipment is a plurality, and the database types of the plurality of local databases differ based on user settings. In order to obtain better text enhancement effect, matching text information that is more suitable for the scene needs to be provided. The user equipment determines a target local database for query matching from the plurality of databases according to the query field information, for example, the user equipment presents the query field information, so that the user selects a corresponding target local database from the plurality of local databases based on the query field information. In some embodiments, the target local database for query matching is determined from the plurality of local databases based on the selection operation of the user. For example, each local database corresponds to a data source, and the data results obtained by matching different data sources are different, and the corresponding enhanced text information is also different. By selecting a local database similar to the scene for matching, the accuracy and adaptability of the final obtained text can be significantly improved. The number of target databases can be one or a plurality, which is not limited here.

[0059] In some embodiments, each of the plurality of local databases includes a type description field for describing the database type; and the target local database for query matching is determined from the plurality of local databases according to the one or more query field information and the type description field of the plurality of local databases. For example, in order to realize intelligent matching, each local database includes a type description field for describing the database type. The user equipment performs similarity matching between the text field of the query text information and the corresponding type description field, so as to determine the similarity between the query text information and each local database. N (N is a positive integer) local databases with the highest similarity ranking are taken as corresponding target local databases, so as to improve the accuracy of the matching text information.

[0060] In some embodiments, the number of the matched text information is multiple; wherein the method further comprises step S105 (not shown) of determining a plurality of target matched text information for text enhancement from the multiple matched text information; wherein in step S103, the corresponding text generation request is sent to the corresponding network device, wherein the text generation request comprises the query text information and the multiple target matched text information. For example, the number of matched text information is not the more the better, and an appropriate and accurate amount of matched text information helps to improve the accuracy of the final enhanced text information. After the user device obtains multiple matched text information, it can determine an appropriate amount (e.g., a preset number threshold or a preset proportion, etc.) of matched text information as target matched text information from the multiple matched text information, and send the multiple target matched text information and the query text information to the network device for outputting the corresponding enhanced text information. The selection method of the target matched text information includes but is not limited to random extraction, user manual selection, similarity-based sorting selection, etc.

[0061] Figure 3 A method for obtaining enhanced multimedia information according to an aspect of the present application is shown, which is applied to a network device 200, wherein the method comprises steps S201, S202 and S203. In step S201, a multimedia generation request corresponding to a user device is received, wherein the multimedia generation request comprises query multimedia information and one or more matched multimedia information corresponding to the query multimedia information, and the one or more matched multimedia information is determined by the user device in the local database according to the query multimedia information; in step S202, the query multimedia information and the one or more matched multimedia information are input into a corresponding matching model, and the output multimedia content of the matching model is determined as the corresponding enhanced multimedia information; in step S203, the enhanced multimedia information is returned to the user device.

[0062] For example, a user holds a user device (e.g., a node in a local area network or a mobile phone, etc.), and the user device can obtain a multimedia query request input by the user, such as the user device including an input device for collecting input operations of the user, such as a keyboard, a mouse, or a touch screen, etc. The user can click the corresponding page through the input device, and input the multimedia information expected to be queried in the multimedia query page, thereby generating the corresponding multimedia query request. Here, the multimedia query request can be generated by the user operating the user device, or can be generated by the user device receiving input operations of the user on other user devices, or the user device directly receiving a multimedia query request generated by other user devices based on input operations of the user, etc. In some cases, the multimedia query request can be issued by a network device, the multimedia query request is generated by other user devices based on corresponding user operations and sent to the network device, further, the network device stores a type description field of a database type of a local database of the user device (e.g., the device identification information of the user device and the type description field have a mapping relationship), and determines that the type description field of the local database of the user device matches the multimedia field of the query multimedia information based on the matching of the query multimedia information in the multimedia query request, and then the network device issues the multimedia query request to the user device, etc. The user device or the local area network stores a corresponding local database, and the local database is used to store related text segments, etc. In some cases, the database type of the local database can be different types of multimedia libraries based on different settings, such as programming, movies, or entertainment, etc. The local database includes a multimedia database constructed according to commonly used multimedia and / or private multimedia of the user / current local area network. The number of local databases can be one or more. After the user device obtains the corresponding multimedia query request, the local database is called and the fields contained in the query multimedia information are matched with the text segments in the local database, one or more matching multimedia information related to the query multimedia information can be determined, such as through semantic matching, or based on vector similarity matching of fields and text segments, etc. The matching multimedia information is used to indicate the knowledge segments related to the query multimedia information in the database, such as a similarity greater than or equal to a similarity threshold, or a cosine distance of fields and knowledge segments in a vector space less than or equal to a preset distance threshold, etc. After the user device obtains one or more matching multimedia information, a corresponding multimedia generation request is generated according to the corresponding query multimedia information and the one or more matching multimedia information, and the multimedia generation request is sent to the network device, and the final output multimedia content is obtained by using cloud resources, etc. Specifically, the user device sends the corresponding multimedia generation request to the network device, and the network device side establishes a corresponding matching model, inputs the corresponding query multimedia information and the one or more matching multimedia information into the matching model, and obtains the output multimedia content output by the matching model, etc.The matching model refers to a deep learning model trained using a large amount of multimedia data, which can generate enhanced multimedia content or understand the meaning of multimedia content. The matching model can process various multimedia content tasks such as multimedia classification, question answering, dialogue, etc. The network device inputs the corresponding query multimedia information and the one or more matching multimedia information into the matching model, and obtains the output multimedia content output by the matching model, so as to take the output multimedia content as the final enhanced multimedia information of this multimedia query, and returns the enhanced multimedia information to the user device. If the multimedia query request is generated by other user devices based on the input operation of other users, the network device will also return the enhanced multimedia information to the other user devices. In some cases, the user device can also update the corresponding local database based on the enhanced multimedia information and the corresponding multimedia query request.

[0063] In some embodiments, the method further comprises a step S204 (not shown), in which one or more cloud matching multimedia information corresponding to the query multimedia information is determined in the cloud matching. In step S202, the query multimedia information, the one or more matching multimedia information and the one or more cloud matching multimedia information are input into the corresponding matching model, and the output multimedia content of the matching model is determined as the corresponding enhanced multimedia information. For example, in addition to matching in the local database with strong privacy, the network device can also be used for matching of multimedia information, such as matching of public data by setting a public database, which is a collection database of all public multimedia captured on the network, or searching or crawling software on the network or other information search (such as search / crawler crawling data in web pages) in real time by the network device and determining the corresponding matching multimedia information according to the similarity of the multimedia information. The network device determines the corresponding cloud matching multimedia information based on the query multimedia information in the public database matching query, which is used to indicate the matching multimedia related to the query multimedia information in the public database, and the matching method is the same as or similar to the matching method of the local database. The network device inputs the query multimedia information, the one or more matching multimedia information matched by the local database and the one or more cloud matching multimedia information matched by the cloud public database into the matching model, so as to determine the output multimedia content of the matching model as the enhanced multimedia information of this query.

[0064] In some embodiments, the method further comprises a step S205 (not shown) of determining one or more final matching multimedia information from the one or more matching multimedia information and the one or more cloud matching multimedia information; wherein in the step S202, the query multimedia information and the one or more final matching multimedia information are input into a corresponding matching model, and output multimedia content of the matching model is determined as the corresponding enhanced multimedia information. For example, appropriate and accurate matching multimedia information helps to improve the accuracy of the final enhanced multimedia information, etc. After the network device obtains the one or more matching multimedia information and the one or more cloud matching multimedia information, if the number of corresponding multimedia is greater than or equal to a preset number threshold, appropriate (e.g., a preset number threshold or a preset proportion, etc.) matching multimedia information can be determined from the plurality of multimedia information as the final matching multimedia information, and the plurality of final matching multimedia information and the query multimedia information are input into the matching model, so that the output multimedia content of the matching model is determined as the enhanced multimedia information of this query. The selection method of the target matching multimedia information includes but is not limited to random extraction, user manual selection, selection based on similarity sorting, etc.

[0065] In the above, the embodiments of the method for obtaining enhanced multimedia information are mainly described. In addition, the application also provides specific devices capable of implementing the above embodiments, which will be described below in combination with Figure 4 、 Figure 5 .

[0066] Figure 4 A user device 100 for obtaining enhanced multimedia information is shown, which comprises a first module 101, a second module 102, a third module 103, and a fourth module 104. The first module 101 is configured to obtain a multimedia query request input by a user, wherein the multimedia query request comprises corresponding query multimedia information. The second module 102 is configured to, in response to the multimedia query request, determine one or more matching multimedia information in a local database according to the query multimedia information. The third module 103 is configured to send a multimedia generation request to a corresponding network device, wherein the multimedia generation request comprises the query multimedia information and the one or more matching multimedia information. The fourth module 104 is configured to receive enhanced multimedia information returned by the network device based on the multimedia generation request, wherein the enhanced multimedia information comprises output multimedia content determined by inputting the query multimedia information and the one or more matching multimedia information into a corresponding matching model.

[0067] In some embodiments, the one-to-two module 102 comprises a one-to-one unit 1021 (not shown) and a two-to-two unit 1022 (not shown); the one-to-one unit 1021 is configured to extract one or more query field information from the query multimedia information in response to the multimedia query request; the two-to-two unit 1022 is configured to query the local database according to the one or more query field information to determine corresponding matching multimedia information.

[0068] In some embodiments, the two-to-two unit 1022 is configured to determine a target local database for query matching from the plurality of local databases; and query the target local database according to the one or more query field information to determine corresponding matching multimedia information.

[0069] In some embodiments, each of the plurality of local databases comprises a type description field for describing the type of the database; wherein the determination of the target local database for query matching from the plurality of local databases comprises: matching the type description field of the plurality of local databases according to the one or more query field information to determine the target local database for query matching from the plurality of local databases.

[0070] Herein, the Figure 4 The specific embodiments of the one-to-one module 101, the one-to-two module 102, the one-to-three module 103 and the one-to-four module 104 are the same as or similar to the aforementioned embodiments of the steps S101, S102, S103 and S104, and thus are not described herein again by way of reference. Figure 2 The embodiments of the steps S101, S102, S103 and S104 are the same as or similar to the aforementioned embodiments of the steps S101, S102, S103 and S104, and thus are not described herein again by way of reference.

[0071] In some embodiments, the number of the matching multimedia information is a plurality; wherein the user equipment 100 further comprises a five-to-one module (not shown) configured to determine a plurality of target matching multimedia information for multimedia enhancement from the plurality of matching multimedia information; wherein the one-to-three module 103 is configured to send a corresponding multimedia generation request to a corresponding network device, wherein the multimedia generation request comprises the query multimedia information and the plurality of target matching multimedia information.

[0072] Herein, the specific embodiments of the five-to-one module are the same as or similar to the aforementioned embodiments of the step S105, and thus are not described herein again by way of reference.

[0073] Figure 5Fig. 2 shows a network device 200 for obtaining enhanced multimedia information according to an aspect of the present application, wherein the network device 200 comprises a first module 201, a second module 202 and a third module 203. The first module 201 is configured to receive a multimedia generation request sent by a user device, wherein the multimedia generation request comprises a query multimedia information and one or more matching multimedia information determined by the user device in a local database according to the query multimedia information. The second module 202 is configured to input the query multimedia information and the one or more matching multimedia information into a matching model, and determine an output multimedia content of the matching model as corresponding enhanced multimedia information. The third module 203 is configured to return the enhanced multimedia information to the user device.

[0074] In this embodiment, the first module 201, the second module 202 and the third module 203 are the same as or similar to the first module 101, the second module 102 and the third module 103 of the above-mentioned embodiment, and thus will not be described here in detail. Figure 5 The specific implementation of the first module 201, the second module 202 and the third module 203 is the same as or similar to the above-mentioned embodiment of the first module 101, the second module 102 and the third module 103, and thus will not be described here in detail. Figure 3 The specific implementation of the step S201, the step S202 and the step S203 is the same as or similar to the above-mentioned embodiment of the step S101, the step S102 and the step S103, and thus will not be described here in detail.

[0075] In some embodiments, the network device 200 further comprises a fourth module (not shown) configured to determine one or more cloud matching multimedia information in a cloud database according to the query multimedia information; wherein the second module 202 is configured to input the query multimedia information, the one or more matching multimedia information and the one or more cloud matching multimedia information into a matching model, and determine an output multimedia content of the matching model as corresponding enhanced multimedia information.

[0076] In some embodiments, the network device 200 further comprises a fifth module (not shown) configured to determine one or more final matching multimedia information from the one or more matching multimedia information and the one or more cloud matching multimedia information; wherein the second module 202 is configured to input the query multimedia information and the one or more final matching multimedia information into a matching model, and determine an output multimedia content of the matching model as corresponding enhanced multimedia information.

[0077] In this embodiment, the fourth module and the fifth module are the same as or similar to the above-mentioned embodiment of the step S204 and the step S205, and thus will not be described here in detail.

[0078] In addition to the methods and apparatuses described in the above embodiments, the present application also provides a computer readable storage medium storing computer code which, when executed, causes the method of any preceding item to be performed.

[0079] The present application also provides a computer program product which, when executed by a computer device, causes the method of any preceding item to be performed.

[0080] The present application also provides a computer device comprising:

[0081] one or more processors;

[0082] a memory for storing one or more computer programs;

[0083] when the one or more computer programs are executed by the one or more processors, cause the one or more processors to implement the method of any preceding item.

[0084] Figure 6 An example system that can be used to implement various embodiments described in the present application is shown;

[0085] As Figure 6 shown, in some embodiments, system 300 can function as any of the above-described apparatuses in various embodiments. In some embodiments, system 300 can include one or more computer readable media (e.g., system memory or non-volatile memory (NVM) / storage device 320) having instructions and one or more processors (e.g., processor(s) 305) coupled with the one or more computer readable media and configured to execute the instructions to implement modules to perform the actions described in the present application.

[0086] For one embodiment, system control module 310 can include any suitable interface controllers to provide for any suitable interface to at least one of the processor(s) 305 and / or any suitable device or component in communication with system control module 310.

[0087] System control module 310 can include a memory controller module 330 to provide an interface to system memory 315. Memory controller module 330 can be a hardware module, a software module, and / or a firmware module.

[0088] System memory 315 can be used, for example, to load and store data and / or instructions for system 300. For one embodiment, system memory 315 can include any suitable volatile memory, such as suitable DRAM. In some embodiments, system memory 315 can include Double Data Rate type four Synchronous Dynamic Random Access Memory (DDR4 SDRAM).

[0089] For one embodiment, system control module 310 can include one or more input / output (I / O) controllers to provide an interface to NVM / storage 320 and communication interface(s) 325.

[0090] For example, NVM / storage 320 can be used to store data and / or instructions. NVM / storage 320 can include any suitable non-volatile memory (e.g., flash memory) and / or can include any suitable non-volatile storage device(s) (e.g., one or more Hard-Disk Drives (HDDs), one or more Compact Disc (CD) drives, and / or one or more Digital Video Disc (DVD) drives).

[0091] NVM / storage 320 can include a storage resource that is physically part of the device on which system 300 is installed or that is accessed via the device but not necessarily physically on the device. For example, NVM / storage 320 can be accessed via communication interface(s) 325 over a network.

[0092] Communication interface(s) 325 can provide an interface for system 300 to communicate over one or more networks and / or with any other suitable device. System 300 can wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and / or protocols.

[0093] For one embodiment, at least one of the processor(s) 305 can be packaged together with logic for one or more controllers of the system control module 310 (e.g., a memory controller module 330). For one embodiment, at least one of the processor(s) 305 can be packaged together with logic for one or more controllers of the system control module 310 to form a System In a Package (SiP). For one embodiment, at least one of the processor(s) 305 can be integrated on the same die with logic for one or more controllers of the system control module 310. For one embodiment, at least one of the processor(s) 305 can be integrated on the same die with logic for one or more controllers of the system control module 310 to form a System On Chip (SoC).

[0094] In various embodiments, the system 300 can be, but is not limited to, a server, a workstation, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet, a netbook, etc.). In various embodiments, the system 300 can have more or less components, and / or different architectures. For example, in some embodiments, the system 300 includes one or more cameras, a keyboard, a Liquid Crystal Display (LCD) screen (including touch screen displays), non- volatile memory port, multiple antennas, a graphics chip, an Application Specific Integrated Circuit (ASIC), and speakers.

[0095] It is noted that the present application can be implemented in software and / or in a combination of software and hardware, e.g., application specific integrated circuit (ASIC), general purpose computer or any other similar hardware devices. In one embodiment, the software program of the present application is implemented by the processor to perform predetermined functions or tasks. Also, the software program of the present application (including related data structures) can be stored in a computer readable recording medium, e.g., RAM memory, magnetic or optical drive or diskette, and so on. Further, some steps or functions of the present application can be implemented in hardware, e.g., as an electronic circuit that cooperates with the processor to perform various steps or functions.

[0096] In addition, part of the present application can be applied as a computer program product, for example, computer program instructions, when executed by a computer, through the operation of the computer, the method and / or technical solutions according to the present application can be called or provided. Those skilled in the art should understand that the form of computer program instructions in computer readable medium includes but is not limited to source file, executable file, installation package file and the like, and accordingly, the way of computer program instructions executed by computer includes but is not limited to: the computer directly executes the instructions, or the computer compiles the instructions and then executes the corresponding compiled program, or the computer reads and executes the instructions, or the computer reads and installs the instructions and then executes the corresponding installed program. Here, the computer readable medium can be any available computer readable storage medium or communication medium accessible to the computer.

[0097] Communication media includes any medium through which computer readable instructions, data structures, program modules, or other data is communicated from one system to another, e.g., according to a communication protocol. Depending on the nature of the communication medium, the communication can be wireless or wired. The communication medium can include a wired transmission medium, such as electrical and optical circuits, cables, and lines (e.g., fiber optic, coaxial, etc.), and wireless transmission media, such as acoustic, electromagnetic, radio frequency (RF), microwave, and infrared. Computer readable instructions, data structures, program modules, or other data can be embodied as modulated data signals, e.g., carrier waves, such as those implemented as part of an extended spectrum technique. The term "modulated data signal" refers to a signal that has one or more of its characteristics set or changed in such a manner as to encode information in the signal. The modulated signals can be analog, digital or mixed.

[0098] By way of example, and not limitation, computer readable storage media can include volatile and non-volatile, removable and non-removable media implemented in any method or technology for storage of information such as computer readable instructions, data structures, program modules or other data. For example, computer readable storage media includes, but is not limited to, RAM, such as SRAM, DRAM, or other types of RAM; and non-volatile memory, such as flash, various types of ROM, PROM, EPROM, EEPROM, MRAM, FeRAM; and magnetic and optical storage devices, such as hard disks, magnetic tapes, CDs, DVDs; or other now known or later developed media that can store computer readable information / data for use by a computer system.

[0099] Herein, according to one embodiment of the present application comprises a device, the device comprises a memory for storing computer program instructions and a processor for executing program instructions, wherein when the computer program instructions are executed by the processor, trigger the device run based on the foregoing according to the method and / or technical solutions of multiple embodiments of the present application.

[0100] It is obvious to a person skilled in the art that the present application is not limited to the details of the above exemplary embodiments, but can be implemented in other concrete forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be considered in all aspects as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and therefore all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any reference signs in the claims should not be considered as limiting the claims involved. Furthermore, it is obvious that the word "comprising" does not exclude other units or steps, and the singular does not exclude the plural. The plurality of units or devices stated in the device claims can also be implemented by one unit or device through software or hardware. The words first, second, etc. are used to indicate names, not any specific order.

Claims

1. A method for acquiring enhanced multimedia information, applied to a user equipment, wherein, The method includes: Obtain a multimedia query request input by the user, wherein the multimedia query request includes corresponding query multimedia information; In response to the multimedia query request, one or more matching multimedia information are determined by querying the local database based on the queried multimedia information; Send a corresponding multimedia generation request to the corresponding network device, wherein the multimedia generation request includes the query multimedia information and one or more matching multimedia information; The network device receives enhanced multimedia information returned based on the multimedia generation request, wherein the enhanced multimedia information includes output multimedia content determined by inputting the queried multimedia information and one or more matching multimedia information into the corresponding matching model.

2. The method according to claim 1, wherein, In response to the multimedia query request, the step of querying and determining one or more matching multimedia information in the local database based on the queried multimedia information includes: In response to the multimedia query request, extract one or more query field information from the queried multimedia information; Based on the information from one or more query fields, the corresponding matching multimedia information is determined by querying and matching in the local database.

3. The method according to claim 2, wherein, The number of local databases is multiple; wherein, the step of querying and matching the corresponding multimedia information in the local databases based on the one or more query field information includes: Based on the information from one or more query fields, the corresponding matching multimedia information is determined by querying and matching in multiple local databases.

4. The method according to claim 3, wherein, The step of querying and determining the corresponding matching multimedia information in the local database based on the one or more query field information includes: From the plurality of local databases, determine the target local database for query matching; Based on the information from one or more query fields, the corresponding matching multimedia information is determined by querying and matching in the target local database.

5. The method according to claim 4, wherein, The step of determining the target local database for query matching from the plurality of local databases includes: Based on the user's selection operation, a target local database for query matching is determined from the plurality of local databases.

6. The method according to claim 4, wherein, Each of the plurality of local databases includes a type description field for describing the database type; wherein, determining the target local database for query matching from the plurality of local databases includes: Based on the matching of one or more query field information with the type description fields of the plurality of local databases, a target local database for query matching is determined from the plurality of local databases.

7. The method according to claim 1, wherein, The number of matched multimedia information is multiple; wherein, the method further includes: From the plurality of matched multimedia information, determine a plurality of target matched multimedia information for multimedia enhancement; The step of sending the corresponding multimedia generation request to the corresponding network device includes: Send a corresponding multimedia generation request to the corresponding network device, wherein the multimedia generation request includes the query multimedia information and the multiple target matching multimedia information.

8. A method for acquiring enhanced multimedia information, applied to a network device, wherein, The method includes: The system receives a multimedia generation request sent by a user equipment, wherein the multimedia generation request includes querying multimedia information and one or more corresponding matching multimedia information, and the one or more matching multimedia information is determined by the user equipment in a local database based on the querying multimedia information. The query multimedia information and one or more matching multimedia information are input into the corresponding matching model, and the output multimedia content of the matching model is determined as the corresponding enhanced multimedia information. The enhanced multimedia information is returned to the user device.

9. The method according to claim 8, wherein, The method further includes: Based on the queried multimedia information, one or more corresponding cloud-matched multimedia information are determined in the cloud; The step of inputting the queried multimedia information and the one or more matching multimedia information into the corresponding matching model, and determining the output multimedia content of the matching model as the corresponding enhanced multimedia information, includes: The query multimedia information, the one or more matching multimedia information, and the one or more cloud-matched multimedia information are input into the corresponding matching model, and the output multimedia content of the matching model is determined as the corresponding enhanced multimedia information.

10. The method according to claim 9, wherein, The method further includes: Determine one or more final matching multimedia information from the one or more matching multimedia information and the one or more cloud-based matching multimedia information; The step of inputting the queried multimedia information and the one or more matching multimedia information into the corresponding matching model, and determining the output multimedia content of the matching model as the corresponding enhanced multimedia information, includes: The queried multimedia information and the one or more final matched multimedia information are input into the corresponding matching model, and the output multimedia content of the matching model is determined as the corresponding enhanced multimedia information.

11. A method for acquiring enhanced multimedia information, wherein, The method includes: The user equipment acquires a multimedia query request input by the user, wherein the multimedia query request includes corresponding query multimedia information; in response to the multimedia query request, it queries and determines one or more matching multimedia information in the local database based on the query multimedia information; and sends a corresponding multimedia generation request to the corresponding network device, wherein the multimedia generation request includes the query multimedia information and the one or more matching multimedia information. The network device receives a corresponding multimedia generation request sent by the user equipment, inputs the queried multimedia information and one or more matching multimedia information into the corresponding matching model, determines the output multimedia content of the matching model as the corresponding enhanced multimedia information, and returns the enhanced multimedia information to the user equipment. The user equipment receives enhanced multimedia information returned by the network device based on the multimedia generation request, wherein the enhanced multimedia information includes output multimedia content determined by inputting the queried multimedia information and one or more matching multimedia information into the corresponding matching model.

12. A computer device, wherein, The device includes: Processor; and A memory configured to store computer-executable instructions, which, when executed, cause the processor to perform the steps of the method as described in any one of claims 1 to 10.

13. A computer-readable storage medium having a computer program / instructions stored thereon, characterized in that, When executed, the computer program / instruction causes the system to perform the steps of the method as described in any one of claims 1 to 10.

14. A computer program product comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method described in any one of claims 1 to 10.

Images