Large model-based search method and system, device and medium

Abstract

The present disclosure provides a large model-based search method and system, a device and a medium, which are applied to the technical fields of artificial intelligence and intelligent cameras. The method comprises: storing a plurality of information pairs into a database, each information pair comprising an acquired image and descriptive text information used for describing the acquired image, and the descriptive text information being generated by using a pre-trained visual-semantic large model to process the acquired image after an intelligent camera acquires the acquired image; in response to receiving a search request, analyzing search information from the search request; and determining from the database a search result matched with the search information, the search result at least comprising one information pair. The large model-based search method provided by the present invention can improve search efficiency, thus improving user experience.

Description

Search method, system, device and medium based on large model Technical Field

[0001] The present disclosure relates to the fields of artificial intelligence and smart camera technology, and in particular to a large model-based search method, system, device, and medium. Background Art

[0002] In related technologies, where monitoring is permitted and in compliance with relevant regulations, users are typically required to manually access footage recorded by cameras and view specific footage directly through a monitoring program. This viewing process is limited to a single method: playback from beginning to end or by time, resulting in low query efficiency and a negative user experience. Furthermore, existing search technologies utilize specific interfaces to access data from third-party providers, potentially raising concerns about data privacy, data dependency, and investment costs.

[0003] Summary of the Invention

[0004] In view of the above problems, the present disclosure provides a large model-based search method, system, device, medium and program product.

[0005] According to a first aspect of the present disclosure, a large model-based search method is provided, comprising: storing a plurality of information pairs in a database, wherein the information pairs comprise: a captured image and descriptive text information for describing the captured image; the descriptive text information is generated after the captured image is captured by an intelligent camera and processed by a pre-trained visual semantic large model; in response to receiving a search request, parsing search information from the search request; and determining search results that match the search information from the database, wherein the search results comprise at least one information pair.

[0006] According to an embodiment of the present disclosure, the search information includes search content information and search identification information; wherein, determining search results that match the search information from a database includes: determining multiple associated information pairs associated with the search identification information from multiple information pairs; and determining search results based on the search content information and the multiple associated information pairs.

[0007] According to an embodiment of the present disclosure, the associated information pair includes: an associated captured image and associated descriptive text information for describing the associated captured image; wherein, based on the search content information and multiple associated information pairs, the search results are determined, including: converting the search content information, the associated captured image and the associated descriptive text information into vector information of the search content, vector information of the associated captured image and vector information of the associated descriptive text, respectively; and determining the search results based on the vector information of the search content, the vector information of the associated captured image and the vector information of the associated descriptive text.

[0008] According to an embodiment of the present disclosure, the search results include a target captured image and target descriptive text information; wherein, based on the vector information of the search content, the vector information of the associated captured image, and the vector information of the associated descriptive text, the search results are determined, including: in the case where the search content information is determined to be search text information, determining the similarity value between the vector information corresponding to the search text information and the vector information of the associated descriptive text; determining the associated descriptive text information corresponding to the vector information of the associated descriptive text whose similarity value exceeds a threshold as the target descriptive text information; and determining the target captured image from a plurality of associated information pairs based on the target descriptive text information; or

[0009] When it is determined that the search content information is search image information, the similarity value between the vector information corresponding to the search image information and the vector information of the associated acquired image is determined; the associated acquired image corresponding to the vector information of the associated acquired image whose similarity value exceeds a threshold is determined as the target acquired image; and based on the target acquired image, the target descriptive text information is determined from multiple associated information pairs.

[0010] According to an embodiment of the present disclosure, the descriptive text information meets a preset character limit condition, and the search information includes the search text information; the search method based on the large model also includes: extracting character feature information of all descriptive text information in the database; generating character number prompt information based on the character feature information, so that the user can determine the number of characters of the search text information based on the character number prompt information.

[0011] According to an embodiment of the present disclosure, the descriptive text information satisfies a preset description form restriction condition, and the search information includes the search text information; the search method based on the large model also includes: extracting the descriptive feature information of all descriptive text information in the database; generating descriptive prompt information based on the descriptive feature information, so that the user can determine the search text information based on the descriptive prompt information.

[0012] According to an embodiment of the present disclosure, the search identification information includes user identification information; the search method based on the big model also includes: extracting user attribute features from the user identification information; determining the display form of the search results based on the user attribute features, wherein the display form includes at least one of the following: image form, video form, and voice broadcast form.

[0013] The second aspect of the present disclosure provides a large model-based search system, including: a storage module for storing multiple information pairs in a database, wherein the information pair includes: a captured image and descriptive text information for describing the captured image; the descriptive text information is generated after the captured image is captured by the intelligent camera and processed by the pre-trained visual semantic large model; a parsing module for parsing search information from the search request in response to receiving a search request; a determination module for determining search results that match the search information from the database, wherein the search results include at least one information pair.

[0014] The third aspect of the present disclosure provides an electronic device comprising: one or more processors; and a memory for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors execute the above-mentioned large model-based search method.

[0015] The fourth aspect of the present disclosure further provides a computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, causes the processor to execute the above-mentioned large model-based search method.

[0016] The fifth aspect of the present disclosure further provides a computer program product, comprising a computer program, which implements the above-mentioned large model-based search method when executed by a processor.

[0017] According to an embodiment of the present disclosure, a smart camera captures an image and uses a pre-trained visual semantic large model to process the captured image to generate descriptive text information for describing the captured image. These data are pre-stored in a database. When a user needs to search, the search results are matched from the database, which can improve search efficiency. Since the visual semantic large model can accurately obtain descriptive text information for describing the captured image by leveraging the large model's strong feature capture performance, generalization ability, image detail processing ability, and more robustness to input noise or changes, it is beneficial for users to quickly know the events occurring in a specific screen and view a specific screen based on the descriptive text information, thereby improving the user experience. In addition, in the large model-based search method provided by the present disclosure, during the search process, the search data is collected by the smart camera during the detection process, without the need for data from a third-party provider, at least partially solving the problems of existing search technologies that use specific interfaces to call data from third-party providers, involving data dependency issues and investment costs. BRIEF DESCRIPTION OF THE DRAWINGS

[0018] The above contents and other objects, features and advantages of the present disclosure will become more apparent through the following description of the embodiments of the present disclosure with reference to the accompanying drawings, in which:

[0019] FIG1 shows an application scenario diagram of a large model-based search method, system, device, medium, and program product according to an embodiment of the present disclosure;

[0020] FIG2 shows a flow chart of a large model-based search method according to an embodiment of the present disclosure;

[0021] FIG3 shows a flowchart of a large model-based search method according to another embodiment of the present disclosure;

[0022] FIG4 shows a structural block diagram of a large model-based search system according to an embodiment of the present disclosure; and

[0023] FIG5 shows a block diagram of an electronic device suitable for implementing a large model-based search method according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. However, it should be understood that these descriptions are merely exemplary and are not intended to limit the scope of the present disclosure. In the detailed description below, for ease of explanation, many specific details are set forth to provide a comprehensive understanding of the embodiments of the present disclosure. However, it is apparent that one or more embodiments may also be implemented without these specific details. In addition, in the following description, descriptions of well-known structures and technologies are omitted to avoid unnecessary confusion of the concepts of the present disclosure.

[0025] The terms used herein are only for describing specific embodiments and are not intended to limit the present disclosure. The terms "comprise," "include," etc. used herein indicate the presence of features, steps, operations, and / or components, but do not exclude the presence or addition of one or more other features, steps, operations, or components.

[0026] All terms used herein (including technical and scientific terms) have the meanings commonly understood by those skilled in the art unless otherwise defined. It should be noted that the terms used herein should be interpreted as having a meaning consistent with the context of this specification and should not be interpreted in an idealized or overly rigid manner.

[0027] When expressions such as "at least one of A, B and C, etc." are used, they should generally be interpreted in accordance with the meaning of the expression commonly understood by those skilled in the art (for example, "a system having at least one of A, B and C" should include but is not limited to a system having A alone, B alone, C alone, A and B, A and C, B and C, and / or A, B, C, etc.).

[0028] In the technical solution disclosed herein, the user information involved (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0029] In the technical solution of the embodiment of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0030] In related technologies, when monitoring is permitted and in compliance with relevant regulations, users are allowed to view the images recorded by the camera and need to view specific images. However, they usually need to manually view the images directly through the monitoring program. During the viewing process, they can only play and view from the beginning to the end or by time in a single way, which has low query efficiency and affects the user experience. In addition, in existing search technologies, using specific interfaces to call data from third-party providers may involve data privacy issues, data dependency issues, and investment costs. For example, if the data provided by a third-party provider is completely dependent on the interface, if the third-party provider stops providing services, it may affect the user experience. If open source tools and libraries are used to build a self-built search system, such as the search server Elasticsearch, although the metadata and description of the image can be stored and then searched based on this data, it requires a lot of time, resources and technical investment, and even requires continuous updating and maintenance of the system. There are still problems with low user search efficiency and low accuracy.

[0031] Embodiments of the present disclosure provide a large-scale model-based search method, system, device, and medium. The method includes: storing multiple information pairs in a database, wherein the information pairs include: a captured image and descriptive text information describing the captured image; the descriptive text information is generated after the captured image is captured by an intelligent camera and processed using a pre-trained visual semantic large-scale model; in response to receiving a search request, parsing search information from the search request; and determining search results from the database that match the search information, wherein the search results include at least one information pair.

[0032] FIG1 shows an application scenario diagram of a large model-based search method, system, device, medium, and program product according to an embodiment of the present disclosure.

[0033] As shown in Figure 1, an application scenario 100 according to this embodiment may include a first terminal device 101, a second terminal device 102, a third terminal device 103, a network 104, and a server 105. The network 104 is a medium for providing a communication link between the first terminal device 101, the second terminal device 102, the third terminal device 103, and the server 105. The network 104 may include various connection types, such as wired or wireless communication links or fiber optic cables.

[0034] The user can use at least one of the first terminal device 101, the second terminal device 102, and the third terminal device 103 to interact with the cloud server 105 via the network 104 to receive or send messages, etc. The first terminal device 101, the second terminal device 102, and the third terminal device 103 can be installed with a smart camera and a client application associated with the smart camera.

[0035] The first terminal device 101 , the second terminal device 102 , and the third terminal device 103 may be various electronic devices having display screens and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, desktop computers, and the like.

[0036] The server 105 may be a server that provides various services, such as a background management server (for example only) that supports users in detecting movement of target objects detected by smart cameras on the first terminal device 101, the second terminal device 102, and the third terminal device 103. The background management server may analyze and process received captured images, and feed back the processing results (e.g., generating descriptive text information describing the scene image based on the captured images) to the terminal devices.

[0037] It should be understood that the number of terminal devices, networks, and servers in Figure 1 is merely illustrative and any number of terminal devices, networks, and servers may be provided as required.

[0038] The following will describe in detail the large model-based search method of the disclosed embodiment based on the scenario described in FIG1 through FIG2 and FIG3 .

[0039] FIG2 shows a flowchart of a large model-based search method according to an embodiment of the present disclosure.

[0040] As shown in FIG. 2 , the large model-based search method 200 of this embodiment includes operations S210 to S230 .

[0041] In operation S210, multiple information pairs are stored in a database, wherein the information pairs include: a captured image and descriptive text information for describing the captured image; the descriptive text information is generated after the captured image is captured by the smart camera and processed using a pre-trained visual semantic model.

[0042] It should be noted that the information (including but not limited to user personal information, user image information, user device information, such as location information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in the detection and image acquisition of smart cameras are all information and data authorized by the user or fully authorized by all parties, and the collection, storage, use, processing, transmission, provision, disclosure and application of relevant data comply with the relevant laws, regulations and standards of relevant countries and regions, take necessary confidentiality measures, do not violate public order and good morals, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0043] According to an embodiment of the present disclosure, a smart camera can capture images using its image acquisition function and upload them to a cloud server pre-deployed with a pre-trained visual semantics model. The pre-trained visual semantics model then processes the captured images and outputs descriptive text describing the captured images. The captured images and descriptive text are stored as pairs in a database. The pairs can also be labeled using the device identifier of the smart camera associated with the pairs before being stored in the database.

[0044] The large visual semantic model is pre-trained based on the attributes of sample objects and the relationship attributes between different sample objects. The initial visual semantic model is constructed through the joint efforts of computer vision and natural language processing models. The sample objects can be mobile, such as vehicles, pedestrians, and animals.

[0045] In operation S220, in response to receiving the search request, search information is parsed from the search request.

[0046] According to an embodiment of the present disclosure, a user may input search information into a search interface, a search engine may initiate a search request based on the search information, and a search system may parse the search information after receiving the search request.

[0047] In operation S230, a search result matching the search information is determined from the database, wherein the search result includes at least one information pair.

[0048] According to embodiments of the present disclosure, search information may include search content information. Search results may be obtained by matching the search content information with the captured image and / or descriptive text describing the captured image in an information pair stored in a database. Search information may also include search content information and search identification information. The search identification information may be used to locate an associated information pair, and then, from within the information pair, search results may be matched based on the search content information.

[0049] According to an embodiment of the present disclosure, a smart camera captures an image and uses a pre-trained visual semantic large model to process the captured image to generate descriptive text information for describing the captured image. These data are pre-stored in a database. When a user needs to search, the search results are matched from the database, which can improve search efficiency. Since the visual semantic large model can accurately obtain descriptive text information for describing the captured image by leveraging the large model's strong feature capture performance, generalization ability, image detail processing ability, and more robustness to input noise or changes, it is beneficial for users to quickly know the events that occurred in a specific screen and view a specific screen based on the descriptive text information, thereby improving the user experience. In addition, in the search method based on the large model provided by the present disclosure, during the search process, the search data is collected by the smart camera during the detection process, without the need for data from a third-party provider, at least partially solving the problems of existing search technologies that use specific interfaces to call data from third-party providers, involving data dependency issues and investment costs.

[0050] According to an embodiment of the present disclosure, the search information may include search content information and search identification information.

[0051] Determining search results matching the search information from the database may include:

[0052] A plurality of associated information pairs associated with the search identification information are determined from the plurality of information pairs; and search results are determined based on the search content information and the plurality of associated information pairs.

[0053] According to an embodiment of the present disclosure, the search identification information may be user identification information. Associated device identification information related to the user's identification information may be determined based on the device identification information used to generate the information pair. Based on the associated device identification information, multiple associated information pairs may be determined from the multiple information pairs. The search content information may be matched with associated captured images and / or associated descriptive text information describing the associated captured images in the multiple associated information pairs, and the matching associated captured images and / or associated descriptive text information describing the associated captured images may be determined as search results.

[0054] According to the disclosed embodiments, by first determining associated information pairs through search identifier information and then determining search results from the associated information pairs, the search computational load can be reduced and search efficiency improved. Furthermore, searching within the associated information pairs associated with the search identifier avoids privacy and security issues involving other unrelated information pairs.

[0055] According to an embodiment of the present disclosure, the associated information pair may include: an associated captured image and associated descriptive text information for describing the associated captured image.

[0056] Among them, determining the search results based on the search content information and multiple related information pairs can include: converting the search content information, the related collected images and the related descriptive text information into vector information of the search content, vector information of the related collected images and vector information of the related descriptive text respectively; determining the search results based on the vector information of the search content, the vector information of the related collected images and the vector information of the related descriptive text.

[0057] According to an embodiment of the present disclosure, the search content information may be search text information or search image information. The search content information, the associated captured image, and the associated descriptive text information may be vectorized respectively.

[0058] According to the embodiment of the present disclosure, after vector processing, it is convenient to use similarity calculation between vectors to accurately determine the search results.

[0059] According to an embodiment of the present disclosure, the search results may include a target acquisition image and target descriptive text information.

[0060] Among them, determining the search results based on the vector information of the search content, the vector information of the associated captured image and the vector information of the associated descriptive text may include: when it is determined that the search content information is the search text information, determining the similarity value between the vector information corresponding to the search text information and the vector information of the associated descriptive text; determining the associated descriptive text information corresponding to the vector information of the associated descriptive text whose similarity value exceeds a threshold as the target descriptive text information; and determining the target captured image from multiple associated information pairs based on the target descriptive text information.

[0061] Among them, determining the search results based on the vector information of the search content, the vector information of the associated acquired image, and the vector information of the associated descriptive text may include: when it is determined that the search content information is the search image information, determining the similarity value between the vector information corresponding to the search image information and the vector information of the associated acquired image; determining the associated acquired image corresponding to the vector information of the associated acquired image whose similarity value exceeds a threshold as the target acquired image; and determining the target descriptive text information from multiple associated information pairs based on the target acquired image.

[0062] According to an embodiment of the present disclosure, the threshold value may be determined based on the actual required search accuracy. The similarity value may be determined based on a similarity calculation function. The present disclosure does not specifically limit the similarity calculation function.

[0063] According to the embodiments of the present disclosure, by calculating the similarity values ​​between vector information, search results can be quickly determined, search efficiency can be improved, and user experience can be enhanced.

[0064] According to an embodiment of the present disclosure, the descriptive text information meets a preset character limit condition, and the search information includes the search text information.

[0065] The search method based on the large model may also include:

[0066] Extracting character feature information of all descriptive text information in the database; generating character count prompt information based on the character feature information, so that the user can determine the number of characters in the search text information based on the character count prompt information.

[0067] According to an embodiment of the present disclosure, the preset character limit condition can be determined based on the actual description of the image and the user's reading of the text information. For example, the preset character limit condition can be that the number of characters in the description length of the text information used to describe the image does not exceed a preset number of characters. Character feature information is used to characterize the quantitative characteristics of the number of characters. The character feature information of all descriptive text information in the database can be counted, and then all the character feature information can be averaged to obtain the target character feature information. Based on the target character feature information, character number prompt information is generated. When the user searches, the search text information can be entered based on the character number prompt information.

[0068] During the implementation of the embodiments of the present disclosure, it was discovered that entering too few characters in search text during a search resulted in reduced search accuracy; entering too many characters in search text increased the complexity of the user's search, resulting in a poor user experience and even prolonged searches, reducing search efficiency. The present disclosure generates a character count prompt based on character feature information in descriptive text, facilitating users to quickly enter an appropriate number of characters in search text, thereby improving search accuracy, enhancing the user experience, and increasing search efficiency.

[0069] According to an embodiment of the present disclosure, the descriptive text information satisfies a preset description form restriction condition, and the search information includes the search text information.

[0070] The search method based on the large model may also include:

[0071] Extracting descriptive feature information of all descriptive text information in the database; generating description prompt information based on the descriptive feature information, so that the user can determine the description form of the search text information based on the description prompt information.

[0072] According to embodiments of the present disclosure, the preset description form restriction conditions can be determined based on the user's reading habits, language description characteristics, etc. The descriptive feature information can be used to characterize the characteristics of the text description form. The descriptive feature information of all descriptive text information in the database can be counted, and then the descriptive feature information with the most occurrences can be selected as the target descriptive feature information. Descriptive feature information is generated based on the descriptive feature information. When a user searches, they can enter search text information based on the description prompt information.

[0073] According to the embodiment of the present disclosure, description prompt information is generated by the descriptive feature information of the descriptive text information that meets the preset description form restriction conditions, which helps the language expression of the search text information input by the user to be clear and organized, and similar to the form of the descriptive text information in the database, which can improve the accuracy of the search while also improving the search efficiency.

[0074] According to an embodiment of the present disclosure, the search identification information may include user identification information.

[0075] The search method based on the large model may also include:

[0076] Extracting user attribute features from user identification information; determining a display format of search results based on the user attribute features, wherein the display format includes at least one of the following: image format, video format, and voice broadcast format.

[0077] According to an embodiment of the present disclosure, the user attribute characteristics may include the user's age characteristics. Different search result display formats may be set according to different age characteristics.

[0078] For example, the user's age characteristics may include teenagers, middle-aged, elderly, etc. For the elderly, it can be displayed in the form of voice broadcast.

[0079] According to the embodiment of the present disclosure, the display form of the search results is determined according to the user attribute characteristics, which increases the interest of the search and improves the user experience. Especially for the elderly, displaying the search results in the form of voice broadcast helps to quickly know the search results.

[0080] According to an embodiment of the present disclosure, when the search results include multiple information pairs, the results can be sorted using a relevance algorithm to determine the order in which the search results are displayed.

[0081] According to an embodiment of the present disclosure, after determining the search results that match the search information from the database, the search results can be functionalized so that when they are returned to the user, they can have interactive functions such as zooming, previewing, sharing or downloading options, so that the user can process the search results more conveniently.

[0082] According to the embodiment of the present disclosure, a preprocessing algorithm can be used to extract keywords from all descriptive text information in the database as search keywords, and prompt keywords can be generated based on the search keywords to enable users to determine the search text information based on the prompt keywords, avoid user input errors, facilitate users to use the search function, and ensure the quality of the search.

[0083] FIG3 shows a flowchart of a large model-based search method according to another embodiment of the present disclosure.

[0084] As shown in FIG3 , the large model-based search method 300 of this embodiment includes operations S310 to S370 .

[0085] In operation S310 , a captured image captured when the smart camera detects that a target object moves is acquired.

[0086] In operation S320 , the captured image is processed using a pre-trained visual semantic model to generate descriptive text information.

[0087] In operation S330 , the captured image and descriptive text information are vectorized and stored in a database.

[0088] In operation S340, in response to receiving the search request, search information is parsed from the search request.

[0089] In operation S350, the search information is vectorized.

[0090] In operation S360 , similarity calculation is performed between the vectorized search information and the vectorized captured image and descriptive text information stored in the database to obtain a similarity value.

[0091] In operation S370, search results matching the search information are determined from the database according to the similarity value.

[0092] According to the embodiments of the present disclosure, the smart camera can monitor the scene in real time through the built-in motion detection algorithm. Once the movement of the target object is detected, it will capture the image of the current scene, and promptly connect the captured image to the visual semantic model deployed on the cloud server to obtain descriptive text information for describing the captured image and store it in the database.

[0093] According to an embodiment of the present disclosure, a large visual semantic model is obtained by pre-training an initial visual semantic model. The training method may include: obtaining a sample image and sample descriptive text information describing the sample image; inputting the sample image into a computer vision model to output sample image features, wherein the sample image features are determined based on the attributes of the sample object and the relationship attributes between different sample objects; inputting the sample image features into a natural language processing model to output initial descriptive text information; and adjusting the parameters of the computer vision model and the natural language processing model based on the initial descriptive text information and the sample descriptive text information to obtain the large visual semantic model. The computer vision model may be composed of a multi-layer convolutional neural network, which may utilize multiple convolutional layers and pooling layers to identify sample image features in the sample image. The natural language processing model may be a recurrent neural network, a long short-term memory network, or a Transformer network. The sample image features may include: attribute features of the sample object in the sample image and attribute features of the relationship between different sample objects. The sample object may be any object in the sample image. For example, if the sample image is a photograph of a living room at a specific moment, the sample object may include a wall, a person, an animal, a lamp, a sofa, a coffee table, a television, etc. The attributes of the sample objects may include basic attributes of the sample objects and location attributes of the sample objects in the sample image. The relationship attributes between different sample objects may include location relationship attributes between different sample objects and association attributes between different sample objects. For example, if a person sits on a sofa, there is an association between the person and the sofa.

[0094] According to the embodiments of the present disclosure, the computer vision model determines the sample image features based on the attributes of the sample objects and the relationship attributes between different sample objects, and can more accurately extract the features used to describe the sample images, which is beneficial to the recognition of the sample objects, and is beneficial for the natural language processing model to accurately generate text information that can describe the sample images based on the input sample image features. The large visual semantic model obtained based on this training can accurately process the input image and output important descriptive text for the image.

[0095] According to an embodiment of the present disclosure, when the similarity value exceeds a threshold, the captured image and the descriptive text information are determined to be search results.

[0096] According to the disclosed embodiments, the visual semantic big model can capture the deep relationship between text descriptions and images, not just based on keywords or obvious tags. For example, if a user searches for "a leisurely afternoon," the visual semantic big model may return images of a person relaxing in a hammock or reading outdoors, even if these images do not have explicit "leisurely" or "afternoon" tags.

[0097] According to the embodiments of the present disclosure, traditional image search systems usually rely on a large amount of labeled data. The visual semantic large model can perform well on a small amount of labeled data or even unlabeled data. Since the pre-training process of the model usually involves a large amount of data, they may be more robust when processing images of different types, styles and sources. For noisy, low-resolution or uncommon image content, the visual semantic large model may still be able to provide meaningful search results. The visual semantic large model can also continue to fine-tune the model through online learning or transfer learning to adapt it to specific data sets or application scenarios, thereby continuously improving the accuracy and relevance of the search.

[0098] Based on the above-mentioned large model-based search method, the present disclosure further provides a large model-based search system, which will be described in detail below in conjunction with FIG4 .

[0099] FIG4 shows a structural block diagram of a large model-based search system according to an embodiment of the present disclosure.

[0100] As shown in FIG. 4 , the large model-based search system 400 of this embodiment includes a storage module 410 , a parsing module 420 , and a determination module 430 .

[0101] Storage module 410 is configured to store multiple information pairs in a database, where each information pair includes a captured image and descriptive text describing the captured image. The descriptive text is generated by processing the captured image using a pre-trained large visual semantic model after the intelligent camera captures the image. In one embodiment, storage module 410 can be configured to perform operation S210 described above, and will not be further described here.

[0102] The parsing module 420 is configured to parse the search request to obtain search information in response to receiving the search request. In one embodiment, the parsing module 420 may be configured to perform the operation S220 described above, which will not be described in detail herein.

[0103] The determination module 430 is used to determine search results that match the search information from the database, wherein the search results include at least one information pair. In one embodiment, the determination module 430 can be used to perform the operation S230 described above, which will not be repeated here.

[0104] According to an embodiment of the present disclosure, the search information includes search content information and search identification information. The determination module 430 includes: a first sub-determination unit and a second sub-determination unit.

[0105] The first sub-determining unit is configured to determine, from the plurality of information pairs, a plurality of associated information pairs associated with the search identification information.

[0106] The second sub-determining unit is configured to determine search results based on the search content information and a plurality of associated information pairs.

[0107] According to an embodiment of the present disclosure, an associated information pair includes: an associated captured image and associated descriptive text information describing the associated captured image. Determining search results based on search content information and multiple associated information pairs includes: converting the search content information, the associated captured image, and the associated descriptive text information into vector information of the search content, vector information of the associated captured image, and vector information of the associated descriptive text, respectively; and determining search results based on the vector information of the search content, the vector information of the associated captured image, and the vector information of the associated descriptive text.

[0108] According to an embodiment of the present disclosure, the search results include a target captured image and target descriptive text information. The search results are determined based on the vector information of the search content, the vector information of the associated captured image, and the vector information of the associated descriptive text, including: when the search content information is determined to be search text information, determining the similarity value between the vector information corresponding to the search text information and the vector information of the associated descriptive text; determining the associated descriptive text information corresponding to the vector information of the associated descriptive text whose similarity value exceeds a threshold as the target descriptive text information; determining the target captured image from multiple associated information pairs based on the target descriptive text information; or

[0109] When it is determined that the search content information is search image information, the similarity value between the vector information corresponding to the search image information and the vector information of the associated acquired image is determined; the associated acquired image corresponding to the vector information of the associated acquired image whose similarity value exceeds a threshold is determined as the target acquired image; and based on the target acquired image, the target descriptive text information is determined from multiple associated information pairs.

[0110] According to an embodiment of the present disclosure, the descriptive text information meets a preset character limit condition, and the search information includes the search text information.

[0111] The large model-based search system 400 may further include: a first extraction module and a first generation module.

[0112] The first extraction module is used to extract character feature information of all descriptive text information in the database.

[0113] The generating module is used to generate character number prompt information according to the character feature information, so that the user can determine the number of characters in the search text information according to the character number prompt information.

[0114] According to an embodiment of the present disclosure, the descriptive text information satisfies a preset description form restriction condition, and the search information includes the search text information.

[0115] The large model-based search system 400 may further include: a second extraction module and a second generation module.

[0116] The second extraction module is used to extract descriptive feature information of all descriptive text information in the database.

[0117] The second generating module is used to generate description prompt information according to the description feature information, so that the user can determine the description form of the search text information according to the description prompt information.

[0118] According to an embodiment of the present disclosure, the search identification information includes user identification information.

[0119] The large model-based search system 400 may further include: a third extraction module and a display form determination module.

[0120] The third extraction module is used to extract user attribute features from the user identification information.

[0121] The display format determination module is used to determine the display format of the search results according to the user attribute characteristics, wherein the display format includes at least one of the following: image format, video format, and voice broadcast format.

[0122] According to an embodiment of the present disclosure, any multiple modules in the storage module 410, the parsing module 420, and the determination module 430 can be combined into one module for implementation, or any one of the modules can be split into multiple modules. Alternatively, at least part of the functions of one or more of these modules can be combined with at least part of the functions of other modules and implemented in one module. According to an embodiment of the present disclosure, at least one of the storage module 410, the parsing module 420, and the determination module 430 can be at least partially implemented as a hardware circuit, such as a field programmable gate array (FPGA), a programmable logic array (PLA), a system on a chip, a system on a substrate, a system on a package, an application specific integrated circuit (ASIC), or can be implemented by hardware or firmware such as any other reasonable way of integrating or packaging the circuit, or implemented in any one of the three implementation methods of software, hardware, and firmware, or in an appropriate combination of any of them. Alternatively, at least one of the storage module 410, the parsing module 420, and the determination module 430 can be at least partially implemented as a computer program module, which can perform the corresponding function when the computer program module is executed.

[0123] According to an embodiment of the present disclosure, the large model-based search system of the present disclosure can perform online learning based on user feedback. For example, if a user marks a search result as irrelevant or low-quality, the system can use this information to fine-tune the model to provide more accurate results in the future. In addition to the basic text search, image search, or video search functions, other search modes are also supported. For example, a user can upload an image and then search for text descriptions or other images related to it. This increases the flexibility of the system, allowing it to meet a variety of different search needs.

[0124] According to an embodiment of the present disclosure, the user-interactive front-end functionality of the large-scale model-based search system of the present disclosure can be pre-configured to not only receive text queries from users, but also include a series of interface designs and algorithms that allow users to interact with the system more easily and intuitively. A concise and intuitive interface design is adopted in the interface design. To simplify the user's search experience, in addition to the traditional search box, some advanced search options may be provided, such as using natural language descriptions, voice input, or context-based suggested queries.

[0125] FIG5 shows a block diagram of an electronic device suitable for implementing a large model-based search method according to an embodiment of the present disclosure.

[0126] As shown in Figure 5, the electronic device 500 according to an embodiment of the present disclosure includes a processor 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage part 508 into a random access memory (RAM) 503. The processor 501 may include, for example, a general-purpose microprocessor (e.g., a CPU), an instruction set processor and / or a related chipset and / or a dedicated microprocessor (e.g., an application-specific integrated circuit (ASIC)), etc. The processor 501 may also include an onboard memory for caching purposes. The processor 501 may include a single processing unit or multiple processing units for performing different actions of the method flow according to an embodiment of the present disclosure.

[0127] Various programs and data required for the operation of the electronic device 500 are stored in the RAM 503. The processor 501, ROM 502, and RAM 503 are connected to each other via a bus 504. The processor 501 executes the various operations of the method flow according to the embodiment of the present disclosure by executing the programs in the ROM 502 and / or RAM 503. It should be noted that the programs may also be stored in one or more memories other than the ROM 502 and RAM 503. The processor 501 may also execute the various operations of the method flow according to the embodiment of the present disclosure by executing the programs stored in one or more memories.

[0128] According to an embodiment of the present disclosure, the electronic device 500 may further include an input / output (I / O) interface 505, which is also connected to the bus 504. The electronic device 500 may further include one or more of the following components connected to the I / O interface 505: an input portion 506 including a keyboard, a mouse, etc.; an output portion 507 including devices such as a cathode ray tube (CRT), a liquid crystal display (LCD), and a speaker; a storage portion 508 including a hard disk; and a communication portion 509 including a network interface card such as a LAN card or a modem. The communication portion 509 performs communication processing via a network such as the Internet. A drive 510 is also connected to the I / O interface 505 as needed. A removable medium 511, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, etc., is installed in the drive 510 as needed, so that a computer program read therefrom can be installed into the storage portion 508 as needed.

[0129] The present disclosure also provides a computer-readable storage medium, which may be included in the device / apparatus / system described in the above embodiments, or may exist independently and not be incorporated into the device / apparatus / system. The computer-readable storage medium carries one or more programs, and when executed, implements the method according to the embodiments of the present disclosure.

[0130] According to an embodiment of the present disclosure, a computer-readable storage medium may be a non-volatile computer-readable storage medium, for example, may include but is not limited to: a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination thereof. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that can be used by or in combination with an instruction execution system, apparatus, or device. For example, according to an embodiment of the present disclosure, a computer-readable storage medium may include the ROM 502 and / or RAM 503 described above and / or one or more memories other than ROM 502 and RAM 503.

[0131] The embodiments of the present disclosure also include a computer program product, which includes a computer program containing program code for executing the method shown in the flowchart. When the computer program product is run in a computer system, the program code is used to enable the computer system to implement the method provided by the embodiments of the present disclosure.

[0132] The computer program executes the above functions defined in the system / device of the embodiment of the present disclosure when the computer program is executed by the processor 501. According to the embodiment of the present disclosure, the system, device, module, unit, etc. described above can be implemented by a computer program module.

[0133] In one embodiment, the computer program may be stored on a tangible storage medium such as an optical storage device or a magnetic storage device. In another embodiment, the computer program may be transmitted and distributed in the form of a signal on a network medium, downloaded and installed via the communication portion 509, and / or installed from a removable medium 511. The program code contained in the computer program may be transmitted using any appropriate network medium, including but not limited to wireless, wired, or any suitable combination thereof.

[0134] In such an embodiment, the computer program can be downloaded and installed from a network via the communication section 509, and / or installed from a removable medium 511. When the computer program is executed by the processor 501, the above-described functions defined in the system of the embodiment of the present disclosure are performed. According to the embodiment of the present disclosure, the systems, devices, means, modules, units, etc. described above can be implemented by computer program modules.

[0135] According to an embodiment of the present disclosure, the program code for executing the computer program provided by the embodiment of the present disclosure can be written in any combination of one or more programming languages. Specifically, these computer programs can be implemented using high-level procedural and / or object-oriented programming languages, and / or assembly / machine languages. Programming languages ​​include, but are not limited to, languages ​​such as Java, C++, python, "C" or similar programming languages. The program code can be executed entirely on the user computing device, partially on the user device, partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device can be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or can be connected to an external computing device (for example, using an Internet service provider to connect via the Internet).

[0136] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the above-mentioned module, program segment, or a part of code contains one or more executable instructions for implementing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in an order different from that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram or flowchart, and the combination of boxes in the block diagram or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0137] Those skilled in the art will appreciate that the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways, even if such combinations and / or couplings are not explicitly described in this disclosure. In particular, the features described in the various embodiments and / or claims of this disclosure may be combined and / or coupled in various ways without departing from the spirit and teachings of this disclosure. All such combinations and / or couplings are intended to fall within the scope of this disclosure.

[0138] The embodiments of the present disclosure are described above. However, these embodiments are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although each embodiment has been described separately above, this does not mean that the measures in each embodiment cannot be used in combination to advantage. The scope of the present disclosure is defined by the appended claims and their equivalents. Without departing from the scope of the present disclosure, those skilled in the art may make various substitutions and modifications, which should all fall within the scope of the present disclosure.

Claims

1. A search method based on a large model, comprising: Storing a plurality of information pairs in a database, wherein the information pairs include: a captured image and descriptive text information for describing the captured image; the descriptive text information is generated after the captured image is captured by an intelligent camera and the captured image is processed by using a pre-trained visual semantics large model; In response to receiving the search request, parsing search information from the search request; A search result matching the search information is determined from the database, wherein the search result includes at least one of the information pairs.

2. The method according to claim 1, wherein: The search information includes search content information and search identification information; Wherein, determining search results matching the search information from the database includes: Determine a plurality of associated information pairs associated with the search identification information from the plurality of information pairs; The search result is determined based on the search content information and the plurality of associated information pairs.

3. The method according to claim 2, wherein: The associated information pair includes: an associated acquisition image and associated descriptive text information for describing the associated acquisition image; Wherein, determining the search result based on the search content information and the plurality of associated information pairs includes: Converting the search content information, the associated captured image, and the associated descriptive text information into vector information of the search content, vector information of the associated captured image, and vector information of the associated descriptive text, respectively; The search result is determined based on the vector information of the search content, the vector information of the associated captured image, and the vector information of the associated descriptive text.

4. The method according to claim 3, wherein: The search results include target acquisition images and target descriptive text information; Wherein, determining the search result based on the vector information of the search content, the vector information of the associated collected image, and the vector information of the associated descriptive text includes: In the case where it is determined that the search content information is search text information, determining a similarity value between the vector information corresponding to the search text information and the vector information of the associated descriptive text; Determining the associated descriptive text information corresponding to the vector information of the associated descriptive text whose similarity value exceeds a threshold as the target descriptive text information; Determining the target acquisition image from the plurality of associated information pairs according to the target descriptive text information; or In the case where it is determined that the search content information is search image information, determining a similarity value between the vector information corresponding to the search image information and the vector information of the associated acquired image; Determine the associated acquired image corresponding to the vector information of the associated acquired image whose similarity value exceeds a threshold as the target acquired image; According to the target acquisition image, the target descriptive text information is determined from the multiple associated information pairs.

5. The method according to claim 1, wherein: The descriptive text information meets a preset character limit condition, and the search information includes search text information; The method further comprises: Extracting character feature information of all descriptive text information in the database; Character number prompt information is generated according to the character feature information, so that the user can determine the number of characters in the search text information according to the character number prompt information.

6. The method according to claim 1, wherein: The descriptive text information satisfies a preset description form restriction condition, and the search information includes search text information; The method further comprises: Extracting descriptive feature information of all descriptive text information in the database; Description prompt information is generated according to the description feature information, so that the user can determine the description form of the search text information according to the description prompt information.

7. The method according to claim 2, wherein: The search identification information includes user identification information; The method further comprises: Extracting user attribute features from the user identification information; The display form of the search results is determined according to the user attribute characteristics, wherein the display form includes at least one of the following: image form, video form, and voice broadcast form.

8. A search system based on a large model, comprising: A storage module, used to store multiple information pairs in a database, wherein the information pairs include: a captured image and descriptive text information for describing the captured image; the descriptive text information is generated after the captured image is captured by an intelligent camera and the captured image is processed by using a pre-trained visual semantics large model; A parsing module, configured to parse the search request to obtain search information in response to receiving the search request; A determination module is used to determine search results matching the search information from the database, wherein the search results include at least one of the information pairs.

9. An electronic device, comprising: one or more processors; a memory for storing one or more programs, When the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

10. A computer-readable storage medium having executable instructions stored thereon, which, when executed by a processor, enables the processor to implement the method according to any one of claims 1 to 7.

Images