Book retrieval method and device based on large model technology, equipment and medium

Abstract

The invention provides a book retrieval method based on a large model technology, and relates to the technical field of artificial intelligence, in particular to the technical field of natural language processing, computer vision, large models and generative models. According to the specific implementation scheme, the method comprises the steps of generating a book description text of a book to be determined according to received input information; determining respective sub-images of a plurality of books to be matched from the target area image; and determining a target book from the plurality of to-be-matched books according to the book description text of the to-be-determined books and the respective sub-images of the plurality of to-be-matched books. The invention further provides a book retrieval device based on the large model technology, electronic equipment and a storage medium.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of natural language processing, computer vision, large models, and generative model technology. More specifically, the present disclosure provides a book retrieval method, device, electronic device, storage medium, and computer program product based on large model technology. Background Art

[0002] With the continuous development of artificial intelligence technology, the application of large models is becoming more and more extensive. For example, when faced with the technical problem of determining the object people actually want to find from a large number of similar items, the current solutions based on traditional models often find it difficult to accurately understand the actual needs of users, making it difficult to help users find the target object. The emergence of large models can more conveniently solve this problem. Summary of the invention

[0003] The present disclosure provides a book retrieval method, device, electronic device, storage medium and computer program product based on large model technology.

[0004] According to a first aspect, a book retrieval method based on a large model technology is provided, the method comprising: generating a book description text of a book to be determined based on received input information; determining a plurality of sub-images of books to be matched from a target area image; and determining a target book from a plurality of books to be matched based on the book description text of the book to be determined and the sub-images of the plurality of books to be matched.

[0005] According to the second aspect, a book retrieval device based on large model technology is provided, which includes: a description text determination module, used to generate a book description text of a to-be-determined book according to received input information; an image determination module, used to determine a plurality of sub-images of respective books to be matched from a target area image; a target book determination module, used to determine a target book from a plurality of books to be matched according to the book description text of the to-be-determined book and the sub-images of respective books to be matched.

[0006] According to a third aspect, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method provided according to the present disclosure.

[0007] According to a fourth aspect, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method provided according to the present disclosure.

[0008] According to a fifth aspect, a computer program product is provided, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and wherein the computer program implements the method provided according to the present disclosure when executed by a processor.

[0009] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0010] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0011] Figure 1 is a schematic diagram of an exemplary system architecture to which a book retrieval method and apparatus based on a large model technology can be applied according to an embodiment of the present disclosure;

[0012] Figure 2 is a flow chart of a book retrieval method based on a large model technology according to an embodiment of the present disclosure;

[0013] Figure 3 is a flow chart of a book retrieval method based on a large model technology according to another embodiment of the present disclosure;

[0014] Figure 4 is a service call diagram of a serverless architecture according to an embodiment of the present disclosure;

[0015] Figure 5 is a block diagram of a book retrieval device based on a large model technology according to an embodiment of the present disclosure; and

[0016] Figure 6 The invention is a block diagram of an electronic device for implementing a book retrieval method based on a large model technology according to an embodiment of the present invention. DETAILED DESCRIPTION

[0017] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0018] In the scenario of determining the books people want to find from a bookshelf with a large number of books, traditional solutions are mostly based on manual rules. Even if artificial intelligence models are introduced, they are traditional classification models, detection models, etc. The visual and language capabilities of traditional models are very weak and lack intelligence, so it is difficult to meet people's needs.

[0019] In the technical solution of the present disclosure, the collection, storage, use, processing, transmission, provision and disclosure of user personal information involved are in compliance with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0020] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0021] The embodiments of the present disclosure provide a book retrieval method based on big model technology. The big model can directly communicate with the user through natural language and understand the user's actual needs. The big model has image understanding capabilities and can accurately locate the location of the book the user wants to find.

[0022] The demand for finding books by users has obvious peaks and valleys. For example, the demand may increase on weekends and holidays, but may be less on weekdays. Therefore, the valley time of the demand for finding books may be longer, and deploying a dedicated server may cause a waste of resources.

[0023] In view of this, the embodiments of the present disclosure adopt the Serverless system architecture to meet the needs of users. Serverless is serverless computing, which provides a micro architecture. End customers do not need to deploy, configure or manage server services. The servers required for code operation are all provided by the cloud platform.

[0024] The serverless system architecture of the embodiment of the present disclosure refers to the use of at least part of the services of the book retrieval method based on the large model technology of the embodiment of the present disclosure without setting up a dedicated server, and at least part of the services can be implemented by the cloud platform. During the peak phase, multiple servers of the cloud platform can be opened to implement at least part of the services. During the trough phase, a small number of servers of the cloud platform can be opened to implement at least part of the services.

[0025] Figure 1 is a schematic diagram of an exemplary system architecture to which a book retrieval method and apparatus based on a large model technology can be applied according to an embodiment of the present disclosure. It should be noted that: Figure 1 What is shown is merely an example of a system architecture to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but it does not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios.

[0026] like Figure 1 As shown, the system architecture 100 according to this embodiment may include a terminal device 101, a robot 102, a network 103, and servers 104, 105, and 106. The network 103 is used to provide a medium for communication links between the terminal device 101, the robot 102, and the servers 104, 105, and 106. The network 103 may include various connection types, such as wired and / or wireless communication links, and the like.

[0027] For example, the terminal device 101 can be an electronic device with a display screen, such as a mobile phone, a tablet, or a notebook, and the user can input voice, text, or upload an image in the page displayed on the display screen. In addition, the terminal device 101 can also be provided with a camera, and the user can use the camera to collect bookshelf images. The terminal device 101 can call the big model to process based on the user's input information and the collected bookshelf images, and determine the target book that the user actually wants to find from the bookshelf images.

[0028] For another example, the robot 102 has a display screen, and the user can input voice, text, or upload an image in the page displayed on the display screen. In addition, the robot 102 can be provided with a camera, and the robot 102 can use the camera to collect bookshelf images. Then, the large model is called based on the user's input information and the collected bookshelf images to determine the target book that the user actually wants to find from the bookshelf images.

[0029] The embodiments of the present disclosure determine the target book that the user actually wants to find from the bookshelf image by calling a large model based on the user's input information and the collected bookshelf image, thereby improving the accuracy of book recognition, meeting the user's needs for finding books, and improving the user experience.

[0030] Figure 2 It is a flowchart of a book retrieval method based on large model technology according to an embodiment of the present disclosure.

[0031] like Figure 2 As shown, the book retrieval method 200 based on the large model technology includes operations S210 to S230.

[0032] In operation S210, a book description text of the to-be-determined book is generated according to the received input information.

[0033] When users use the terminal to search for books, they can input voice, text or upload images through the terminal. When users use the robot to search for books, they can input voice and text through the robot's display screen, and upload images through the robot's Bluetooth or connection cable.

[0034] For input voice and input text, the input voice and input text may contain a book title or a description of the book. Therefore, descriptive information of the book to be determined may be extracted from the input voice and input text, such as the name, author, type, etc. of the book, and the types may include textbooks, essays, poetry collections, novels, etc. For uploaded images, the name, author, color, publishing information, etc. of the book may be extracted from the image, and the information extracted from the image may be used to generate a description text.

[0035] The description information extracted from the input speech and the input text and the description information extracted from the input image may be combined to obtain a book description text of the book to be determined.

[0036] In operation S220, a plurality of sub-images of books to be matched are determined from the target region image.

[0037] The target area may be, for example, a bookshelf.

[0038] In the scenario where a user uses a terminal to search for books, the user can carry the terminal in front of the bookshelf and control the terminal to turn on the camera to collect the bookshelf image. The terminal can call the corresponding image processing service, pass the collected bookshelf image to the image processing service, and enable the image processing service to extract the sub-image of each book from the bookshelf image.

[0039] In the scenario where a user uses a robot to search for books, the robot can move to the bookshelf and turn on the camera to capture the bookshelf image after determining that the user has a need to search for books, for example, after receiving the user's input information. The robot can also call the image processing service and pass the captured bookshelf image to the image processing service, so that the image processing service can extract the sub-image of each book from the bookshelf image.

[0040] Each book in the bookshelf image can be used as a book to be matched. The image processing service can perform target detection on the bookshelf image and detect the position of each book. Based on the position, a sub-image of each book can be captured from the bookshelf image as the sub-image of the book to be matched.

[0041] In operation S230, a target book is determined from the multiple books to be matched according to the book description text of the book to be determined and the sub-images of the multiple books to be matched.

[0042] For example, a multimodal large model can be called to input the book description text and the sub-images of each book to be matched into the large model. The large model processes the book description text and each sub-image, and determines the sub-image that matches the book description text from multiple sub-images. The book to be matched in this sub-image is the target book.

[0043] The large model can return the sub-image of the target book to the terminal or robot, and the terminal or robot can mark the location of the target book in the bookshelf image according to the location of the sub-image in the bookshelf image, and display it to the user, so that the user can quickly confirm the book to be found. In addition, the robot can also emit laser or other rays according to the location information of the target book to point out the target image.

[0044] The embodiments of the present disclosure generate a book description text of the book to be determined according to the user's input information, capture a plurality of sub-images of the books to be determined from the bookshelf image, and determine the target book from the books to be determined based on the book description text and the sub-images of the books to be determined. This can accurately understand the user's needs, improve the accuracy of book recognition, meet the user's book retrieval needs, and enhance the user experience.

[0045] Figure 3 is a flowchart of a book retrieval method based on large model technology according to another embodiment of the present disclosure.

[0046] like Figure 3 As shown, this embodiment includes operations S311 to S314, operations S321 to S323, and operations S331 to S332. Operations S311 to S314 are steps for processing user input information. Operations S321 to S323 are steps for processing bookshelf images. Operations S331 to S332 are steps for calling large model services for processing.

[0047] In operation S311, input information is received, the input information including at least one of an input voice, an input text, and an input image.

[0048] In operation S312, the input speech is converted into a converted text, and a first feature description text is extracted from at least one of the converted text and the input text.

[0049] According to an embodiment of the present disclosure, the input speech is converted into converted text; and a description text containing at least one of the name, author, version information and publishing information of the book to be determined is extracted from at least one of the converted text and the input text as the first feature description text.

[0050] For example, the converted text obtained by converting the input speech can be combined with the input text to obtain a combined text. Next, the large language model can be called to summarize and refine the combined text, and the name, author, type, version information, publishing information, etc. of the book that the user wants to find can be extracted from the combined text. Types include textbooks, essays, poetry collections, novels, etc. The description text of these features is extracted from the combined text as the first feature description text.

[0051] In operation S313, a second feature description text of the to-be-determined book is extracted from the input image.

[0052] According to an embodiment of the present disclosure, at least one of shape, color and text is extracted from an input image; and a description text of at least one of the shape, color and text is generated as a second feature description text.

[0053] For example, a large multimodal model can be called to extract the shape, color, text, etc. of a book from an input image, describe the shape and color with text, and then combine the shape and color description text with the text in the image to obtain a second feature description text.

[0054] In operation S314, a book description text is determined based on at least one of the first feature description text and the second feature description text.

[0055] For example, the first feature description text and the second feature description text may be integrated, and the same features may be merged to obtain a book description text.

[0056] In operation S321, a bookshelf image is acquired.

[0057] In operation S322, a plurality of books to be matched and a position of each book to be matched are identified from the bookshelf image.

[0058] In operation S323, a plurality of sub-images of the books to be matched are captured from the bookshelf image.

[0059] For example, a user carries a terminal in front of a bookshelf to collect a bookshelf image, or a robot moves to a bookshelf to collect a bookshelf image. The terminal or robot can call the image processing service and pass the collected bookshelf image to the image processing service. The image processing service can identify the position of each book from the bookshelf image and extract a sub-image of each book from the bookshelf image.

[0060] In operation S331, a target book is determined from a plurality of books to be matched according to the book description text and the respective sub-images of the plurality of books to be matched.

[0061] For example, the multimodal large model can be called to input the book description text and the sub-images of multiple books to be matched into the multimodal large model. The multimodal large model processes the book description text and each sub-image, and determines the sub-image that matches the book description text from the multiple sub-images. The book in the sub-image is the target book.

[0062] In one example, color is an important feature of books. When a user specifies the color of a book to be searched, sub-images of books that are not of the specified color can be filtered out first, and then a large model can be used for matching, which can improve the speed.

[0063] In operation S332, the location of the target book is marked in the bookshelf image, a marked image is obtained, and the marked image is output.

[0064] Since the sub-image of each book cannot represent the location of the book on the bookshelf, in order to allow the user to find the target book, the book can be marked in the bookshelf image according to the location of the target book in the bookshelf image to obtain a marked image. The terminal or robot can display the marked image to allow the user to quickly confirm the book to be found.

[0065] In one example, the user can provide feedback on whether the target book is the actual book to be found, and input the feedback information into the big model, so that the big model can be fine-tuned to improve the accuracy of the big model in identifying books, making the big model more and more accurate in finding books.

[0066] The embodiments of the present disclosure call a large model to extract a first feature description text of a book to be determined from input speech and input text and a second feature description text of a book to be determined from an input image, generate a book description text of the book to be determined according to the first feature description text and the second feature description text, extract a plurality of sub-images of respective books to be matched from the bookshelf image, and then call the large model to determine a target book from the books to be matched based on the book description text and the sub-images of the books to be matched, thereby improving the accuracy of book recognition.

[0067] Figure 4 It is a service call diagram of a serverless architecture according to an embodiment of the present disclosure.

[0068] The embodiments of the present disclosure may use a serverless architecture to implement a book retrieval method based on a large model technology. Figure 4 As shown, the Serverless architecture provides an access service 40, which is used to call various other services to process user requests. Other services may include a speech conversion service 41, a text feature extraction service 42, an image feature extraction service 43, a retrieval service 44, and an image processing service 45.

[0069] For example, a user inputs voice, text, or uploads an image through a terminal or a display page of a robot, and the terminal can transmit the voice, text, or image input by the user to the access service 40 by initiating an http request. For the input voice, the access service 40 can call the voice conversion service 41 to convert the input voice into converted text. For the converted text and the text input by the user, the access service 40 can call the text feature extraction service 42 for processing, and the text feature extraction service 42 can specifically call the large model service 431 to extract features from the converted text and the input text to obtain a first feature description text. For example, the large model service 431 can include a large language model, and the large model service 431 can call the large language model to summarize the converted text and the input text to obtain a first feature description text. For the input image, the access service 40 can call the image feature extraction service 43 for processing, and the image feature extraction service 43 can specifically call the large model service 431 to extract features from the input image to obtain a second feature description text. For example, the big model service 431 may include a multimodal big model, and the big model service 431 may call the multimodal big model to extract features from the input image and generate a description text of the extracted image features, namely, a second feature description text.

[0070] For example, the terminal or robot can also transmit the bookshelf image captured by the camera to the access service 40 by initiating an http request. The access service 40 calls the image processing service 45 to process the bookshelf image. The image processing service 45 can specifically call the target detection service 451 for processing. For example, the target detection service 451 can use the target detection model to identify the books to be matched from the bookshelf image, and extract the sub-image of each book to be matched from the bookshelf image. The target detection model can be, for example, a YOLO model. The image processing service 45 can also call the cache service 452 to cache the sub-images of each book to be matched extracted by the target detection service 451.

[0071] The retrieval service 44 can obtain the sub-images of each to-be-matched book from the cache service 452, and then call the large model service 431 to determine the target book from multiple to-be-matched books according to the book description text and the sub-images of each to-be-matched book.

[0072] The search service 44 can also return the sub-image and location information of the target book to the access service 40. The access service 40 can re-mark the location of the target book in the bookshelf image, obtain a marked image, and return the marked image to the terminal. The terminal displays the marked image to the user, so that the user can intuitively observe the location of the book to be found.

[0073] The embodiments of the present disclosure adopt a Serverless architecture, which does not require the setting of a dedicated server. By transmitting user input information and bookshelf images captured by the camera to the access service, the access service calls various services to implement a book retrieval method based on large model technology, which can reduce resource usage and save costs.

[0074] According to an embodiment of the present disclosure, the present disclosure also provides a book retrieval device based on large model technology.

[0075] Figure 5 It is a block diagram of a book retrieval device based on a large model technology according to an embodiment of the present disclosure.

[0076] like Figure 5 As shown, the book search device 500 based on the large model technology includes a description text determination module 510 , an image determination module 520 and a target book determination module 530 .

[0077] The description text determination module 510 is used to generate a book description text of the book to be determined according to the received input information.

[0078] The image determination module 520 is used to determine the sub-images of the multiple books to be matched from the target area image.

[0079] The target book determination module 530 is used to determine a target book from a plurality of books to be matched according to the book description text of the book to be determined and the respective sub-images of the plurality of books to be matched.

[0080] According to an embodiment of the present disclosure, the input information includes at least one of input voice, input text and input image. The description text determination module 510 includes a first feature description text determination unit, a second feature description text determination unit and a book description text determination unit.

[0081] The first feature description text determination unit is used to extract the first feature description text of the book to be determined from at least one of the input voice and the input text.

[0082] The second feature description text determination unit is used to extract the second feature description text of the book to be determined from the input image.

[0083] The book description text determination unit is used to determine the book description text according to at least one of the first feature description text and the second feature description text.

[0084] The first feature description text determination unit includes a conversion subunit and a first extraction subunit.

[0085] The conversion subunit is used to convert the input speech into converted text.

[0086] The first extraction subunit is used to extract a description text containing at least one of the name, author, version information and publication information of the book to be determined from at least one of the conversion text and the input text as the first feature description text.

[0087] The second feature description text determination unit includes a second extraction subunit and a generation subunit.

[0088] The second extraction subunit is used to extract at least one of shape, color and text from the input image.

[0089] The generating subunit is used to generate a description text of at least one of shape, color and text as the second feature description text.

[0090] The image determination module 520 includes a recognition unit and a capture unit.

[0091] The recognition unit is used to recognize a plurality of books to be matched from the target area image.

[0092] The interception unit is used to intercept the sub-images of the multiple books to be matched from the target area image.

[0093] The book retrieval device 500 based on the large model technology further includes a position information determination module, a mark image generation module and an output module.

[0094] The position information determination module is used to determine the position information of each of the multiple books to be matched in the target area image.

[0095] The marked image generation module is used to mark the position of the target book in the target area image to obtain a marked image.

[0096] The output module is used to output the labeled image.

[0097] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0098] Figure 6 A schematic block diagram of an example electronic device 600 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0099] like Figure 6As shown, the device 600 includes a computing unit 601, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 602 or a computer program loaded from a storage unit 608 into a random access memory (RAM) 603. In the RAM 603, various programs and data required for the operation of the device 600 can also be stored. The computing unit 601, the ROM 602, and the RAM 603 are connected to each other via a bus 604. An input / output (I / O) interface 605 is also connected to the bus 604.

[0100] A number of components in the device 600 are connected to the I / O interface 605, including: an input unit 606, such as a keyboard, a mouse, etc.; an output unit 607, such as various types of displays, speakers, etc.; a storage unit 608, such as a disk, an optical disk, etc.; and a communication unit 609, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 609 allows the device 600 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0101] The computing unit 601 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 601 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 601 performs the various methods and processes described above, such as a book retrieval method based on a large model technology. For example, in some embodiments, a book retrieval method based on a large model technology may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 608. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 600 via the ROM 602 and / or the communication unit 609. When the computer program is loaded into the RAM 603 and executed by the computing unit 601, one or more steps of the book retrieval method based on the large model technology described above may be performed. Alternatively, in other embodiments, the computing unit 601 may be configured to execute the book retrieval method based on the large model technology in any other appropriate manner (for example, by means of firmware).

[0102] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0103] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0104] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0105] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0106] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0107] A computer system may include clients and servers. Clients and servers are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship to each other.

[0108] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.

[0109] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A book retrieval method based on large model technology, comprising: Generate a book description text of the book to be determined according to the received input information; Determine respective sub-images of a plurality of books to be matched from the target area image; as well as A target book is determined from the multiple books to be matched according to the book description text of the book to be determined and the sub-images of the multiple books to be matched.

2. The method according to claim 1, wherein: The input information includes at least one of input voice, input text and input image; and generating a book description text of the to-be-determined book according to the received input information includes: Extracting a first feature description text of the to-be-determined book from at least one of the input speech and the input text; Extracting a second feature description text of the to-be-determined book from the input image; and The book description text is determined according to at least one of the first feature description text and the second feature description text.

3. The method according to claim 2, wherein: The extracting the first feature description text of the to-be-determined book from at least one of the input speech and the input text comprises: Converting the input speech into converted text; and A description text including at least one of the name, author, version information and publication information of the to-be-determined book is extracted from at least one of the conversion text and the input text as the first feature description text.

4. The method according to claim 2, wherein: The step of extracting the second feature description text of the to-be-determined book from the input image comprises: Extracting at least one of shape, color and text from the input image; and Generate a description text of at least one of the shape, color and text as the second feature description text.

5. The method according to claim 1, wherein: The step of determining the respective sub-images of the plurality of books to be matched from the target area image comprises: identifying the plurality of books to be matched from the target area image; and The sub-images of the plurality of books to be matched are captured from the target area image.

6. The method according to claim 1, further comprising: Determine the position information of each of the plurality of books to be matched in the target area image; Marking the location of the target book in the target area image to obtain a marked image; as well as The marked image is output.

7. A book retrieval device based on large model technology, comprising: A description text determination module, used to generate a book description text of the book to be determined according to the received input information; An image determination module, used to determine the respective sub-images of a plurality of books to be matched from the target area image; as well as The target book determination module is used to determine the target book from the multiple books to be matched according to the book description text of the book to be determined and the sub-images of the multiple books to be matched.

8. The device according to claim 7, wherein: The input information includes at least one of input voice, input text and input image; the book description text determination module includes: A first feature description text determination unit, configured to extract a first feature description text of the to-be-determined book from at least one of the input speech and the input text; A second feature description text determination unit is used to extract a second feature description text of the to-be-determined book from the input image; and The book description text determination unit is used to determine the book description text according to at least one of the first feature description text and the second feature description text.

9. The device according to claim 8, wherein: The first feature description text determination unit includes: a conversion subunit, configured to convert the input speech into a conversion text; and The first extraction subunit is used to extract a description text containing at least one of the name, author, version information and publication information of the to-be-determined book from at least one of the conversion text and the input text as the first feature description text.

10. The device according to claim 8, wherein: The second feature description text determination unit includes: A second extraction subunit, configured to extract at least one of shape, color and text from the input image; and The generating subunit is used to generate a description text of at least one of the shape, color and text as the second feature description text.

11. The device according to claim 6, wherein: The sub-image determination module comprises: an identification unit, configured to identify the plurality of books to be matched from the target area image; and The interception unit is used to intercept the sub-images of the multiple books to be matched from the target area image.

12. The apparatus according to claim 6, further comprising: A position information determination module, used to determine the position information of each of the plurality of books to be matched in the target area image; A marked image generating module, used for marking the position of the target book in the target area image to obtain a marked image; as well as An output module is used to output the marked image.

13. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 6.

14. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1 to 6.

15. A computer program product, comprising a computer program, wherein the computer program is stored on at least one of a readable storage medium and an electronic device, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Images