Content extraction method and device, electronic equipment, storage medium and intelligent agent

Abstract

The invention provides a content extraction method and device, electronic equipment, a storage medium and an intelligent agent, relates to the technical field of artificial intelligence, in particular to the technical fields of intelligent recommendation, computer vision, deep learning, large models and the like, and can be applied to scenes such as AIGC content generation based on artificial intelligence. The method comprises the following steps: according to at least one image in input information, determining an image processing strategy matched with the image; according to the image processing strategy, the image is processed, at least one image block is obtained, the first size of the image is larger than the second size of the image block, and the number of the image blocks is determined by the image processing strategy; and extracting the large model from the input information and the image block input content to generate output information.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the fields of intelligent recommendation, computer vision, deep learning, large models, etc., and can be applied to scenarios such as AIGC content generation based on artificial intelligence. Specifically, it relates to a content extraction method, device, electronic device, storage medium, and intelligent agent. Background Art

[0002] With the continuous development of artificial intelligence technology, big model technology has also been applied in many fields. For example, big models are used to extract content from images.

[0003] However, there are many problems in using large models to extract content from images. For example, the input of content extraction is variable, such as the number of images, image size, image resolution, etc. Compatibility with variable inputs places high demands on the performance of large models. Summary of the invention

[0004] The present disclosure provides a content extraction method, device, electronic device, storage medium and intelligent agent.

[0005] According to one aspect of the present disclosure, a content extraction method is provided, comprising: determining an image processing strategy that matches the image based on at least one image in input information; processing the image according to the image processing strategy to obtain at least one image block, wherein a first size of the image is greater than a second size of the image block, and the number of image blocks is determined according to the image processing strategy; and inputting the input information and the image block into a content extraction model to obtain output information.

[0006] According to another aspect of the present disclosure, a content extraction device is provided, including: a determination module, used to determine an image processing strategy that matches an image based on at least one image in input information; a processing module, used to process at least one image according to the image processing strategy to obtain at least one image block, wherein a first size of the image is greater than a second size of the image block, and the number of image blocks is determined by the image processing strategy; and a content extraction module, used to input the input information and the image block into a content extraction model to obtain output information.

[0007] According to another aspect of the present disclosure, an intelligent agent is provided, including: an input module for receiving input information; a processing module for determining a target task based on the input information received by the input module, determining a large model based on the target task, and obtaining output information by calling the large model to execute the above content extraction method; and an output module for outputting the output information obtained by the processing module.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above method.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above method.

[0010] According to another aspect of the present disclosure, a computer program product is provided, including a computer program, and the computer program implements the above method when executed by a processor.

[0011] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.

[0013] Figure 1 An exemplary system architecture that can be applied to a content extraction method and device according to an embodiment of the present disclosure is schematically shown;

[0014] Figure 2 A flowchart of a content extraction method according to an embodiment of the present disclosure is schematically shown;

[0015] Figure 3A A scene graph schematically illustrates at least one image block obtained by processing at least one image based on an image processing strategy according to an embodiment of the present disclosure;

[0016] Figure 3B A scene graph schematically shows at least one image block obtained by processing at least one image based on an image processing strategy according to another embodiment of the present disclosure;

[0017] Figure 3C A scene graph schematically illustrates determining an image processing strategy according to a first parameter and a second parameter and obtaining at least one image block according to the determined image processing strategy according to an embodiment of the present disclosure;

[0018] Figure 4 A schematic diagram of a data flow for determining a third parameter of a first image processing strategy according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 5A schematic diagram of a data flow for determining candidate parameters according to an embodiment of the present disclosure is shown;

[0020] Figure 6 Schematically shows a flow chart of updating at least one image block according to an embodiment of the present disclosure;

[0021] Figure 7 A schematic diagram of a content extraction framework according to an embodiment of the present disclosure is illustrated;

[0022] Figure 8 A block diagram of a content extraction device according to an embodiment of the present disclosure is schematically shown;

[0023] Fig. 9 A schematic diagram of a structure of an intelligent agent according to an embodiment of the present disclosure is shown; and

[0024] Fig.10 A block diagram of an electronic device suitable for implementing a content extraction method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0025] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0026] With the rapid development of the computer vision industry, users' demand for content continues to increase. The input for content extraction using large models is not limited to text, but also includes a comprehensive understanding of multimodal information such as images and videos. In addition, the input for content extraction varies under various needs, such as content extraction for different numbers of images, content extraction for videos of different lengths, and content extraction for multimodal information consisting of images, videos, and text.

[0027] However, changing inputs place higher demands on the performance of large content extraction models. For example, when extracting content from a single image, it is difficult for a large content extraction model to extract accurate content because a single image contains limited information. When extracting content from multiple images or videos, multiple images and videos contain a large amount of information, which not only affects the efficiency of content extraction due to the large amount of information processed, but also affects the accuracy of content extraction due to the inclusion of redundant information. In addition, existing large content extraction models are usually designed for single-image tasks, and the cost of modifying the large content extraction model to adapt to the changing number of images is too high, especially the modification of the number of tokens received by the large content extraction model. In summary, the existing technology at least has technical problems such as how to be compatible with changing inputs and how to increase the speed of content extraction.

[0028] Therefore, the present disclosure provides a content extraction method including: determining an image processing strategy matching the image according to at least one image in the input information; processing at least one image according to the image processing strategy to obtain at least one image block, wherein the first size of the image is greater than the second size of the image block, and the number of image blocks is determined according to the image processing strategy; inputting the input information and the image block into a content extraction model to obtain output information. Through the embodiments of the present disclosure, the technical effect of improving the content extraction speed when compatible with variable input is achieved.

[0029] Figure 1 An exemplary system architecture that can be applied to a content extraction method and apparatus according to an embodiment of the present disclosure is schematically shown.

[0030] It should be noted that Figure 1 The examples shown are only examples of system architectures to which the embodiments of the present disclosure can be applied, in order to help those skilled in the art understand the technical content of the present disclosure, but do not mean that the embodiments of the present disclosure cannot be used in other devices, systems, environments or scenarios. For example, in another embodiment, the exemplary system architecture to which the content extraction method and apparatus can be applied may include a terminal device, but the terminal device may implement the content extraction method and apparatus provided by the embodiments of the present disclosure without interacting with the server.

[0031] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104 and a server 105. The network 104 is used to provide a medium for communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0032] Users can use terminal devices 101, 102, 103 to interact with server 105 through network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients and / or social platform software, etc. (only as examples).

[0033] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0034] The server 105 may be a server that provides various services, such as a background management server (only an example) that provides support for the content browsed by the user using the terminal devices 101, 102, and 103. The background management server may analyze and process the received data such as user requests, and feed back the processing results (such as web pages, information, or data obtained or generated according to the user request) to the terminal device.

[0035] The server can be a cloud server, also known as a cloud computing server or cloud host, which is a host product in the cloud computing service system to solve the defects of difficult management and weak business scalability in traditional physical hosts and VPS services ("Virtual Private Server", or "VPS" for short). The server can also be a server of a distributed system, or a server combined with blockchain.

[0036] It should be noted that the content extraction method provided in the embodiment of the present disclosure can generally be executed by the terminal device 101, 102, or 103. Accordingly, the content extraction apparatus provided in the embodiment of the present disclosure can also be set in the terminal device 101, 102, or 103.

[0037] Alternatively, the content extraction method provided in the embodiment of the present disclosure may also be generally performed by the server 105. Accordingly, the content extraction device provided in the embodiment of the present disclosure may generally be arranged in the server 105. The content extraction method provided in the embodiment of the present disclosure may also be performed by a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the content extraction device provided in the embodiment of the present disclosure may also be arranged in a server or server cluster that is different from the server 105 and can communicate with the terminal devices 101, 102, 103 and / or the server 105.

[0038] For example, the user can edit the input information by interacting with the terminal devices 101, 102, and 103. The terminal devices 101, 102, and 103 send the acquired input information to the server 105, and the server 105 determines an image processing strategy that matches the image based on at least one image in the input information; processes the image according to the image processing strategy to obtain at least one image block, wherein the first size of the image is greater than the second size of the image block, and the number of image blocks is determined by the image processing strategy; extracts a large model from the input information and the image block input content to generate output information. The terminal devices 101, 102, and 103 can receive and display the output information from the server 105. Alternatively, a server or server cluster that can communicate with the terminal devices 101, 102, 103 and / or the server 105 determines an image processing strategy that matches the image based on at least one image in the input information; processes the image according to the image processing strategy to obtain at least one image block; extracts a large model from the input information and the image block input content to generate output information. Alternatively, the terminal devices 101, 102, 103 determine an image processing strategy that matches the image based on at least one image in the input information; and process the image based on the image processing strategy to obtain at least one image block. The input information and at least one image block are then sent to the server 105, or to a server or server cluster that can communicate with the terminal devices 101, 102, 103 and / or the server 105, which inputs the input information and the image block into the content extraction model to generate output information. Alternatively, the content extraction model is set in the terminal devices 101, 102, 103, and the terminal devices 101, 102, 103 can directly call the local content extraction model to generate output information.

[0039] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is only for illustration. Any number of terminal devices, networks and servers may be provided according to implementation requirements.

[0040] In the embodiments of the present disclosure, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved all comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0041] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information.

[0042] Figure 2 The flowchart of the content extraction method according to the embodiment of the present disclosure is schematically shown. Figure 2 As shown, embodiment 200 includes operations S210 to S230.

[0043] In operation S210, an image processing strategy matching the image is determined according to at least one image in the input information.

[0044] The input information may be directly input by the user. For example, the input information may only include at least one image of a single mode; or the input information may also include multi-modal input, such as at least one image and text. The input information may also be obtained by processing the user input. For example, the user may input video information, and at least one image may be obtained by processing the video information.

[0045] The image processing strategy is used to indicate at least one processing to be performed on the image, for example, it may be processing such as segmentation, reduction, sampling, etc.

[0046] The image processing strategy matched with the image may be an image processing strategy matched with each image, and the image processing strategy matched with each image may be the same or different, thereby the image may be processed according to the image processing strategy matched with each image. Alternatively, at least one image may be regarded as a whole, and the image processing strategy matched with the image is also a unified image processing strategy matched with all images, thereby all images may be processed using a unified image processing strategy.

[0047] It should be noted that, when the input information includes images, the images in the multiple input information obtained through multiple inputs may be the same or different, such as the content, quantity, size, resolution, etc. of the images may be the same or different. Therefore, the image processing strategy matched with at least one image in each input information may be different.

[0048] For example, the correspondence between the content, quantity, size and resolution of the image and the image processing strategy can be pre-configured. For at least one image in the input information, the image processing strategy that matches the at least one image can be determined based on the content, quantity, size and / or resolution of the at least one image and the above correspondence. For example, an image with content of type 1 corresponds to image processing strategy 1; an image with size 2 corresponds to image processing strategy 2... When it is determined that the content of at least one image is type 1 and has size 2, image processing strategy 1 can be used as the image processing strategy that matches the at least one image. Alternatively, the image processing strategy that matches the at least one image can also be determined based on the amount of information contained in the at least one image.

[0049] In operation S220, the image is processed according to the image processing strategy to obtain at least one image block, wherein a first size of the image is greater than a second size of the image block, and the number of the image blocks is determined according to the image processing strategy.

[0050] After determining the image processing strategy that matches the image, the image can be processed according to the image processing strategy.

[0051] For example, at least one image may be uniformly processed based on an image processing strategy to obtain at least one image block. For example, a single image in at least one image may be processed to obtain a single image block, thereby obtaining at least one image block; or, a single image may be processed to obtain multiple image blocks, thereby obtaining at least one image block. Alternatively, each image may be processed according to an image processing strategy matching each image, and processing each image may obtain a single or multiple image blocks, thereby obtaining at least one image block.

[0052] After processing at least one image according to the image processing strategy, the number of image blocks obtained is greater than or equal to the image. For example, for a single image, at least one processing can be performed on the image according to the image processing strategy to obtain at least one image block, and the first size of the image in the input information is greater than the second size of the processed image block. For example, a single image can be reduced / downsampled to obtain a single image block of smaller size. Alternatively, a single image can be segmented to obtain multiple image blocks of smaller size. Alternatively, a single image can be first reduced to obtain an image block, and then the single image can be segmented to obtain multiple image blocks. In addition, at least one image can be screened according to the image processing strategy before performing the above-mentioned segmentation, reduction and downsampling operations.

[0053] In operation S230, a large model is extracted from the input information and the image block input content to obtain output information.

[0054] The content extraction large model may be a large multimodal model (LMMs) or a large language model (LLMs), which is used to extract content matching the user's needs from the input information and at least one image block and generate output information. For image-based content extraction tasks, the content extraction large model may perform content extraction from at least one image block and generate output information.

[0055] In an embodiment of the present disclosure, an image processing strategy matching an image is determined based on at least one image in the input information; the image is processed according to the image processing strategy to obtain at least one image block, wherein the first size of the image is greater than the second size of the image block, and the number of image blocks is determined according to the image processing strategy; the input information and at least one image block are input into a large content extraction model to obtain output information. Since an image processing strategy matching at least one image in the input information is determined in the input stage of the large content extraction model, and at least one image is preprocessed using the image processing strategy to obtain an image block of a smaller size, the large content extraction model can understand the image block of a smaller size more quickly, thereby improving the content extraction speed. In addition, by determining an image processing strategy matching at least one image, the number of generated image blocks can be adjusted through the image processing strategy without modifying the token of the large content extraction model, thereby achieving compatibility with multiple image tasks.

[0056] According to an embodiment of the present disclosure, for operation S210, an image processing strategy matching the image is determined based on at least one image in the input information, including: determining the image processing strategy based on an information amount parameter of the image; wherein the image processing strategy includes a first image processing strategy and a second image processing strategy, the first image processing strategy being used to instruct the image to be segmented and reduced to obtain image blocks; the second image processing strategy being used to instruct the image to be reduced to obtain image blocks.

[0057] The information volume parameter is used to indicate the amount of information contained in at least one image. It can be understood that the more images there are and the richer the semantics of the images are, the more information at least one image includes. Therefore, the information volume parameter may include at least one of the following: a parameter indicating the number of images, a parameter indicating the semantic richness of the images.

[0058] For content extraction tasks, the larger the size of the image, the slower the processing speed of the content extraction model, and the more images are processed, the slower the processing speed; however, blindly reducing the size and number of processed images may affect the accuracy of content extraction. Therefore, an image processing strategy matching at least one image can be determined according to the information volume parameter of at least one image to improve the accuracy of content extraction and / or improve the speed of content extraction.

[0059] In one embodiment, when the information volume parameter indicates that at least one image contains a large amount of information, a first image processing strategy may be determined; otherwise, a second image processing strategy may be determined to improve the accuracy of content extraction.

[0060] In another embodiment, when the information volume parameter indicates that the information volume of at least one image is large, the second image processing strategy is determined; otherwise, the first image processing strategy is determined to increase the content extraction speed.

[0061] In yet another embodiment, when the information volume parameter indicates that the information volume of at least one image is within a certain interval, a first image processing strategy is determined; otherwise, a second image processing strategy is determined, thereby improving the content extraction speed while maximizing the effective information of the image and improving the content extraction accuracy.

[0062] According to an embodiment of the present disclosure, the information volume parameter includes a first parameter, and the first parameter is used to indicate the number of images in the input information. According to the information volume parameter of the image, the image processing strategy is determined, including: when the first parameter is less than or equal to a first threshold, determining the first image processing strategy; when the first parameter is greater than the first threshold, determining the second image processing strategy.

[0063] For example, if the input information includes 5 images, the first parameter may be 5.

[0064] The first threshold is used to limit the amount of information of at least one image from a quantitative perspective, and the first threshold can be determined based on experience. For example, the first threshold can be determined based on the number of tokens limited by the content extraction model architecture. Thus, an image processing strategy that matches the at least one image can be determined based on the comparison relationship between the first parameter and the first threshold.

[0065] The first parameter is less than or equal to the first threshold, indicating that the number of images in the input information can meet the token number limit of the content extraction large model architecture, and a unified first image processing strategy can be determined for at least one image. Therefore, after at least one image is segmented according to the first image processing strategy, the obtained image blocks can indicate the local features of the image, so that the content extraction large model can fully extract the features in the image blocks and improve the accuracy of content extraction. In addition, the first image processing strategy also includes image blocks after the image is reduced, that is, image blocks including overall features. After the image blocks including overall features and image blocks including local features are input into the content extraction large model, the content extraction large model can further improve the accuracy of content extraction based on spatial features.

[0066] The first parameter is greater than the first threshold, indicating that the number of images in the input information may not meet the token number limit of the content extraction large model architecture, and a unified second image processing strategy can be determined for at least one image. Each image is reduced according to the second image processing strategy to obtain an image block, thereby effectively fusing the features of a large number of images into vector features corresponding to a small number of image sizes (such as vector features of a single image size). In addition, the reduction operation can also avoid the decrease in accuracy caused by the lack of image content, thereby improving the content extraction accuracy while improving the content extraction efficiency.

[0067] For example, the number of tokens that the content extraction large model can receive at the same time is 10. When the input information includes 1 image, the 1 image can be divided into 9 smaller image blocks, and the image can be reduced to 1 smaller image block. In this case, while meeting the 10-token limit of the content extraction large model, the content extraction large model can understand the content of each image block at a finer granularity, thereby improving the accuracy of content extraction. When the input information includes 5 images, the 5 images can be reduced into 5 smaller image blocks respectively. As a result, the content extraction large model can still extract content from the complete 5 image blocks at the same time, thereby improving the efficiency of content extraction while improving the accuracy of content extraction.

[0068] In an embodiment of the present disclosure, a first image processing strategy is determined when the first parameter is less than or equal to a first threshold; and a second image processing strategy is determined when the first parameter is greater than the first threshold, so that the number of image blocks of the input content extraction model is controlled by the first image processing strategy or the second image processing strategy, thereby achieving content extraction compatible with a variety of image quantities and improving content extraction efficiency and accuracy.

[0069] Figure 3A The schematic diagram shows a scene diagram of at least one image block obtained by processing at least one image based on an image processing strategy according to an embodiment of the present disclosure. Figure 3A As shown, in embodiment 300A, the input information includes at least one image, and the at least one image can constitute an image set 301 .

[0070] When the first parameter is less than or equal to the first threshold, the first image processing strategy can be determined, and the image set 301 is processed according to the first image processing strategy to obtain at least one image block, and at this time, the at least one image block can form a first image block set 302. Taking the image set 301 including M images as an example, the first image block set 302 may include 5*M image blocks. Taking the first image as an example, after the first image is processed according to the first image processing strategy, multiple image blocks 302-1 are obtained, and the multiple image blocks 302-1 are, from left to right, an image block after the first image is reduced and four image blocks obtained by cutting from the first image.

[0071] When the first parameter is greater than the first threshold, a second image processing strategy can be determined, and the image set 301 is processed according to the second image processing strategy to obtain at least one image block. At this time, the at least one image block can form a second image block set 303. The second image block set 303 can include M image blocks, all of which are obtained by reducing the image in the input information.

[0072] According to an embodiment of the present disclosure, the information volume parameter includes a second parameter for each image, and the second parameter is used to indicate the semantic richness of the image; based on the information volume parameter of the image, an image processing strategy is determined, including: for each image, when the second parameter is less than or equal to a second threshold, determining a second image processing strategy that matches the image; when the second parameter is greater than the second threshold, determining a first image processing strategy that matches the image.

[0073] Semantic richness can be understood as the amount of image information in an image, such as the number of objects in the image, the spatial relationship between objects in the image, the posture information of the objects, etc.

[0074] The second parameter may be the number of objects in the image or the number of objects of a specific object type in the image. The second threshold is used to limit the amount of information in each image from the perspective of semantic richness, and may be determined empirically or based on historical data of a content extraction macro model.

[0075] It is understandable that the content and amount of information contained in images may differ from one image to another. For example, the semantic richness of image 1 including multiple objects is higher than that of image 2 including a single object, because image 1 includes more objects and image 1 also includes a spatial relationship indicating the relationship between multiple objects. Therefore, for each image in the input information, an image processing strategy matching each image can be determined based on the second parameter of each image.

[0076] The second parameter is less than or equal to the second threshold, indicating that the semantic richness of the current image is low and the importance of the image content is low, and a second image processing strategy matching the image can be determined. Therefore, after the image is reduced according to the second image processing strategy to obtain the image block, not only the number of image blocks input into the content extraction model can be reduced, but also the impact on the accuracy of content extraction can be reduced.

[0077] The second parameter is greater than the second threshold, indicating that the semantic richness of the current image is high and the importance of the image content is high, and the first image processing strategy matching the image can be determined. Therefore, after the image is divided into multiple image blocks according to the first image processing strategy and the image is reduced to obtain image blocks, the content extraction model can fully extract the local features indicated by each image block, reduce the global features of the image block, and improve the accuracy of content extraction.

[0078] Figure 3B A scene graph of at least one image block obtained by processing at least one image based on an image processing strategy according to another embodiment of the present disclosure is schematically shown.

[0079] like Figure 3B As shown, in embodiment 300B, at least one image in the input information can constitute an image set 301. For each image, an image processing strategy matching each image is determined according to the second parameter, and after the image is processed based on the determined image processing strategy, at least one image block can be obtained. After processing multiple images, a third image block set 304 is obtained.

[0080] Taking the image set 301 including M images as an example, the image processing strategy matching the first image in the image set 301 may be the second image processing strategy, and after the first image is reduced according to the second image processing strategy, an image block 304-1 is obtained; ... the image processing strategy matching the Mth image in the image set 301 may be the first image processing strategy, and after the M images are processed according to the first image processing strategy, multiple image blocks 304-M are obtained, and the multiple image blocks 304-M include an image block obtained by reducing the Mth image and 4 image blocks obtained by splitting the Mth image, for a total of 5 image blocks.

[0081] In this embodiment, different image processing strategies can be determined for multiple images in the input information according to the second parameter to control the number of obtained image blocks while maximizing the effective information in a single image, thereby simultaneously improving the content extraction speed of the content extraction model.

[0082] According to an embodiment of the present disclosure, the method further includes: performing object detection on the image to obtain an object detection result, wherein the object detection result includes at least one of the number of objects and the type of objects in the image; and determining a second parameter based on the object detection result.

[0083] Object detection algorithms can be used to perform object detection on images to obtain object detection results. For example, object detection algorithms can include: target detection algorithms based on deep convolutional neural networks (You Only Look Once, YOLO), algorithms based on region convolutional neural networks (Regions with CNN features, R-CNN), algorithms based on single shot multibox detectors (SSD), and algorithms based on regions with FCN features (R-FCN).

[0084] For example, when the object detection result includes the number of objects, the number of objects can be determined as the second parameter. Alternatively, when the object detection result includes the object type, the second parameter can be determined according to the object type, such as when the object type is a predetermined object type, the second parameter is 1; otherwise, the second parameter is zero.

[0085] Alternatively, when the object detection result includes the number of objects and the object type, the number of objects of the predetermined object type is determined as the second parameter. The predetermined object type can be specified by text in the input information, or can be determined by an input path of the input information. For example, after the user specifies to extract character information in the input information, or inputs the above input information through a character analysis path, the predetermined object type can be determined as a character type.

[0086] In an embodiment of the present disclosure, by performing object detection on an image and determining a second parameter based on the object detection result, the semantic richness of the image can be determined from the perspective of the number of objects and / or object type, thereby achieving an image processing strategy that matches each image based on the number of objects and / or object type of each image in the current input, thereby controlling the number of image blocks obtained while maximizing the effective information within a single image, thereby improving the content extraction speed and accuracy.

[0087] According to an embodiment of the present disclosure, the information volume parameter also includes a second parameter, which is used to indicate the semantic richness of the image; the method also includes: when the first parameter is greater than the first threshold and the second parameter is less than or equal to the second threshold, determining the second image processing strategy; when the first parameter is greater than the first threshold and the second parameter is greater than the second threshold, determining the first image processing strategy.

[0088] The first parameter is greater than the first threshold, that is, the input information includes multiple images. When the number of images included in the input information is large enough, the large content extraction model can only use a limited number of tokens to extract content from multiple reduced image blocks at the same time. However, when the input information includes multiple images, but the number of images is moderate, the large content extraction model can not only use a limited number of tokens to extract content from multiple reduced image blocks, but there may also be remaining token resources, but these tokens are not sufficient to support segmenting all images into image blocks for processing. In this case, it is necessary to selectively segment some of the multiple images.

[0089] Therefore, when the first parameter is greater than the first threshold, for each image, a comparison relationship between the second parameter of the image and the second threshold is determined, so as to selectively segment part of the images in the plurality of images.

[0090] For example, the first threshold may be 1, and the second threshold may be 3 to limit the number of person-type objects in the image to 3. If the input information includes image 1 and image 2, and image 1 and image 2 include 1 and 5 person-type objects, respectively. In this case, the first parameter "2" is greater than the first threshold "1", and the second parameter of each image is determined. Since the second parameter "1" of image 1 is less than the second threshold "3", and the second parameter "5" of image 2 is greater than the second threshold "3", the second image processing strategy can be used for image 1 and the first image processing strategy can be used for image 2.

[0091] In this embodiment, when the first parameter is less than or equal to the first threshold, the image processing strategy matched with each image is the first image processing strategy. For example, still taking the first threshold 1 as an example, when the input information includes a single image, the first parameter is equal to the first threshold. In this case, when the content extraction large model only extracts content from a single image block, there will be remaining token resources. Therefore, when the first parameter is less than or equal to the first threshold, the first image processing strategy is adopted for the single image, and the single image is segmented and reduced to obtain an image block including finer-grained image features. The above only takes a single image as an example, but the first threshold can be 2, 3, etc., and is not limited here.

[0092] In the embodiment of the present disclosure, when the first parameter is greater than the first threshold and the second parameter is less than or equal to the second threshold, the second image processing strategy is determined. At this time, the large content extraction model can use a limited number of tokens to simultaneously extract content from multiple smaller image blocks, and the content extraction speed is faster; in addition, within the limit of the number of tokens, the large content extraction model can also extract content from image blocks containing finer-grained image features, and the content extraction accuracy is better. When the first parameter is greater than the first threshold and the second parameter is greater than the second threshold, the first image processing strategy is determined, so that in the content extraction scenario, the effective information of the image in the input information is maximized to improve the content extraction accuracy, and the number of image blocks is controlled to improve the content extraction speed.

[0093] Figure 3C The schematic diagram shows a scene diagram of determining an image processing strategy according to a first parameter and a second parameter and obtaining at least one image block according to the determined image processing strategy according to an embodiment of the present disclosure. Figure 3C As shown, embodiment 300C includes operations S310-S340 and operations S341-S343.

[0094] In operation S310, a first parameter is determined according to at least one image.

[0095] In operation S320, it is determined whether the first parameter is greater than a first threshold value. When the first parameter is greater than the first threshold value, operation S330 is performed; when the first parameter is less than or equal to the first threshold value, operation S341 is performed.

[0096] In operation S330, a second parameter of each image is determined.

[0097] In operation S340, it is determined whether the second parameter of the i-th image is greater than a second threshold. The i-th image is any image of at least one image in the input information, and i is greater than or equal to 1. When the second parameter of the i-th image is greater than the second threshold, operation S342 is performed; when the second parameter of the i-th image is less than or equal to the second threshold, operation S343 is performed.

[0098] In operation S341, each image is processed according to the first image processing strategy. In operation S341, each image in the input information may be segmented and reduced according to the first image processing strategy to obtain a plurality of image blocks corresponding to each image.

[0099] In operation S342, the i-th image is processed according to the first image processing strategy.

[0100] In operation S343, the i-th image is processed according to the second image processing strategy.

[0101] By repeatedly performing operation S342 and / or repeatedly performing operation S343, each image in the at least one image may be processed to obtain at least one image block. For example, by repeating only operation S343, the number of the at least one image block obtained is equal to the number of images in the input information; otherwise, the number of the at least one image block obtained is greater than the number of images in the input information.

[0102] In the above embodiment of the present disclosure, after determining the first image processing strategy, the first image processing strategy may indicate to segment the image into a predetermined number of image blocks and to reduce the image. Alternatively, when determining the first image processing strategy, the number of image blocks obtained by segmenting the image may be dynamically determined for at least one image input each time.

[0103] According to an embodiment of the present disclosure, the first image processing strategy includes a third parameter, and the third parameter is used to indicate the number of image blocks obtained by segmenting the image. The method also includes: determining multiple candidate parameters according to the number restriction parameter and the scale information of the image, wherein the number restriction parameter is determined according to the architecture of the content extraction model; and determining the third parameter from the multiple candidate parameters according to the first size of the image and the second size of the image block to be segmented.

[0104] It should be noted that the first image processing strategy determined according to at least one of the first parameter and the second parameter may include a third parameter. For example, when determining the first image processing strategy according to the first parameter, a unified third parameter may be determined for at least one image, that is, the number of image blocks obtained by segmenting each image is consistent. When determining the first image processing strategy matching each image according to the second parameter, or according to the first parameter and the second parameter, the third parameter of the first image processing strategy matching each image may be the same or different.

[0105] The quantity limit parameter indicates the token quantity limit of the content extraction model, which can be determined according to the architecture of the content extraction model. For example, the quantity limit parameter can be determined according to the number of tokens designed in the architecture of the content extraction model, such as the quantity limit parameter is equal to the number of tokens, one-half of the number of tokens, one-quarter of the number of tokens, etc.

[0106] The first size of the image can be expressed by the number of pixels, such as 2560*1920, where the units of 2560 and 1920 are both the number of pixels; the size of the image can also be expressed by length, such as 22mm*32mm.

[0107] The ratio information of the image may be the aspect ratio of the image, such as 3:2, 5:8, 1:1, etc. The ratio information of the image may be determined according to the first size of the image. For example, when the first size is 2560*1920, the ratio information of the image is 4:3; when the first size is 22mm*32mm, the ratio information of the image is 11:16.

[0108] The candidate parameter can be understood as the candidate third parameter, that is, the number of candidate image blocks to be segmented. The candidate parameter can be in the form of row parameter (m) * column parameter (n), where the row parameter (m) represents the number of image blocks to be segmented in a certain row, and the column parameter (n) represents the number of image blocks to be segmented in a certain column. The product of the row parameter (m) and the column parameter (n) is also the candidate parameter.

[0109] Determining multiple candidate parameters may include: selecting candidate parameters that are less than or equal to the quantity limit parameter and whose ratio information of the number of rows to the number of columns is the same as the ratio information of the image. For example, when the ratio information of the image is 4:3, the candidate parameters may be 4*3, 8*6, 12*9, etc.

[0110] For the determined multiple candidate parameters, a third parameter is determined from the multiple candidate parameters according to the first size of the image and the second size of the image block to be segmented, so that the number of image blocks obtained by segmentation can meet the predetermined condition. The predetermined condition can be: maximizing the effective information of a single image and the number of image blocks is as small as possible. In the embodiment of the present disclosure, the second size can be predetermined, such as 224*224; it can also be dynamically changed within a certain range.

[0111] In an embodiment of the present disclosure, a plurality of candidate parameters are determined based on a quantity limitation parameter and image proportion information, wherein the quantity limitation parameter is determined based on the architecture of a large content extraction model; a third parameter is determined from a plurality of candidate parameters based on a first size of the image and a second size of the image block to be segmented, and the number of image blocks to be segmented can be dynamically determined for at least one image in current input information, thereby improving the accuracy and speed of subsequent content extraction of the input information and at least one image block.

[0112] In the embodiment of the present disclosure, the edge of the image may be used as a starting point and the image may be segmented inward to obtain multiple image blocks. The multiple segmented image blocks may include overlapping areas.

[0113] Figure 4 The data flow diagram for determining the third parameter of the first image processing strategy according to an embodiment of the present disclosure is schematically shown.

[0114] like Figure 4As shown, in the embodiment 400, a quantity restriction parameter 402 can be determined based on the architecture of the content extraction model 401, and image scale information 404 and a first size 405 can be determined according to at least one image 403 in the input information. A plurality of candidate parameters 407 can be determined according to the quantity restriction parameter 402 and the scale information 404. A third parameter 408 of the first image processing strategy can be determined according to the first size 405, the second size 406 of the image block, and the plurality of candidate parameters 407.

[0115] According to an embodiment of the present disclosure, a plurality of candidate parameters are determined based on a quantity limitation parameter and ratio information of an image, including: determining a plurality of pairs of initial row parameters and initial column parameters based on the quantity limitation parameter, wherein the product of each pair of initial row parameters and initial column parameters is an initial parameter; and screening out a plurality of candidate parameters from the plurality of initial parameters based on the ratio information.

[0116] The initial row parameter and the initial column parameter are used to indicate the initial row parameter and the initial column parameter respectively. When a certain initial parameter is determined as a candidate parameter, the initial row parameter and the initial column parameter are also the row parameter and the column parameter of the candidate parameter. For a two-dimensional image, the initial row parameter and the initial column parameter appear in pairs to determine the number of image blocks to be segmented from the row dimension and the column dimension.

[0117] For example, pairs of initial row parameters and initial column parameters can be randomly generated, traversing 1*1, 2*1, 2*2, 3*1...m*n, and ensuring that the product of each pair of initial row parameters and initial column parameters is less than or equal to the quantity limit parameter, that is, the initial parameters are less than or equal to the quantity limit parameter, such as m*n<= L, where L is the quantity limit parameter. Afterwards, the ratio of the initial row parameters to the initial column parameters corresponding to the initial parameters is calculated, and multiple candidate parameters are screened from multiple initial parameters based on the ratio of the initial row parameters to the initial column parameters and the proximity of the image's proportion information. For example, the numerical value of the ratio of the initial row parameters to the initial column parameters and the numerical value of the image's proportion information can be calculated, and the degree of proximity can be determined based on the difference between the two numerical values.

[0118] For example, multiple pairs of initial row parameters and initial column parameters may be 3*1, 3*2, 5*4, 6*4, respectively, and the multiple initial parameters are 3, 6, 20, and 24. When the ratio information of the image is 16:11, 3*2=6 and 6*4=24 may be determined as candidate parameters according to 16 / 11=1.45, 3 / 1=3, 3 / 2=1.5, 5 / 4=1.25, and 6 / 4=1.5, respectively.

[0119] In an embodiment of the present disclosure, a plurality of pairs of initial row parameters and initial column parameters are determined according to a quantity restriction parameter, wherein the product of each pair of initial row parameters and initial column parameters is an initial parameter; and a plurality of candidate parameters are screened out from the plurality of initial parameters according to the ratio information. Since the candidate parameters are dynamically determined based on the quantity restriction parameter and the ratio information and conform to the ratio information of the image in the input information, the third parameter determined from the candidate parameters can avoid losing valid pixels in the image during segmentation, thereby improving the accuracy and speed of subsequent content extraction of the input information and at least one image block.

[0120] Figure 5 The following is a schematic diagram showing a data flow diagram for determining candidate parameters according to an embodiment of the present disclosure. Figure 5 As shown, in the embodiment 500, a plurality of pairs of initial row parameters 502 and initial column parameters 503 are determined according to the quantity restriction parameter 501, and the product of each pair of initial row parameters 502 and initial column parameters 503 is an initial parameter 505. For each initial parameter 505, it can be determined whether to determine the initial parameter 505 as a candidate parameter 506 according to the ratio of the initial row parameter 502 to the initial column parameter 503 and the ratio information 504.

[0121] According to an embodiment of the present disclosure, a third parameter is determined from multiple candidate parameters based on a first size of an image and a second size of an image block to be segmented, including: determining an intermediate parameter for each candidate parameter based on the multiple candidate parameters and the second size; and determining the third parameter based on a comparison relationship between the intermediate parameter and the first size, and a comparison relationship between the multiple intermediate parameters.

[0122] The intermediate parameter is used to indicate the amount of information of the image contained in the multiple image blocks obtained by segmenting the image with the candidate parameter. The intermediate parameter can be the product of the candidate parameter and the second size. The larger the intermediate parameter is, the more information of the image is contained in the multiple image blocks obtained by segmenting the image with the candidate parameter. For example, the second size can be 224*224 and the candidate parameter is m*n, then the intermediate parameter = m*n*224*224.

[0123] The comparison relationship between the intermediate parameter and the first size may be a size comparison result between the intermediate parameter and the first size, or a size comparison result between the intermediate parameter and a predetermined multiple of the first size. For example, the first size may be M*N, and the size comparison result may be whether m*n*224*224 is smaller than M*N, or whether m*n*224*224 is smaller than M*N*2, etc.

[0124] The comparison relationship between the multiple intermediate parameters may be a size comparison of the multiple intermediate parameters, for example, 3*2*224*224<6*4*224*224<16*11*224*224.

[0125] For example, the largest intermediate parameter among multiple intermediate parameters that are smaller than twice the first size may be determined as the third parameter, such as m*n that maximizes m*n*224*224 and is smaller than M*N*2.

[0126] In the embodiments of the present disclosure, an intermediate parameter for each candidate parameter is determined based on multiple candidate parameters and the second size; a third parameter is determined based on a comparison relationship between the intermediate parameter and the first size and a comparison relationship between the multiple intermediate parameters. The third parameter that matches the first size of the image can be determined while maximizing the amount of information contained in the multiple image blocks obtained by segmenting the image. This not only achieves dynamic determination of the number of image blocks to be segmented, but also improves the accuracy and speed of subsequent content extraction of the input information and at least one image block.

[0127] Figure 6 The flowchart of updating at least one image block according to an embodiment of the present disclosure is schematically shown. Figure 6 As shown, embodiment 600 includes operations S610-S620.

[0128] In operation S610, a fourth parameter is determined according to an image processing strategy, wherein the fourth parameter is used to indicate the number of image blocks obtained by processing the image according to the image processing strategy.

[0129] For example, when the first image processing strategy includes the third parameter, the fourth parameter can be determined according to the number of first image processing strategies, the third parameter of each first image processing strategy, and the number of second image processing strategies. For example, if the third parameter of the first image processing strategy matching the i-th image is Ai, then the fourth parameter is: , B is the number of second image processing strategies, Ai+1 indicates the number of image blocks after the i-th image is processed according to the first image processing strategy, and A is the number of first image processing strategies.

[0130] Alternatively, when the first image processing strategy is used to indicate that the image is divided into a predetermined number of image blocks, the fourth parameter can be determined according to the number of the first image processing strategies and the number of the second image processing strategies, such as the fourth parameter is B+A(1+q), where q is the predetermined number.

[0131] In operation S620, the image block is updated according to a comparison relationship between the fourth parameter and the quantity limitation parameter.

[0132] When the fourth parameter is greater than the quantity limit parameter, it indicates that the large content extraction model cannot extract content from multiple image blocks simultaneously, and multiple operations are required to complete the content extraction from multiple image blocks, and the content extraction speed is low. Therefore, the number of image blocks can be reduced to obtain at least one updated image block.

[0133] When the fourth parameter is less than or equal to the quantity limit parameter, at least one image block may not be updated.

[0134] In the embodiment of the present disclosure, the fourth parameter is determined according to the image processing strategy; and the image block is updated according to the comparison relationship between the fourth parameter and the quantity limit parameter. Since the content extraction model compares the total number of at least one image block after segmentation with the quantity limit parameter before performing content extraction, it is possible to avoid the total number of at least one image block after segmentation exceeding the model token limit, thereby improving the content extraction speed.

[0135] According to an embodiment of the present disclosure, an image block is updated according to a comparison relationship between a fourth parameter and a quantity limit parameter, including: when the fourth parameter is greater than the quantity limit parameter, an image processing strategy is updated by adjusting at least one of a first threshold and a second threshold; and at least one image is processed based on the updated image processing strategy to obtain at least one updated image block.

[0136] For the scheme of determining the image processing strategy based on the first parameter, when the fourth parameter is greater than the quantity limit parameter, the image processing strategy matching at least one image is updated by lowering the first threshold; at least one image is processed according to the updated image processing strategy to obtain at least one updated image block, and the number of the at least one updated image block is smaller.

[0137] For the scheme of determining the image processing strategy according to the second parameter, when the fourth parameter is greater than the quantity limit parameter, the image processing strategy matching the at least one image is updated by increasing the second threshold; the at least one image is processed according to the updated image processing strategy to obtain at least one updated image block. Since the number of the first image processing strategies is reduced after the second threshold is increased, the number of image blocks is reduced.

[0138] Regarding the manner of determining the image processing strategy according to the first parameter and the second parameter, only the first threshold may be lowered or the threshold may be increased, or both the first threshold may be lowered and the threshold may be increased.

[0139] In an embodiment of the present disclosure, when the fourth parameter is greater than the quantity limit parameter, by adjusting at least one of the first threshold and the second threshold, the number of at least one updated image block is made smaller, while taking into account the amount of information of at least one image, thereby achieving an improvement in content extraction speed while improving content extraction accuracy.

[0140] According to an embodiment of the present disclosure, the method further includes: when the number of fourth parameters is greater than the number limit parameter, screening the image blocks according to the fifth parameter of each image block to obtain updated image blocks, wherein the fifth parameter is used to indicate the semantic richness of the image blocks.

[0141] When the number of the fourth parameter is greater than the number limit parameter, object detection can be performed on each image block to obtain an object detection result, which includes at least one of the number of objects and the type of objects. The fifth parameter of each image block is determined according to the object detection result.

[0142] According to the comparison relationship between the fifth parameter of each image block and the third threshold, it is determined whether to retain the image block. The third threshold has a similar meaning to the second threshold and is not described here. Optionally, the third threshold is less than the second threshold and can be determined based on an empirical value.

[0143] Screening at least one image block according to the fifth parameter of each image block may be: screening out image blocks whose fifth parameters are less than a third threshold from the at least one image block to obtain the at least one updated image block.

[0144] In an embodiment of the present disclosure, when the number of the fourth parameters is greater than the number limit parameter, the image blocks are screened according to the fifth parameter of each image block to obtain updated image blocks, so that the number of at least one updated image block is smaller, thereby improving the content extraction speed.

[0145] According to an embodiment of the present disclosure, a large model is extracted from input information and image block input content to generate output information, including: using image blocks to replace at least one image in the input information to obtain updated input information; and extracting a large model from the updated input information input content to generate output information.

[0146] For at least one image in the input information, after processing the at least one image into at least one image block, the at least one image block can be directly used to replace the original at least one image in the input information to obtain updated input information. After pre-processing operations such as image processing strategy determination, image processing, and input information update, the content extraction model can be called to perform content extraction tasks according to the updated input information to generate output information.

[0147] When the input information only includes at least one image, the content extraction model can directly extract content based on the processed at least one image block. When the input information includes text and at least one image, the content extraction model can be called to first extract content from the at least one image block to obtain image content information, and then the image content information and text are merged to obtain output information.

[0148] For the video information input by the user, the video information includes a large amount of redundant information, and at least one video frame can be extracted from the video information to form the input information.

[0149] In an embodiment of the present disclosure, at least one image in input information is replaced with at least one image block to obtain updated input information, and the updated input information is input into a large content extraction model. This allows content extraction of single-image / multi-image tasks to be achieved through the same content extraction time without changing the token restriction of the large content extraction model, and allows extraction of more fine-grained image features, thereby achieving the technical effect of being compatible with multiple image tasks, increasing content extraction speed, and increasing content extraction accuracy.

[0150] For example, the input information can be: You are an assistant who is good at summarizing image information. Please describe the main content of the image. You need to give XXX. The images are: Image 1~3.

[0151] According to the "image 1~3" in the input information, the image processing strategy matching "image 1~3" is determined, and "image 1~3" is processed into "image block 1~5" according to the image processing strategy. After the above image processing, "image block 1~5" can replace "image 1~3" and form the updated input information with "You are an assistant who is good at summarizing image information. Please describe the main content of the image. You need to give XXX". After extracting the large model from the updated input information input content, the output information matching the user's needs can be generated, such as the output information can be: a brown puppy in the image is playing happily, and the breed of the puppy is YY.

[0152] In addition, the input information can also include a variety of user needs, such as "need to give XXX" can be required to give the key figures, main events, main viewpoints, main steps or actions, there are text content to be extracted, please summarize it in one paragraph. The title is: BT.

[0153] According to another embodiment of the present disclosure, after at least one image block and input information are input into the content extraction model, the image encoder of the content extraction model can further reduce the number of image blocks based on a pixel rearrangement strategy, that is, reduce the number of tokens, so as to further improve the content extraction speed.

[0154] For example, after at least one image block is input into the image encoder, the image encoder decomposes each channel of the input image block into multiple sub-channels, and then rearranges the pixels of the multiple sub-channels to form a new channel. In this process, the total number of pixels of the original image block remains unchanged. Finally, the rearranged sub-channels are merged into a new image block. Note that the image block input to the image encoder here is also the feature map.

[0155] According to the embodiments of the present disclosure, a large model can be trained using a public data set, such as LAION-multi. The training process has three stages: visual-language contrast training, visual-language generation training, and supervised fine-tuning. During the supervised fine-tuning process, pre-annotations can be generated by the labeled large model, and then manually rewritten to obtain incremental data for model fine-tuning for supervised fine-tuning. The labeled large model can be a variety of leading models constructed based on the large language model, such as GPT4o (Generative Pre-trained Transformer), etc.

[0156] For ease of understanding, the following will be Figure 7 Take the content extraction scenario as an example to describe the content extraction scenario.

[0157] Figure 7 FIG. 1 is a schematic diagram of a content extraction framework according to an embodiment of the present disclosure. Figure 7 As shown, in embodiment 700, the user can input various information such as text 701, image 702, video 703, etc. For image 702, it can be multiple article pictures 704. For video 703, multiple video frames 705 can be obtained by extracting frames. In this embodiment, multiple article pictures 704 and multiple video frames 705 can be regarded as multiple images in the above input information.

[0158] In the embodiment 700, the content extraction model M10 may include a multimodal model M11 and a large language model M12. The content extraction model M10 may also include a preprocessing module for determining an image processing strategy matching at least one image according to at least one image in the input information; and processing at least one image according to the image processing strategy to obtain at least one image block.

[0159] The preprocessing module can process the multiple article pictures 704 and the multiple video frames 705 to obtain multiple image blocks. Then, the multimodal large model M11 extracts the content of the multiple image blocks to obtain multimodal information 706, which is used to indicate the image content information included in the multiple article pictures 704 and the multiple video frames 705.

[0160] The large language model M12 fuses and summarizes the text 701 and the multimodal information 706 to generate output information.

[0161] Figure 8 The block diagram of the content extraction device according to the embodiment of the present disclosure is schematically shown. Figure 8 As shown, the content extraction device 800 includes: a determination module 810 , a processing module 820 , and a content extraction module 830 .

[0162] The determination module 810 is used to determine an image processing strategy that matches the image according to at least one image in the input information.

[0163] The processing module 820 is used to process the image according to the image processing strategy to obtain at least one image block, wherein the first size of the image is larger than the second size of the image block, and the number of the image blocks is determined according to the image processing strategy.

[0164] The content extraction module 830 is used to input the input information and the image block into the content extraction model to obtain output information.

[0165] According to an embodiment of the present disclosure, the determination module 810 includes: a determination submodule, used to determine an image processing strategy based on an information volume parameter; wherein the image processing strategy includes a first image processing strategy and a second image processing strategy, the first image processing strategy is used to instruct to segment and reduce the image to obtain image blocks; the second image processing strategy is used to instruct to reduce the image to obtain image blocks.

[0166] According to an embodiment of the present disclosure, the information volume parameter includes a first parameter, and the first parameter is used to indicate the number of images in the input information; the determination submodule includes: a first determination unit and a second determination unit.

[0167] The first determining unit is configured to determine a first image processing strategy when a first parameter is less than or equal to a first threshold.

[0168] The second determining unit is configured to determine a second image processing strategy when the first parameter is greater than the first threshold.

[0169] According to an embodiment of the present disclosure, the information volume parameter includes a second parameter of each image, and the second parameter is used to indicate the semantic richness of the image. The determination submodule includes: a third determination unit and a fourth determination unit. For each image,

[0170] The third determining unit is configured to determine a second image processing strategy that matches the image when the second parameter is less than or equal to the second threshold.

[0171] The fourth determining unit is used to determine a first image processing strategy that matches the image when the second parameter is greater than a second threshold.

[0172] According to an embodiment of the present disclosure, the determination submodule further includes: a detection unit and a parameter determination unit.

[0173] The detection unit is used to perform object detection on the image to obtain an object detection result, wherein the object detection result includes at least one of the number of objects and the type of objects in the image.

[0174] The parameter determination unit is used to determine the second parameter according to the object detection result.

[0175] According to an embodiment of the present disclosure, the information volume parameter further includes a second parameter, and the second parameter is used to indicate the semantic richness of the image; the determination submodule further includes: a fifth determination unit and a sixth determination unit.

[0176] The fifth determining unit is used to determine the second image processing strategy when the first parameter is greater than the first threshold and the second parameter of the image is less than or equal to the second threshold.

[0177] The sixth determining unit is used to determine the first image processing strategy when the first parameter is greater than the first threshold and the second parameter of the image is greater than the second threshold.

[0178] According to an embodiment of the present disclosure, the first image processing strategy includes a third parameter, and the third parameter is used to indicate the number of image blocks obtained by segmenting the image; the determination submodule also includes: a seventh determination unit and an eighth determination unit.

[0179] The seventh determination unit is used to determine multiple candidate parameters according to the quantity limitation parameter and the ratio information of the image, wherein the quantity limitation parameter is determined according to the architecture of the content extraction large model.

[0180] The eighth determination unit is used to determine a third parameter from a plurality of candidate parameters according to the first size of the image and the second size of the image block to be segmented.

[0181] According to an embodiment of the present disclosure, the seventh determining unit includes: a first determining subunit and a screening subunit.

[0182] A first determining subunit is used to determine a plurality of pairs of initial row parameters and initial column parameters according to a quantity restriction parameter, wherein the product of each pair of initial row parameters and initial column parameters is an initial parameter; and

[0183] The screening subunit is used to screen out a plurality of candidate parameters from a plurality of initial parameters according to the ratio information.

[0184] According to an embodiment of the present disclosure, the eighth determining unit includes: a second determining subunit and a third determining subunit.

[0185] The second determining subunit is configured to determine an intermediate parameter for each candidate parameter according to the plurality of candidate parameters and the second size.

[0186] The third determining subunit is used to determine the third parameter according to the comparison relationship between the intermediate parameter and the first size and the comparison relationship between the plurality of intermediate parameters.

[0187] According to an embodiment of the present disclosure, the content extraction device 800 further includes: a parameter determination module and an update module.

[0188] The parameter determination module is used to determine a fourth parameter according to an image processing strategy, wherein the fourth parameter is used to indicate the number of image blocks obtained by processing the image according to the image processing strategy.

[0189] The updating module is used to update the image block according to the comparison relationship between the fourth parameter and the quantity restriction parameter.

[0190] According to an embodiment of the present disclosure, the update module includes: a first update sub-module, used to update the image processing strategy that matches the image by adjusting the first threshold and / or the second threshold when the fourth parameter is greater than the quantity limit parameter; a processing sub-module, used to process the image based on the updated image processing strategy to obtain an updated image block.

[0191] According to an embodiment of the present disclosure, the update module also includes: a second update submodule, which is used to screen the image blocks according to the fifth parameter of each image block to obtain updated image blocks when the number of the fourth parameter is greater than the number limit parameter, wherein the fifth parameter is used to indicate the semantic richness of the image block.

[0192] According to an embodiment of the present disclosure, the content extraction module 830 includes: a third updating submodule and a generating submodule. The third updating submodule is used to replace at least one image in the input information with an image block to obtain updated input information. The generating submodule is used to input the updated input information into the content extraction macro model to generate output information.

[0193] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0194] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the above method.

[0195] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to enable a computer to execute the above method.

[0196] According to an embodiment of the present disclosure, a computer program product includes a computer program, and the computer program implements the above method when executed by a processor.

[0197] Fig. 9 The structural block diagram of an intelligent agent according to an embodiment of the present disclosure is schematically shown.

[0198] In the embodiments of the present disclosure, inspired by the von Neumann structure in modern computer theory, such as Fig. 9 As shown, the AI ​​agent 900 may include three core modules: an input module 910 , a processing module 920 , and an output module 930 .

[0199] The input module 910 is responsible for receiving or sensing information such as queries, requests, instructions, signals or data from the outside world (such as users or external environment), and converting them into a format that can be understood and processed by the AI ​​agent 900. The input module 910 is the primary link for the AI ​​agent 900 to interact with the outside world. It enables the AI ​​agent 900 to efficiently and accurately obtain the necessary "sensory" information from the outside world and respond to this information.

[0200] In an example, the input module 910 can input the input information described above, and can input images, texts, videos, and the like.

[0201] In the example, the processing module 920 is the core support for the AI ​​agent 900 to process complex tasks. The processing module 920 is used to determine the target task based on the input information received by the input module 910, determine the large model based on the target task, and execute the content extraction method described above by calling the large model.

[0202] In an example, the output module 930 may output the output information described above.

[0203] In an example, the processing module 920 may include a control unit 921 , a storage unit 922 , and a computing unit 923 .

[0204] During operation, the control unit 921 will continuously interact with the storage unit 922, the computing unit 923 and / or the output module 930. However, it should be noted that in the embodiment of the present disclosure, the control unit 921 acts as a single initiator to initiate communication with the storage unit 922, the computing unit 923 and / or the output module 930, and there is no communication coupling between the storage unit 922, the computing unit 923 and the output module 930.

[0205] In the example, the performance of the control unit 921 may be closely related to the large model on which the AI ​​agent 900 is based. In order to give full play to the capabilities of the large model, the internal structure of the control unit 921 may be designed to be highly configurable and scalable so as to cope with various types of tasks and requirements in real scenarios.

[0206] The storage unit 922 may be responsible for memorizing information such as historical conversations, event flows, etc. The input information, images, image blocks, image processing strategies, etc. as described above may be included in the storage unit 922 .

[0207] The operation unit 923 may be regarded as a predefined tool library. As before, the image processing control for processing the image to obtain the image block may be included in the operation unit 923.

[0208] In the example, after the AI ​​agent 900 obtains the input information, the AI ​​agent 900 can use the input information to determine the target task and determine the large model based on the target task. The input information can be stored in the storage unit 922 of the processing module 920. The control unit 921 can call the large model to obtain the input information from the storage unit 922 and determine the image processing strategy that matches the at least one image based on at least one image in the input information, and store the image processing strategy in the storage unit 922. The control unit 921 calls the image processing control in the operation module 923 to process at least one image according to the image processing strategy, obtain at least one image block, and store it in the storage unit 922. The control unit 921 can obtain at least one image block and input information from the storage unit 922, and extract the large model from the input information and at least one image block input content to generate output information. Then, the control unit 921 transmits the output information to the output module 930.

[0209] The AI ​​agent 900 according to the embodiment of the present disclosure can simply and effectively improve the level of intelligence, and improve flexibility and versatility.

[0210] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.

[0211] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method as described above.

[0212] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method as described above.

[0213] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method as described above.

[0214] Fig.10 A block diagram of an electronic device suitable for implementing a content extraction method according to an embodiment of the present disclosure is schematically shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.

[0215] like Fig.10 As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. In the RAM 1003, various programs and data required for the operation of the electronic device 1000 can also be stored. The computing unit 1001, the ROM 1002, and the RAM 1003 are connected to each other via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0216] Multiple components in the electronic device 1000 are connected to an input / output (I / O) interface 1005, including: an input unit 1006, such as a keyboard, a mouse, etc.; an output unit 1007, such as various types of displays, speakers, etc.; a storage unit 1008, such as a disk, an optical disk, etc.; and a communication unit 1009, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 1009 allows the electronic device 1000 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.

[0217] The computing unit 1001 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 1001 performs the various methods and processes described above, such as the content extraction method. For example, in some embodiments, the content extraction method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 1008. In some embodiments, part or all of the computer program may be loaded and / or installed on the electronic device 1000 via the ROM 1002 and / or the communication unit 1009. When the computer program is loaded into the RAM 1003 and executed by the computing unit 1001, one or more steps of the content extraction method described above may be performed. Alternatively, in other embodiments, the computing unit 1001 may be configured to perform the content extraction method in any other appropriate manner (e.g., by means of firmware).

[0218] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0219] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.

[0220] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or device, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0221] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0222] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.

[0223] A computer system may include a client and a server. The client and the server are generally remote from each other and typically interact through a communication network. The relationship of client and server arises through computer programs running on respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server combined with a blockchain.

[0224] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solution of this disclosure can be achieved, and this document is not limited here.

[0225] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.

Claims

1. A content extraction method, comprising: Determining, based on at least one image in the input information, an image processing strategy that matches the image; Processing the image according to the image processing strategy to obtain at least one image block, wherein a first size of the image is greater than a second size of the image block, and the number of the image blocks is determined according to the image processing strategy; The input information and the image block input content are extracted into a large model to obtain output information.

2. The method according to claim 1, wherein: The step of determining, based on at least one image in the input information, an image processing strategy that matches the image comprises: Determining the image processing strategy according to the information volume parameter of the image; The image processing strategy includes a first image processing strategy and a second image processing strategy, wherein the first image processing strategy is used to instruct the image to be segmented and reduced to obtain the image block; and the second image processing strategy is used to instruct the image to be reduced to obtain the image block.

3. The method according to claim 2, wherein: The information volume parameter includes a first parameter, and the first parameter is used to indicate the number of the images in the input information; Determining the image processing strategy according to the information volume parameter of the image includes: When the first parameter is less than or equal to a first threshold, determining the first image processing strategy; When the first parameter is greater than the first threshold, the second image processing strategy is determined.

4. The method according to claim 2, wherein: The information volume parameter includes a second parameter of each of the images, where the second parameter is used to indicate the semantic richness of the image; Determining the image processing strategy according to the information volume parameter of the image includes: For each of the images, When the second parameter is less than or equal to a second threshold, determining the second image processing strategy that matches the image; When the second parameter is greater than the second threshold, the first image processing strategy matching the image is determined.

5. The method according to claim 4, further comprising: Performing object detection on the image to obtain an object detection result, wherein the object detection result includes at least one of the number of objects and the type of objects in the image; and The second parameter is determined according to the object detection result.

6. The method according to claim 3, wherein: The information volume parameter further includes a second parameter, and the second parameter is used to indicate the semantic richness of the image; The method further comprises: When the first parameter is greater than the first threshold and the second parameter is less than or equal to a second threshold, determining the second image processing strategy; When the first parameter is greater than the first threshold and the second parameter is greater than a second threshold, the first image processing strategy is determined.

7. The method according to any one of claims 1 to 6, wherein: The first image processing strategy includes a third parameter, and the third parameter is used to indicate the number of the image blocks obtained by segmenting the image; The method further comprises: Determining a plurality of candidate parameters according to a quantity restriction parameter and the scale information of the image, wherein the quantity restriction parameter is determined according to the architecture of the content extraction large model; and The third parameter is determined from a plurality of candidate parameters according to a first size of the image and a second size of the image block to be segmented.

8. The method according to claim 7, wherein: The determining of a plurality of candidate parameters according to the quantity restriction parameter and the ratio information of the image includes: Determine a plurality of pairs of initial row parameters and initial column parameters according to the quantity restriction parameter, wherein the product of each pair of the initial row parameter and the initial column parameter is an initial parameter; and According to the ratio information, a plurality of candidate parameters are screened out from the plurality of initial parameters.

9. The method according to claim 7 or 8, wherein: The determining the third parameter from a plurality of candidate parameters according to the first size of the image and the second size of the image block to be segmented comprises: Determine an intermediate parameter for each of the candidate parameters according to the plurality of candidate parameters and the second size; and The third parameter is determined according to a comparison relationship between the intermediate parameter and the first size and a comparison relationship between a plurality of the intermediate parameters.

10. The method according to any one of claims 1 to 9, further comprising: determining a fourth parameter according to the image processing strategy, wherein the fourth parameter is used to indicate the number of the image blocks obtained by processing the image according to the image processing strategy; and The image block is updated according to a comparison relationship between the fourth parameter and the quantity limit parameter.

11. The method according to claim 10, wherein: The updating of the image block according to the comparison relationship between the fourth parameter and the quantity limit parameter includes: When the fourth parameter is greater than the quantity limit parameter, updating the image processing strategy by adjusting at least one of the first threshold and the second threshold; At least one of the images is processed based on the updated image processing strategy to obtain at least one updated image block.

12. The method according to claim 10, further comprising: In a case where the number of the fourth parameters is greater than the number limit parameter, the image blocks are screened according to the fifth parameter of each image block to obtain updated image blocks, wherein the fifth parameter is used to indicate the semantic richness of the image block.

13. The method according to any one of claims 1 to 12, wherein: The step of extracting a large model from the input information and the image block input content to obtain output information includes: replacing at least one image in the input information with the image block to obtain updated input information; and The updated input information is input into the content extraction model to obtain the output information.

14. A content extraction device, comprising: A determination module, used to determine an image processing strategy matching the image according to at least one image in the input information; a processing module, configured to process the image according to the image processing strategy to obtain at least one image block, wherein the first size of the image is greater than the second size of the image block, and the number of the image blocks is determined according to the image processing strategy; The content extraction module is used to input the input information and the image block into a content extraction model to obtain output information.

15. The device according to claim 14, wherein: The determination module comprises: A determination submodule, used to determine the image processing strategy according to the information volume parameter of the image; The image processing strategy includes a first image processing strategy and a second image processing strategy, wherein the first image processing strategy is used to instruct the image to be segmented and reduced to obtain the image block; and the second image processing strategy is used to instruct the image to be reduced to obtain the image block.

16. The device according to claim 15, wherein: The information volume parameter includes a first parameter, and the first parameter is used to indicate the number of the images in the input information; The determination submodule comprises: A first determining unit, configured to determine the first image processing strategy when the first parameter is less than or equal to a first threshold; The second determining unit is configured to determine the second image processing strategy when the first parameter is greater than the first threshold.

17. The device according to claim 15, wherein: The information volume parameter includes a second parameter of each of the images, and the second parameter is used to indicate the semantic richness of the image; the determination submodule includes: For each of the images, a third determining unit, configured to determine the second image processing strategy matching the image when the second parameter is less than or equal to a second threshold; A fourth determining unit is used to determine the first image processing strategy that matches the image when the second parameter is greater than the second threshold.

18. The apparatus according to claim 17, wherein the determining submodule further comprises: a detection unit, configured to perform object detection on the image to obtain an object detection result, wherein the object detection result includes at least one of the number of objects and the type of objects in the image; and A parameter determination unit is used to determine the second parameter according to the object detection result.

19. The device according to claim 16, wherein: The information volume parameter further includes a second parameter, and the second parameter is used to indicate the semantic richness of the image; the determination submodule further includes: a fifth determining unit, configured to determine the second image processing strategy when the first parameter is greater than the first threshold and the second parameter is less than or equal to a second threshold; A sixth determining unit is used to determine the first image processing strategy when the first parameter is greater than the first threshold and the second parameter is greater than a second threshold.

20. The device according to any one of claims 14 to 19, wherein: The first image processing strategy includes a third parameter, and the third parameter is used to indicate the number of the image blocks obtained by segmenting the image; The determining submodule further includes: a seventh determining unit, configured to determine a plurality of candidate parameters according to a quantity restriction parameter and the scale information of the image, wherein the quantity restriction parameter is determined according to the architecture of the content extraction macro model; and An eighth determining unit is used to determine the third parameter from a plurality of the candidate parameters according to the first size of the image and the second size of the image block to be segmented.

21. The device according to claim 20, wherein: The seventh determining unit comprises: a first determining subunit, configured to determine a plurality of pairs of initial row parameters and initial column parameters according to the quantity restriction parameter, wherein the product of each pair of the initial row parameter and the initial column parameter is an initial parameter; and The screening subunit is used to screen out a plurality of candidate parameters from the plurality of initial parameters according to the ratio information.

22. The device according to claim 20 or 21, wherein: The eighth determining unit includes: A second determining subunit, configured to determine an intermediate parameter for each of the candidate parameters according to the plurality of candidate parameters and the second size; and The third determining subunit is used to determine the third parameter according to the comparison relationship between the intermediate parameter and the first size and the comparison relationship between the plurality of intermediate parameters.

23. The device according to any one of claims 14 to 22, further comprising: a parameter determination module, configured to determine a fourth parameter according to the image processing strategy, wherein the fourth parameter is used to indicate the number of image blocks obtained by processing the image according to the image processing strategy; and An updating module is used to update the image block according to a comparison relationship between the fourth parameter and the quantity limitation parameter.

24. The device according to claim 23, wherein: The update module includes: A first updating submodule, configured to update an image processing strategy matching the image by adjusting at least one of the first threshold and the second threshold when the fourth parameter is greater than the quantity limit parameter; The image is processed based on the updated image processing strategy to obtain the updated image block.

25. The apparatus according to claim 23, wherein the updating module further comprises: The second updating submodule is used to screen the image blocks according to the fifth parameter of each image block to obtain the updated image blocks when the number of the fourth parameters is greater than the number limit parameter, wherein the fifth parameter is used to indicate the semantic richness of the image blocks.

26. The device according to any one of claims 14 to 25, wherein: The content extraction module comprises: A third updating submodule, configured to replace at least one image in the input information with the image block to obtain updated input information; and The generation submodule is used to input the updated input information into the content extraction model to obtain the output information.

27. An intelligent agent, comprising: An input module, used for receiving input information; a processing module, configured to determine a target task based on the input information received by the input module, determine a large model based on the target task, and obtain output information by calling the large model to execute the method according to any one of claims 1 to 13; An output module is used to output the output information obtained by the processing module.

28. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can execute the method according to any one of claims 1 to 13.

29. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to enable the computer to execute the method according to any one of claims 1 to 13.

30. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the computer program implements the method according to any one of claims 1 to 13.

Images