Image generation method and device based on large model, electronic equipment and storage medium

Abstract

The invention provides an image generation method and device based on a large model, electronic equipment and a storage medium, and relates to the technical field of artificial intelligence, in particular to the technical field of computer vision and large models. Generating a position of at least one object in the initial image; based on the position, at least one target image is extracted from the initial image, and the target image comprises a single object; processing the attributes of the at least one target image and the at least one object by using a second large model, and generating a corresponding relationship between the target image and the attributes; wherein the attribute is obtained by processing the initial image by using a second large model; and according to the position and the corresponding relationship, marking at least one object in the initial image to generate a first marked image.

Description

Technical Field

[0001] The present disclosure relates to the field of artificial intelligence technology, in particular to the field of computer vision and large-scale model technology, and specifically to a large-scale model-based image generation method, device, electronic device, and storage medium. Background Art

[0002] With the development of deep learning technology, large models based on neural networks are being used more and more widely in the field of computer vision. For example, in the field of autonomous driving, obstacles can be identified under complex and changing road conditions to ensure safe driving; in the medical field, medical images can be identified to assist doctors in early diagnosis of diseases.

[0003] When large models are deeply applied in the field of automated control, such as intelligent robotic arms, the visual recognition capabilities of large models are still difficult to meet user needs. Summary of the Invention

[0004] The present disclosure provides a large model-based image generation method, device, electronic device, and storage medium.

[0005] According to one aspect of the present disclosure, a large model-based image generation method is provided, comprising: segmenting an initial image using a first large model to generate the position of at least one object in the initial image; extracting at least one target image from the initial image based on the position, wherein the target image includes a single object; processing the at least one target image and the attributes of at least one object using a second large model to generate a correspondence between the target image and the attributes; wherein the attributes are obtained by processing the initial image using the second large model; and generating a first labeled image by marking at least one object in the initial image based on the position and the correspondence.

[0006] According to another aspect of the present disclosure, a large model-based image generation device is provided, including: a first generation module, a first extraction module, a second generation module and a first labeling module.

[0007] The first generation module is configured to segment the initial image using the first large model and generate the position of at least one object in the initial image. The first extraction module is configured to extract at least one target image from the initial image based on the position, wherein the target image includes a single object. The second generation module is configured to process the at least one target image and the attributes of at least one object using the second large model to generate a correspondence between the target image and the attributes; wherein the attributes are obtained by processing the initial image using the second large model. The first labeling module is configured to generate a first labeled image by labeling at least one object in the initial image based on the position and the correspondence.

[0008] According to another aspect of the present disclosure, an electronic device is provided, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0009] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute the method described above.

[0010] According to another aspect of the present disclosure, a computer program product is provided, comprising a computer program, wherein the computer program implements the method described above when executed by a processor.

[0011] It should be understood that the contents described in this section are not intended to identify the key or important features of the embodiments of the present disclosure, nor are they intended to limit the scope of the present disclosure. Other features of the present disclosure will become readily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] The accompanying drawings are used to better understand the present invention and do not constitute a limitation of the present invention.

[0013] Figure 1 Schematically illustrates an exemplary system architecture to which a large model-based image generation method and apparatus according to an embodiment of the present disclosure can be applied;

[0014] Figure 2 Schematically shows a flow chart of a method for generating an image based on a large model according to an embodiment of the present disclosure;

[0015] Figure 3 Schematically illustrates a schematic diagram of processing an image based on a large model to generate a first labeled image according to an embodiment of the present disclosure;

[0016] Figure 4 Schematically illustrating a schematic diagram of processing an operation interface image using a large model based on a user instruction to generate a second marked image according to an embodiment of the present disclosure;

[0017] Figure 5 Schematically illustrates a schematic diagram of processing an operation interface image using a large model based on a user voice instruction to generate a second marked image according to an embodiment of the present disclosure;

[0018] Figure 6 The interactive diagram of the large model-based image generation method according to an embodiment of the present disclosure is schematically shown;

[0019] Figure 7A block diagram schematically shows an image generation apparatus based on a large model according to an embodiment of the present disclosure; and

[0020] Figure 8 A block diagram of an electronic device suitable for implementing a large model-based image generation method according to an embodiment of the present disclosure is schematically shown. DETAILED DESCRIPTION

[0021] The following description of exemplary embodiments of the present disclosure is made in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding. These details should be considered as merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, descriptions of well-known functions and structures are omitted in the following description.

[0022] A large visual model refers to a large-scale visual processing model based on deep learning, which can extract useful features and information from visual data such as images and videos. Therefore, it is widely used in many fields such as medical care, transportation, security, and intelligent manufacturing.

[0023] For example, in defect detection scenarios in the field of intelligent manufacturing, related examples generally use trained target detection models to identify and mark defects. However, the recognition and positioning accuracy of target detection models is limited by the diversity of training data sets, and it is difficult to accurately identify defects that are significantly different from sample data.

[0024] For example, in the intelligent robotic arm operation scenario, it is difficult for the intelligent robotic arm to accurately identify the positions of different control buttons when faced with the complex and changing content in the visual operation interface, resulting in the robotic arm's operation objects being currently limited to physical objects.

[0025] In view of this, the embodiments of the present disclosure provide an image generation method based on a large model. This method utilizes the natural language understanding capabilities of the large model to accurately identify each object in the image without pre-training, thereby reducing the impact of the training data set on recognition accuracy. At the same time, the semantic segmentation capabilities of the large model are used to semantically segment the image to obtain an image containing only a single object, further improving recognition accuracy and positioning accuracy. Finally, the positioning information obtained based on semantic segmentation is marked in the initial image, further improving the visual focusing ability of the large model, meeting user needs, and improving the user experience.

[0026] Figure 1 An exemplary system architecture to which the large model-based image generation method and apparatus according to an embodiment of the present disclosure can be applied is schematically shown.

[0027] It should be noted that Figure 1The examples shown are merely examples of system architectures to which the embodiments of the present disclosure may be applied, to help those skilled in the art understand the technical content of the present disclosure, but do not imply that the embodiments of the present disclosure may not be applied to other devices, systems, environments, or scenarios. For example, in another embodiment, an exemplary system architecture to which the large-model-based image generation method and apparatus may be applied may include a terminal device, but the terminal device may implement the large-model-based image generation method and apparatus provided by the embodiments of the present disclosure without interacting with a server.

[0028] like Figure 1 As shown, the system architecture 100 according to this embodiment may include terminal devices 101, 102, 103, a network 104, and a server 105. The network 104 is used as a medium for providing communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired and / or wireless communication links, etc.

[0029] Users can use terminal devices 101, 102, and 103 to interact with server 105 via network 104 to receive or send messages, etc. Various communication client applications can be installed on terminal devices 101, 102, and 103, such as knowledge reading applications, web browser applications, search applications, instant messaging tools, email clients, and / or social platform software (for example only).

[0030] The terminal devices 101 , 102 , and 103 may be various electronic devices having a display screen and supporting web browsing, including but not limited to smart phones, tablet computers, laptop computers, and desktop computers.

[0031] Server 105 may be a server that provides various services, such as a background management server (for example only) that supports content browsed by users using terminal devices 101, 102, and 103. The background management server may analyze and process received data such as user requests, and feed back processing results (e.g., web pages, information, or data obtained or generated based on user requests) to the terminal device.

[0032] It should be noted that the large model-based image generation method provided in the embodiment of the present disclosure can generally be executed by the terminal device 101, 102, or 103. Accordingly, the large model-based image generation apparatus provided in the embodiment of the present disclosure can also be set in the terminal device 101, 102, or 103.

[0033] Alternatively, the image generation method based on the large model provided in the embodiment of the present disclosure may also be generally executed by the server 105. Accordingly, the image generation apparatus based on the large model provided in the embodiment of the present disclosure may generally be provided in the server 105. The image generation method based on the large model provided in the embodiment of the present disclosure may also be performed by a server or server cluster that is different from the server 105 and that is capable of communicating with the terminal devices 101, 102, 103 and / or the server 105. Accordingly, the image generation apparatus based on the large model provided in the embodiment of the present disclosure may also be provided in a server or server cluster that is different from the server 105 and that is capable of communicating with the terminal devices 101, 102, 103 and / or the server 105.

[0034] For example, a user can enter a tagging instruction via voice or text into the client, along with the image to be tagged. The client then sends the tagging instruction and the image to be tagged to the server. The server then calls the large model and executes the methods provided by the disclosed embodiments to generate a tagged image that marks the areas of the image that match the user's tagging instruction. Finally, the tagged image is returned to the client for display to the user.

[0035] It should be understood that Figure 1 The number of terminal devices, networks and servers in the embodiment is merely illustrative. Any number of terminal devices, networks and servers may be provided as required.

[0036] In the technical solution disclosed herein, the collection, storage, use, processing, transmission, provision, disclosure and application of user personal information involved comply with the provisions of relevant laws and regulations, take necessary confidentiality measures, and do not violate public order and good morals.

[0037] In the technical solution of the present disclosure, the user's authorization or consent is obtained before obtaining or collecting the user's personal information. The images used in the examples are all from public image data sources.

[0038] Figure 2 The flowchart of the large model-based image generation method according to an embodiment of the present disclosure is schematically shown.

[0039] like Figure 2 As shown, the method 200 includes operations S210 to S240.

[0040] In operation S210 , an initial image is segmented using a first large model to generate a position of at least one object in the initial image.

[0041] In operation S220, at least one target image is extracted from the initial image based on the location, wherein the target image includes a single object.

[0042] In operation S230, at least one target image and attributes of at least one object are processed using the second large model to generate a correspondence between the target image and the attributes.

[0043] In operation S240, a first marked image is generated by marking at least one object in the initial image according to the position and the correspondence.

[0044] According to embodiments of the present disclosure, the initial image can be a photograph of a physical object in a real environment, or a screenshot of a visualization interface. The object can be a foreground object in the photograph, such as an animal, plant, scenery, person, or building. It can also be a page control icon or text in the visualization interface.

[0045] According to an embodiment of the present disclosure, the first large model can be a large visual segmentation model. For example, the large visual segmentation model can be used to perform semantic segmentation on the initial image, and the position of each object in the initial image can be represented as a mask array. The number of elements in the mask array can be determined based on the size of the initial image. For example, the number of elements in the mask array corresponding to a 3x3 initial image is 9. Each element value can represent the pixel value at a different position in the initial image.

[0046] According to an embodiment of the present disclosure, the attributes of each object in the initial image are obtained by processing the initial image using a second large model. The second large model can be a visual perception large model. The attributes of the object include, but are not limited to, the object's name, color, text content, font, font size, etc.

[0047] For example, the initial image may be a photo of an animal. The visual perception model can be used to identify the specific names of the animals in the photo, such as cat, dog, lion, horse, etc.

[0048] According to an embodiment of the present disclosure, a target image can be extracted from an initial image based on its position by calling an Open CV function. The target image includes a single object. For example, the target images extracted from the aforementioned animal photo may include: an image containing only cats, an image containing only dogs, an image containing only lions, and an image containing only horses.

[0049] By performing semantic segmentation on the initial image, not only can the positioning accuracy of the object be improved, but also the accurate labeling of each object in the initial image can be achieved.

[0050] According to an embodiment of the present disclosure, a Prompt may be constructed based on the attributes of each target image and each object, so as to utilize the second largest model to process the attributes of at least one target image and at least one object and generate a correspondence between the target image and the attributes.

[0051] For example, the prompt could be "Please identify the object in the image. The object must be in the following list. If not, return [No]. The list is: [cat], [dog], [lion], [horse]; [image to be identified]." The [image to be identified] can be any target image extracted above.

[0052] This prompt limits the visual perception model to identifying only the target image from the list of objects, further improving object recognition accuracy. Because the target image has a one-to-one correspondence with the position of each object in the initial image, the correspondence between the target image and its attributes obtained using the second large model also reflects the correspondence between each object's attributes and its position in the initial image.

[0053] According to an embodiment of the present disclosure, a first marked image is generated by marking at least one object in the initial image according to the position of each object in the initial image, i.e., the mask array, and the correspondence between each target image and each object attribute.

[0054] For example, each object in the initial image may be marked using a rectangular frame or graphics of any other geometric shape to generate a first marked image, so that other applications can perform specific tasks based on the first marked image.

[0055] For example, in the operation scenario of an intelligent robotic arm, by accurately marking and identifying the attributes of each object, the marked image can be sent to the robotic arm, so that the robotic arm can read the marked position and accurately control the robotic arm's touch-screen visual operation interface, further improving the intelligence level of the robotic arm.

[0056] According to the embodiments of the present disclosure, the natural language understanding capabilities of the large model are leveraged to accurately identify each object in an image without pre-training, reducing the impact of the training dataset on recognition accuracy. Furthermore, the semantic segmentation capabilities of the large model are leveraged to perform semantic segmentation on the image, resulting in an image containing only a single object, further improving recognition accuracy and positioning accuracy. Finally, the positioning information obtained based on semantic segmentation is marked in the initial image, further improving the large model's visual focusing capabilities, meeting user needs, and enhancing the user experience.

[0057] Reference below Figures 3 to 6 , combined with specific embodiments Figure 2 The method shown is further explained.

[0058] Figure 3 A schematic diagram of processing an image based on a large model to generate a first labeled image according to an embodiment of the present disclosure is schematically shown.

[0059] like Figure 3As shown, the initial image 301 is input into the visual perception model 310, and the object name [dog] 302 in the initial image 301 is output. The initial image 301 is input into the visual segmentation model 320, and the initial image 301 is semantically segmented to obtain a segmentation map 303. Then, the target image 304 including only a single object is extracted from the segmentation map 303. Then, the target image 304 and the object name [dog] 302 are input into the visual perception model 310 to obtain a correspondence 305. This correspondence represents the correspondence between the target image, the position of the object in the initial image, and the object name. Finally, based on the correspondence 305, the position of the object in the initial image 301 is marked with a rectangular box, and the object name is annotated in the rectangular box to generate a marked image.

[0060] When the initial image includes multiple objects, since the target image of each object corresponds to the position of each object one-to-one, the target attribute of each object can be determined from the corresponding relationship according to the position of each object; based on the position of each object, each marking graphic can be determined separately; and each marking graphic can be used to mark the target attribute of each object at each position to generate a first marking image.

[0061] Figure 4 A schematic diagram schematically illustrates the process of processing an operation interface image using a large model based on a user instruction to generate a second marker image according to an embodiment of the present disclosure.

[0062] like Figure 4 As shown, the initial image 401 of this embodiment 400 shows the operation interface for purchasing tickets. The initial image 401 includes not only text information but also interface icon information that can be operated, and is a relatively complex image.

[0063] For example, control icons for adjusting the date are respectively provided at the upper left corner and the upper right corner of the initial image 401. When the control icons are clicked, the train operation information and ticket price information of the day before or after the current date can be displayed.

[0064] By visually perceiving the initial image 401, the attributes of each object can be identified, such as "11:56," "Hard seat ticket price is 156.5 yuan," etc. By visually segmenting the initial image 401, the positions of each object, such as [P1], [P2], etc., can be determined. Based on the positions of each object, corresponding single images 402 are extracted from the initial image 401. The single images 402 and the attributes of each object are then processed using a large visual perception model to obtain corresponding relationships 403 between each single image, the object's attributes, and its position.

[0065] Therefore, according to the position of each object, the attribute of each object can be determined from the corresponding relationship 403. For example, the attribute of the control icon located at the [Pi] position is "the hard seat ticket price is 156.5 yuan".

[0066] According to an embodiment of the present disclosure, the position of each object can be represented as pixel coordinates or as a mask array, so each marker pattern can be determined based on the position of each object. Each marker pattern represents the smallest geometric shape that can enclose each object.

[0067] According to an embodiment of the present disclosure, the shape of the marker graphic includes but is not limited to a regular geometric image and an irregular geometric graphic. The shape of the marker graphic can be determined according to the shape of the region where each object is located in the initial image.

[0068] For example: Figure 3 The dog's running posture is approximately rectangular, so a rectangle can be selected as the marker graphic.

[0069] According to an embodiment of the present disclosure, target attributes of each object are marked at each position using respective marking graphics. When generating the first marking image, marking graphics of different shapes may also be used to mark each object.

[0070] According to an embodiment of the present disclosure, by marking each object in the initial image using a geometric figure with the smallest area that can enclose each object, the marking accuracy in scenes with numerous objects and crowded locations is further improved.

[0071] According to an embodiment of the present disclosure, determining each marking graphic based on the position of each object may include the following operations: determining the boundary points of the area where each object is located based on the position of each object; and determining each marking graphic based on the boundary points of the area where each object is located.

[0072] According to the embodiments of the present disclosure, the location of each object in the initial image is usually an irregular area. Whether the location is represented by pixel coordinates or a mask array, the boundary points of the irregular area can be determined. Then, based on the requirements of the application scenario, a marker pattern is determined.

[0073] For example, if there are a small number of objects in the same image and the objects are relatively far apart, a rectangular frame can be generated by using the two boundary points with the largest distance between them in the mask array as the diagonal vertices of the rectangle.

[0074] For example, when there are a large number of objects in the same image and the relative distances between the objects are relatively close, the boundary points in the mask array can be connected in sequence to generate an irregular polygonal box.

[0075] According to an embodiment of the present disclosure, by determining each marking graphic based on the boundary points of the area where each object is located, interference between adjacent objects can be reduced, thereby reducing the impact of interference information in the marking image on subsequent tasks to be performed.

[0076] For example, for two adjacent touch buttons in a visual operation interface, since the contact area between the touch buttons and the robotic arm is large, when the marking areas of adjacent touch buttons overlap, it is easy for the robotic arm to accidentally touch other touch buttons, increasing the difficulty of operation.

[0077] In practical applications, the objects that need to be located and identified typically occupy a large area within the initial image. Therefore, when the initial image contains multiple objects, the target ratios can be calculated based on their positions and the size of the initial image. Based on these ratios, a first target object can be identified from the multiple objects. Finally, based on the position of the first target object, the target image can be extracted from the initial image.

[0078] According to an embodiment of the present disclosure, each target ratio represents the area ratio of the region where each object is located to the initial image.

[0079] According to an embodiment of the present disclosure, the position of each object can be represented as a mask array. The pixel area corresponding to the mask array and each object is usually an irregular polygon. The boundary points corresponding to the mask array can be connected sequentially to obtain the polygon of the pixel area, and the area of ​​the polygon area can be calculated based on the coordinates of each pixel.

[0080] Next, the area ratio of the area where each object is located to the area of ​​the initial image is calculated based on the area of ​​the polygonal area and the area of ​​the initial image.

[0081] According to embodiments of the present disclosure, a ratio threshold, such as 3%, can be configured to identify an object with a target ratio ≥ 3% from among multiple objects as the first target object. Based on the position of the first target object, the target image is extracted from the initial image. The remaining objects are not processed.

[0082] For example: Figure 4 As shown, the ratio between the area of ​​the icon before the train number "1461" in the initial image 401 and the area of ​​the initial image is less than the ratio threshold. Therefore, there is no image corresponding to the icon in the single image 402 after visual segmentation.

[0083] According to the embodiments of the present disclosure, objects with a larger area ratio are targeted and screened for processing based on the area ratio of the area where each object is located in the initial image. This can reduce the amount of redundant information processed by the large model and further improve the processing efficiency of the large model, so as to adapt to scenarios with higher requirements for recognition and positioning efficiency and further improve the user experience.

[0084] Since the area where each object is located is usually an irregular area, and the computing resources required to calculate irregular graphics are relatively high, in order to further reduce the consumption of computing resources, the area of ​​the area where each object is located can be estimated based on the position of each object, the position of each object and the size of the initial image to calculate the target ratio.

[0085] For example: based on the position of each object, the geometric center point of each object and the boundary points of the area where each object is located are determined respectively; with each geometric center point as a circle point, the estimated area of ​​the area where each object is located is generated according to the radius of each target; and based on each estimated area and the size of the initial image, the proportion of each target is calculated.

[0086] According to an embodiment of the present disclosure, the boundary points of the area where each object is located can be determined based on the mask array of the position of each object, and then the coordinates of the geometric center point can be calculated based on the coordinates corresponding to the boundary points.

[0087] According to embodiments of the present disclosure, each target radius is determined by the distance between the geometric center point of each object and the boundary points of the region in which the object resides. For example, the target radius can be determined by the maximum distance between the geometric center point and the boundary points of the region in which the object resides. Alternatively, the target radius can be determined by the average distance between the geometric center point and the boundary points of the region in which the object resides.

[0088] According to the embodiment of the present disclosure, since any irregular shape has a circumscribed circle, the approximate circumscribed circle corresponding to the irregular shape corresponding to the area is obtained by taking each geometric center point as the circle point and each target radius.

[0089] Therefore, the area of ​​the approximate circumscribed circle is used as the estimated area of ​​the region where each object is located, and then the proportion of each target is calculated based on each estimated area and the size of the initial image. This not only reduces the computing resources occupied by the area calculation process, but also does not affect the accuracy of screening the target objects.

[0090] According to an embodiment of the present disclosure, the target image including only a single object, the attributes of each object, and the correspondence between the target image and each attribute obtained based on the method described above can all be stored in a database, so that when the user recognizes the initial image for the second time, the marking result can be directly queried from the database.

[0091] For example, an index structure can be created in the database to facilitate user retrieval. This index structure may include the initial image ID, the operation record ID, the storage address of the initial image, the storage address of the target image, the attribute information of each object, the geometric center coordinates of each object, the storage address of each object's mask array, etc.

[0092] When the user recognizes the initial image for the second time, he or she can also query the database for objects that meet the search criteria based on different search information and mark the initial image again.

[0093] For example: in response to the received information of the object to be identified, each target image and the attributes of each object corresponding to each target image are obtained; the information of the object to be identified and each target image are processed using the second largest model to generate a matching result; based on the matching result, target object information that matches the object to be identified is obtained; and based on the target object information, a second marked image is generated by marking a second target object in the initial image.

[0094] According to an embodiment of the present disclosure, the information of the object to be identified may include the name of the specific object that the user needs to read from the initial image, feature information of a certain type of object, and may also include relative position information of the object.

[0095] For example, for a picture of three pandas sitting side by side, the object to be identified could be "panda", "panda eating bamboo", "panda sitting in the middle", or a combination of the above features, for example, "panda sitting in the middle eating bamboo".

[0096] According to an embodiment of the present disclosure, the attributes of each target image and each object corresponding to each target image may be acquired from the database described above by calling a database interface.

[0097] like Figure 4 As shown, in this embodiment 400, for the initial image 401, the object information to be identified may be a fare 404. The fare may be a type of information, for example, including hard seat fare, hard sleeper fare, soft sleeper fare, no seat fare, etc.

[0098] The corresponding relationship 403 between the single image of each object and the attributes and positions of each object is obtained through visual perception and visual segmentation during the initial image recognition, and can be directly obtained from the database during the secondary image recognition.

[0099] The single image 402 of each object is obtained by visual segmentation during the first recognition of the image, or can be directly obtained from the database during the second recognition of the image.

[0100] The ticket price 404 and the single image 402 of each object are processed by the visual perception large model 310, and the understanding ability of the large model can be used to determine the target single image 405 of the object that meets the user's search requirements from the single image 402 of each object.

[0101] For example, a single image of each object that meets the user's search requirements may include an icon for a hard seat fare control, an icon for a hard sleeper fare control, an icon for a soft sleeper fare control, an icon for a no-seat fare control, and icons for upper and lower berth controls.

[0102] Based on the target single image 405 , each target object is marked in the initial image 401 according to the corresponding position of each target single image in the correspondence relationship 403 , thereby generating a second marked image 406 .

[0103] According to the embodiments of the present disclosure, based on the information of the object to be identified proposed by the user, the language understanding ability of the large model is used to search for a single image of each object to obtain the target object that matches the user's search requirements, further improving the intelligent object focusing ability of the large model and improving the user experience.

[0104] When there are multiple target objects, each target object can be marked individually, or all target objects can be marked as a whole. The specific marking method can be determined according to actual application requirements.

[0105] like Figure 4 As shown, the target objects include multiple ones, and the second labeled image 406 shows the result of overall labeling of all target objects.

[0106] Based on the target object information, generating a second marked image by marking the second target object in the initial image can include the following operations: determining the boundary points of the area where each second target object is located based on the position of the second target object; determining the target marking graphic based on the maximum value of the boundary points of the area where each second target object is located; and using the target marking graphic to mark the attributes of each second target object at the position of the second target object to generate the second marked image.

[0107] According to an embodiment of the present disclosure, the target mark graphic represents a geometric figure with the smallest area that can enclose all second target objects.

[0108] For example, there may be three second target objects, each of which corresponds to a mask array. Based on each mask array, the boundary point of the area where each second target object is located can be determined.

[0109] The target marking graphic can also be any geometric figure determined based on the shape of the area where all the second target objects are located, for example: it can be a rectangular box with the maximum value of the boundary points as the vertex, or it can be a circle with the distance between the maximum value of the boundary points and the geometric center of the area where all the second target objects are located as the radius, and the geometric center as the center.

[0110] Finally, the target marking graphic is used to mark the attributes of each second target object at the position of the second target object to generate a second marking image. In the second marking image, the attributes of each second target object can be marked separately, or the information of the object to be identified can be directly marked.

[0111] According to an embodiment of the present disclosure, a geometric figure with the smallest area that can surround all second target objects is used as a target marking figure, thereby improving the flexibility and accuracy of marking to meet the needs of different application scenarios.

[0112] The object information to be recognized may be text input by the user or a voice command input by the user.

[0113] Figure 5 A schematic diagram schematically illustrates the process of processing an operation interface image using a large model based on a user voice instruction to generate a second marked image according to an embodiment of the present disclosure.

[0114] like Figure 5 As shown, the difference between embodiment 500 and embodiment 400 lies in the method of obtaining the information to be identified.

[0115] For example: in response to a received voice instruction, a text instruction is obtained by recognizing the voice instruction; and information about the object to be recognized is extracted from the text instruction using the third model.

[0116] According to an embodiment of the present disclosure, the third large model may be a Large Language Model (LLM).

[0117] In a human-computer interaction scenario, a user can request object recognition by issuing a voice command. Upon receiving a voice command 501, speech conversion can be used to convert the voice command into a text command 502, "User: Circle the fare information for me." This text command 502 is then input into the LLM 510, where the language understanding capabilities of the large model are leveraged to extract the object information to be recognized, "fare," 404. The processing flow for the object information to be recognized, "fare," 404 is then the same as in Example 400 and will not be further elaborated here.

[0118] According to the embodiments of the present disclosure, leveraging the natural language understanding capabilities of a large model to analyze user commands can lower the barrier to entry and enhance the user experience. Furthermore, the object information output by the large model is more closely aligned with the language the large model can recognize, thereby improving the retrieval accuracy of each target object and enhancing recognition accuracy and efficiency.

[0119] Figure 6 The interactive diagram of the large model-based image generation method according to an embodiment of the present disclosure is schematically shown.

[0120] like Figure 6 As shown, in embodiment 600 , the user can input voice commands and an initial image on the client 610 , and page 611 can be used to display the interaction record between the user and the large model.

[0121] The client 610 may send the voice command to the command processing service module 620. In the command processing service module 620, the object to be recognized 623 is obtained by performing voice conversion S621 and text extraction S622.

[0122] At the same time, client 610 can send the initial image to recognition service module 630. Recognition service module 630 performs visual perception S631 to obtain object attributes 631. Visual segmentation S632 is performed to obtain a single image 632. The large model is then used to process object attributes 631 and single image 632, generating a correspondence 635 between the single image and the object attributes.

[0123] Then, the object attributes 631 , the correspondence 635 , the single image 632 , and the initial image may all be stored in a database 640 .

[0124] Then, the large model is called to process the object to be identified 623 and the single image 632 to obtain an image 633 corresponding to the object to be identified. Furthermore, based on the image 633 corresponding to the object to be identified, the initial image is marked according to the corresponding relationship 635 stored in the database 640 to generate a marked image 634.

[0125] Finally, the marked image 634 is displayed to the user through the page 611 of the client 610 .

[0126] By calling the large model, the positioning mark of each object in the initial image is achieved, which not only improves the object focusing ability of the large model, but also reduces the difficulty of industrialization.

[0127] Figure 7 The block diagram of the apparatus for generating an image based on a large model according to an embodiment of the present disclosure is schematically shown.

[0128] like Figure 7 As shown, the image generating device 700 may include a first generating module 710 , a first extracting module 720 , a second generating module 730 and a first marking module 740 .

[0129] The first generating module 710 is configured to segment the initial image using the first large model to generate a position of at least one object in the initial image.

[0130] The first extraction module 720 is configured to extract at least one target image from the initial image based on a position, wherein the target image includes a single object.

[0131] The second generation module 730 is used to process at least one target image and at least one object's attributes using the second large model to generate a correspondence between the target image and the attributes; wherein the attributes are obtained by processing the initial image using the second large model.

[0132] The first marking module 740 is configured to generate a first marked image by marking at least one object in the initial image according to the position and the correspondence.

[0133] According to an embodiment of the present disclosure, the object includes multiple objects; the first marking module includes: a first determination submodule, a second determination submodule and a first marking submodule.

[0134] The first determining submodule is configured to determine a target attribute of each object from a corresponding relationship according to the position of each object.

[0135] The second determining submodule is configured to determine each marker graphic based on the position of each object; wherein each marker graphic represents a geometric figure with the smallest area that can enclose each object.

[0136] The first marking submodule is configured to mark target attributes of each object at each position using each marking graphic to generate a first marking image.

[0137] According to an embodiment of the present disclosure, the second determining submodule includes: a first determining unit and a second determining unit.

[0138] The first determining unit is configured to determine the boundary points of the region where each object is located based on the position of each object. The second determining unit is configured to determine each marking graphic according to the boundary points of the region where each object is located.

[0139] According to an embodiment of the present disclosure, the first extraction module includes: a calculation submodule, a third determination submodule and an extraction submodule.

[0140] The calculation submodule is used to calculate the target ratio based on the position of each object and the size of the initial image; wherein each target ratio represents the area ratio of the area where each object is located to the initial image.

[0141] The third determining submodule is configured to determine a first target object from the plurality of objects based on the target ratios.

[0142] The extraction submodule is configured to extract a target image from the initial image based on the position of the first target object.

[0143] According to an embodiment of the present disclosure, the calculation submodule includes: a third determination unit, a generation unit, and a calculation unit.

[0144] The third determining unit is configured to determine, based on the position of each object, a geometric center point of each object and a boundary point of an area where each object is located.

[0145] The generating unit is used to generate the estimated area of ​​the area where each object is located with each geometric center point as a circle point and each target radius; wherein each target radius is determined by the distance between the geometric center point of each object and the boundary point of the area where each object is located.

[0146] The calculation unit is used to calculate each target ratio based on each estimated area and the size of the initial image.

[0147] According to an embodiment of the present disclosure, the above-mentioned device further includes: a first acquisition module, a matching module, a second acquisition module and a second marking module.

[0148] The first acquisition module is configured to acquire, in response to the received information of the object to be identified, each target image and the attributes of each object corresponding to each target image.

[0149] The matching module is used to use the second largest model to process the information of the object to be identified and each target image to generate a matching result.

[0150] The second acquisition module is used to acquire target object information that matches the object to be identified based on the matching result.

[0151] The second marking module is configured to generate a second marked image by marking a second target object in the initial image based on the target object information.

[0152] According to an embodiment of the present disclosure, the second marking module includes: a fourth determination submodule, a fifth determination submodule and a second marking submodule.

[0153] The fourth determining submodule is configured to determine the boundary points of the area where each second target object is located based on the position of the second target object.

[0154] The fifth determining submodule is configured to determine a target marking graphic based on the maximum value of the boundary points of the area where each second target object is located; wherein the target marking graphic represents a geometric figure with the smallest area that can surround all the second target objects.

[0155] The second marking submodule is configured to mark the attributes of each second target object at the position of the second target object using the target marking graphic to generate a second marking image.

[0156] According to an embodiment of the present disclosure, the above-mentioned device further includes: an identification module and a second extraction module.

[0157] The recognition module is used to obtain a text instruction by recognizing the voice instruction in response to the received voice instruction.

[0158] The second extraction module is used to extract the information of the object to be identified from the text instruction using the third model.

[0159] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0160] According to an embodiment of the present disclosure, an electronic device includes: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to execute the method described above.

[0161] According to an embodiment of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause a computer to execute the method described above.

[0162] According to an embodiment of the present disclosure, a computer program product includes a computer program, and when the computer program is executed by a processor, the computer program implements the method described above.

[0163] Figure 8 A schematic block diagram of an example electronic device 800 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital assistants, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are provided as examples only and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0164] like Figure 8 As shown, device 800 includes a computing unit 801, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 802 or a computer program loaded from a storage unit 808 into a random access memory (RAM) 803. RAM 803 may also store various programs and data required for the operation of device 800. Computing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to bus 804.

[0165] Various components in device 800 are connected to I / O interface 805, including an input unit 806, such as a keyboard, mouse, etc.; an output unit 807, such as various types of displays, speakers, etc.; a storage unit 808, such as a magnetic disk, optical disk, etc.; and a communication unit 809, such as a network card, modem, wireless communication transceiver, etc. The communication unit 809 allows device 800 to exchange information / data with other devices via a computer network such as the Internet and / or various telecommunication networks.

[0166] The computing unit 801 can be any general-purpose and / or specialized processing component with processing and computing capabilities. Some examples of the computing unit 801 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various specialized artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 801 performs the various methods and processes described above, such as the large-model-based image generation method. For example, in some embodiments, the large-model-based image generation method can be implemented as a computer software program tangibly embodied in a machine-readable medium, such as the storage unit 808. In some embodiments, part or all of the computer program can be loaded and / or installed onto the device 800 via the ROM 802 and / or the communication unit 809. When the computer program is loaded into the RAM 803 and executed by the computing unit 801, one or more steps of the large-model-based image generation method described above can be performed. Alternatively, in other embodiments, the computing unit 801 can be configured to perform the large-model-based image generation method by any other suitable means (e.g., via firmware).

[0167] Various embodiments of the systems and techniques described above can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), system-on-a-chip systems (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments can include being implemented in one or more computer programs that are executable and / or interpreted on a programmable system that includes at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.

[0168] The program code for implementing the method of the present disclosure can be written in any combination of one or more programming languages. These program codes can be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device so that when the program code is executed by the processor or controller, the functions / operations specified in the flow chart and / or block diagram are implemented. The program code can be executed entirely on the machine, partially on the machine, as a stand-alone software package, partially on the machine and partially on a remote machine, or entirely on a remote machine or server.

[0169] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, apparatus, or device, or any suitable combination of the foregoing. More specific examples of machine-readable storage media may include an electrical connection based on one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), optical fibers, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0170] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).

[0171] The systems and techniques described herein can be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include a local area network (LAN), a wide area network (WAN), and the Internet.

[0172] A computer system may include a client and a server. The client and server are generally remote from each other and typically interact through a communication network. The client-server relationship arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other. The server may be a cloud server, a server in a distributed system, or a server integrated with a blockchain.

[0173] It should be understood that the various forms of the processes shown above can be used to reorder, add, or delete steps. For example, the steps described in this disclosure can be performed in parallel, sequentially, or in a different order, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved. This is not a limitation herein.

[0174] The above specific embodiments do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art will appreciate that various modifications, combinations, sub-combinations, and substitutions may be made based on design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure shall be included within the scope of protection of this disclosure.

Claims

1. A method for generating an image based on a large model, comprising: Segmenting the initial image using the first large model to generate a position of at least one object in the initial image; extracting at least one target image from the initial image based on the location, wherein the target image includes a single object; Processing the at least one target image and the at least one object's attributes using the second large model to generate a corresponding relationship between the target image and the attributes; wherein the attributes are obtained by processing the initial image using the second large model; and A first marked image is generated by marking the at least one object in the initial image according to the position and the corresponding relationship.

2. The method according to claim 1, wherein: The object includes a plurality of objects; and generating a first marked image by marking at least one object in the initial image according to the position and the corresponding relationship, comprising: Determining target attributes of each object from the corresponding relationship according to the position of each object; Based on the positions of the objects, determining the marking graphics respectively; wherein the marking graphics represent the geometric figures with the smallest area that can surround the objects; and The target attributes of each of the objects are marked at each of the positions using each of the marking graphics to generate the first marking image.

3. The method according to claim 2, wherein: The step of determining each marking graphic based on the position of each object comprises: Based on the positions of the objects, determining the boundary points of the areas where the objects are located; and Each marking graphic is determined according to the boundary points of the area where each object is located.

4. The method according to claim 1, wherein: The object includes a plurality of objects; and extracting at least one target image from the initial image based on the position includes: Based on the position of each object and the size of the initial image, each target ratio is calculated; wherein each target ratio represents the area ratio of the area where each object is located to the initial image; Based on each of the target ratios, determining a first target object from a plurality of objects; and The target image is extracted from the initial image based on the position of the first target object.

5. The method according to claim 4, wherein: The calculating of each target ratio based on the position of each object and the size of the initial image includes: Based on the position of each object, the geometric center point of each object and the boundary point of the area where each object is located are determined respectively; Taking each of the geometric center points as a circle point and each target radius as a circle point, respectively generating an estimated area of ​​the region where each object is located; wherein each of the target radiuses is determined by the distance between the geometric center point of each object and the boundary point of the region where each object is located; and Based on the estimated areas and the size of the initial image, the target proportions are calculated.

6. The method according to claim 1, further comprising: In response to the received information of the object to be identified, acquiring the attributes of each of the target images and each of the objects corresponding to each of the target images; Processing the information of the object to be identified and each of the target images using the second large model to generate a matching result; Based on the matching result, obtaining target object information matching the object to be identified; as well as Based on the target object information, a second marked image is generated by marking the second target object in the initial image.

7. The method according to claim 6, wherein: The target object information includes the position of the second target object in the initial image and the attributes of the second target object; the second target object includes multiple; The step of generating a second marked image by marking the second target object in the initial image based on the target object information includes: Based on the positions of the second target objects, determining boundary points of the areas where the second target objects are located; Determine a target marking graphic based on the maximum value of the boundary points of the area where each second target object is located; wherein the target marking graphic represents a geometric figure with the smallest area that can surround all the second target objects; and The target marking graphic is used to mark the attributes of each of the second target objects at the position of the second target objects, so as to generate the second marking image.

8. The method according to claim 6 or 7, further comprising: In response to the received voice command, obtaining a text command by recognizing the voice command; as well as The third model is used to extract the information of the object to be identified from the text instruction.

9. An image generation device based on a large model, comprising: A first generating module, configured to segment the initial image using the first large model, and generate a position of at least one object in the initial image; a first extraction module, configured to extract at least one target image from the initial image based on the position, wherein the target image includes a single object; a second generating module, configured to process the at least one target image and the attributes of the at least one object by using the second large model to generate a corresponding relationship between the target image and the attributes; wherein the attributes are obtained by processing the initial image by using the second large model; and The first marking module is used to generate a first marked image by marking the at least one object in the initial image according to the position and the corresponding relationship.

10. The device according to claim 9, wherein: The object includes a plurality of objects; the first marking module includes: A first determination submodule, configured to determine a target attribute of each object from the corresponding relationship according to the position of each object; A second determination submodule is used to determine each marking graphic based on the position of each object; wherein each marking graphic represents a geometric figure with the smallest area that can surround each object; and The first marking submodule is used to mark the target attributes of each of the objects at each of the positions using each of the marking graphics to generate the first marking image.

11. The device according to claim 10, wherein: The second determining submodule includes: A first determining unit, configured to determine a boundary point of an area where each object is located based on the position of each object; and The second determining unit is used to determine each marking graphic according to the boundary points of the area where each object is located.

12. The device according to claim 9, wherein: The object includes a plurality of objects; the first extraction module includes: A calculation submodule, configured to calculate each target ratio based on the position of each object and the size of the initial image; wherein each target ratio represents the area ratio of the area where each object is located to the initial image; A third determination submodule is configured to determine a first target object from a plurality of objects based on the target ratios; and The extraction submodule is used to extract the target image from the initial image based on the position of the first target object.

13. The device according to claim 12, wherein: The calculation submodule includes: A third determining unit, configured to determine, based on the position of each object, a geometric center point of each object and a boundary point of an area where each object is located; A generating unit, configured to generate an estimated area of ​​a region where each object is located using each of the geometric center points as a circle point and each of the target radii; wherein each of the target radii is determined by a distance between the geometric center point of each of the objects and a boundary point of the region where each of the objects is located; and A calculation unit is used to calculate the target proportions based on the estimated areas and the size of the initial image.

14. The apparatus according to claim 9, further comprising: A first acquisition module, configured to acquire the attributes of each target image and each object corresponding to each target image in response to the received information of the object to be identified; A matching module, used for processing the information of the object to be identified and each of the target images using the second large model to generate a matching result; A second acquisition module, used to acquire target object information matching the object to be identified based on the matching result; as well as The second marking module is used to generate a second marked image by marking the second target object in the initial image based on the target object information.

15. The device according to claim 14, wherein: The target object information includes the position of the second target object in the initial image and the attributes of the second target object; The second target object includes a plurality of; The second marking module comprises: a fourth determination submodule, configured to determine, based on the position of the second target object, a boundary point of an area where each second target object is located; a fifth determination submodule, configured to determine a target marking graphic based on the maximum value of the boundary points of the regions where the second target objects are located; wherein the target marking graphic represents a geometric figure with the smallest area that can surround all the second target objects; and The second marking submodule is used to mark the attributes of each second target object at the position of the second target object by using the target marking graphic to generate the second marking image.

16. The device according to claim 14 or 15, further comprising: A recognition module, configured to obtain a text instruction by recognizing the voice instruction in response to the received voice instruction; as well as The second extraction module is used to extract the information of the object to be identified from the text instruction by using the third large model.

17. An electronic device comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 8.

18. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-8.

19. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 8.

Images