Image processing method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides an image processing method and device, electronic equipment and a storage medium, and the method comprises the steps: obtaining a to-be-processed image which comprises at least two image elements; an image analysis model is called to process the to-be-processed image to obtain structured data corresponding to the image elements, the structured data comprises position information and appearance information of the image elements, the position information represents position features of the image elements in the to-be-processed image, and the appearance information represents appearance features of the image elements in the to-be-processed image; and in response to an editing instruction for the structured data, generating an output image comprising at least one image element, so that the two-dimensional unstructured image information of the to-be-processed image is analyzed into the structured data; the problem that image element decomposition is inaccurate due to the fact that a composite image is decomposed based on image element contours to extract image elements is solved.

Description

Technical Field

[0001] The embodiments of the present disclosure relate to the field of image generation technology, and in particular, to an image processing method, device, electronic device, and storage medium. Background Art

[0002] Currently, in the process of digital media production, composite images usually contain multiple layers, such as background layer, main image layer, decorative graphic layer and text layer. There are complex superposition and interdependence relationships between these layers. It is crucial to understand the layered structure of these images for image editing and material archiving of composite images.

[0003] In the prior art, decomposing a composite image to obtain image elements in different layers is usually achieved based on the contours of the image elements. However, the solutions in the prior art have the problem of inaccuracy in decomposing a composite image to extract image elements. Summary of the invention

[0004] The embodiments of the present disclosure provide an image processing method, an apparatus, an electronic device, and a storage medium to overcome the problem of inaccurate extraction of image elements by decomposing a composite image.

[0005] In a first aspect, an embodiment of the present disclosure provides an image processing method, including:

[0006] Acquire an image to be processed, wherein the image to be processed includes at least two image elements;

[0007] Calling an image parsing model to process the image to be processed to obtain structured data corresponding to the image element, wherein the structured data includes position information and appearance information of the image element, the position information represents the position feature of the image element in the image to be processed, and the appearance information represents the appearance feature of the image element in the image to be processed;

[0008] In response to an editing instruction for the structured data, an output image including at least one of the image elements is generated.

[0009] In a second aspect, an embodiment of the present disclosure provides an image processing device, including:

[0010] An acquisition module, used for acquiring an image to be processed, wherein the image to be processed includes at least two image elements;

[0011] A processing module, used for invoking an image parsing model to process the image to be processed, and obtaining structured data corresponding to the image element, wherein the structured data includes position information and appearance information of the image element, the position information represents the position feature of the image element in the image to be processed, and the appearance information represents the appearance feature of the image element in the image to be processed;

[0012] A generating module is used for generating an output image including at least one of the image elements in response to an editing instruction for the structured data.

[0013] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: a processor and a memory;

[0014] The memory stores computer-executable instructions;

[0015] The processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the image processing method described in the first aspect and various possible designs of the first aspect.

[0016] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the image processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0017] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, including a computer program, which, when executed by a processor, implements the image processing method as described in the first aspect and various possible designs of the first aspect.

[0018] The image processing method, device, electronic device and storage medium provided in this embodiment obtain an image to be processed, wherein the image to be processed includes at least two image elements; call an image parsing model to process the image to be processed to obtain structured data corresponding to the image elements, wherein the structured data includes position information and appearance information of the image elements, the position information represents the position feature of the image element in the image to be processed, and the appearance information represents the appearance feature of the image element in the image to be processed; and in response to an editing instruction for the structured data, an output image including at least one of the image elements is generated. On the basis of segmenting each image element in the image to be processed by the image parsing model, position features and appearance features of each image element in the image to be processed are extracted by the image parsing model to obtain the corresponding editable position information and appearance information, thereby generating editable structured data corresponding to the image element; further, based on the editing instruction for the structured data, the structured data is updated and edited to generate an output image including at least one image element; the two-dimensional unstructured image information of the image to be processed is parsed into structured data, and the problem of inaccurate image element decomposition caused by decomposing and extracting image elements from a composite image based on the image element contour is solved. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present disclosure or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative labor.

[0020] Figure 1 An application scenario diagram of the image processing method provided by the embodiment of the present disclosure;

[0021] Figure 2 Schematic diagram of the image processing method provided in the embodiment of the present disclosure Figure 1 ;

[0022] Figure 3 A schematic diagram of the relationship between the superposition relationship and the dependency relationship of image elements of an image to be processed provided by an embodiment of the present disclosure;

[0023] Figure 4 for Figure 2 A flowchart of a specific implementation method of step S102 in the illustrated embodiment;

[0024] Figure 5 for Figure 4 A flowchart of a specific implementation method of step S1021 in the illustrated embodiment;

[0025] Figure 6 for Figure 4 A flowchart of a specific implementation method of step S1022 in the illustrated embodiment;

[0026] Figure 7 A schematic diagram of the structure of an image analysis model provided in an embodiment of the present disclosure;

[0027] Figure 8 A schematic diagram of a mapping relationship between image elements and structured data provided in an embodiment of the present disclosure;

[0028] Fig. 9 for Figure 2 A flowchart of a specific implementation method of step S103 in the illustrated embodiment;

[0029] Fig.10 A schematic diagram of editing structured data provided by an embodiment of the present disclosure;

[0030] Fig.11 The image processing method provided by the present invention is shown in FIG. Figure 2 ;

[0031] Fig.12 for Fig.11 A flowchart of a specific implementation method of step S204 in the illustrated embodiment;

[0032] Fig.13 A structural block diagram of an image processing device provided by an embodiment of the present disclosure;

[0033] Fig.14 A schematic diagram of the structure of an electronic device provided in an embodiment of the present disclosure;

[0034] Fig.15 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0035] In order to make the purpose, technical solution and advantages of the embodiments of the present disclosure clearer, the technical solution in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the drawings in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present disclosure.

[0036] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0037] The application scenarios of the embodiments of the present disclosure are explained below:

[0038] The image processing method provided in the embodiments of the present disclosure can be applied to applications (APPs) with image editing functions, such as camera applications, video / photo editing applications, short video applications, etc. The execution subject of this embodiment can be a terminal device running the above-mentioned application with image editing function, or a server deploying the service end corresponding to the above-mentioned application, or other electronic devices with similar functions.

[0039] Among them, in some embodiments, the terminal device or server can implement the image processing method provided by the embodiment of the present application by running various computer executable instructions or computer programs. For example, computer executable instructions can be program-level commands, machine instructions or software instructions. The computer program can be a native program or software module in the operating system; it can be a local application, that is, a program that needs to be installed in the operating system to run, or it can be a small program embedded in any APP, that is, a program that runs based on a browser environment. In summary, the above-mentioned computer executable instructions can be instructions in any form, and the above-mentioned computer program can be an application, module or plug-in in any form, and the specific implementation form can be configured as needed. Further, in some embodiments, the server can be an independent physical server, or a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides cloud services, cloud storage, cloud communication, cloud database, cloud computing, cloud functions, network services, middleware services, domain name services, security services, content delivery networks (Content Delivery Network, CDN), and basic cloud computing services such as big data and artificial intelligence platforms, wherein the cloud service can be an interactive processing service for terminal devices to call.

[0040] Figure 1 This is an application scenario diagram of the image processing method provided by the embodiment of the present disclosure, referring to Figure 1As shown in, taking the case where the server is the execution subject as an example, an application with an image editing function (hereinafter referred to as the application) is running in the terminal device. After the composite image to be processed (hereinafter referred to as the image to be processed) is loaded through the application, the terminal device sends the image to be processed to the server side for processing by triggering a response functional component, such as the "image element parsing" component shown in the figure, wherein the image to be processed includes two image elements, the first image element is an icon element "rectangle", the interior of the "rectangle" is filled with grid lines, and the second image element is a text element "Cut Prices". The server generates corresponding structured data based on the image to be processed through the method provided in this embodiment, and then sends it back to the terminal device. Then, the terminal device generates an output image containing at least one image element in response to the editing instruction for the structured data input by the user, thereby decomposing and extracting image elements from the composite image to be processed. Of course, it is understandable that in another possible implementation method, the executor of the method provided in this embodiment may also be the terminal device itself, that is, the above-mentioned processing process is all executed by the terminal device, and the models and algorithms used in the execution process may be all deployed locally on the terminal device, or may be partially deployed locally on the terminal device and partially deployed on external devices and called by the terminal device. The specific implementation process is similar and will not be repeated here.

[0041] In the prior art, decomposition and extraction of image elements of a composite image to be processed is usually achieved based on the contours of the image elements. More specifically, for example, a user manually selects and captures the image elements according to the contours, or an application automatically identifies and captures the image elements according to the contours. Since there are complex superposition and interdependence relationships between the layers corresponding to the image elements, for example, the first image element occludes the second image element, the image element contours of the two have complex superposition and interdependence relationships, and then the image element features of the second image element cannot be determined when decomposing and extracting the second image element based on the image element contour. Therefore, the solutions in the prior art lead to the problem of inaccurate decomposition and extraction of image elements when decomposing and extracting the composite image.

[0042] The embodiment of the present disclosure provides an image processing method to solve the above problems.

[0043] refer to Figure 2 , Figure 2 Schematic diagram of the image processing method provided in the embodiment of the present disclosure Figure 1 The method of this embodiment can be applied in a server, a terminal device or other electronic devices. The image processing method includes:

[0044] Step S101: Acquire an image to be processed, where the image to be processed includes at least two image elements.

[0045] refer to Figure 1 In the schematic diagram of the application scenario shown, the image processing method provided in this embodiment is introduced with the server as the execution subject. Exemplarily, a server corresponding to an application with an image editing function is deployed in the server, and the terminal device runs the client of the application. The server and the terminal device obtain the image to be processed sent by the terminal device based on the communication between the server and the client. Further, specifically, the image to be processed contains at least two image elements, for example, the image to be processed includes four image elements, then there are complex superposition relationships and mutual dependence relationships between the layers corresponding to the four image elements, that is, there are complex superposition relationships and mutual dependence relationships between the four image elements.

[0046] Further, for example, refer to Figure 3 , Figure 3 A schematic diagram of the relationship between superposition and dependency of image elements of an image to be processed provided in an embodiment of the present disclosure, wherein the image to be processed includes five image elements, the five image elements are a first image element, a second image element, a third image element, a fourth image element and a fifth image element, the first image element blocks the second image element, the third image element is the image background of the image to be processed, and the combined image element of the fourth image element and the fifth image element is used to indicate the image meaning of the image to be processed, the image meaning refers to information that can be read and understood by a user, and the fourth image element alone or the fifth image element alone cannot be used to indicate the image meaning of the image to be processed, reference Figure 3 As shown in the relationship diagram, the first image element is a banner, the second image element is a power adapter, the third image element is the image background of the image to be processed, the fourth image element is the word "door", and the fifth image element is a lightning icon. The image to be processed is a promotional page of a power adapter, the power adapter corresponding to the second image element is the target product picture, the banner corresponding to the first image element is a decoration of the promotional page, and the combined image element of the fourth image element and the fifth image element is another decoration of the promotional page. The combined image element can be regarded as a deformation of the word "flash", and the image it indicates means that the power adapter charges quickly, that is, "flash charging".

[0047] Step S102: Calling the image parsing model to process the image to be processed to obtain structured data corresponding to the image elements, wherein the structured data includes position information and appearance information of the image elements, the position information represents the position characteristics of the image elements in the image to be processed, and the appearance information represents the appearance characteristics of the image elements in the image to be processed.

[0048] Exemplarily, after obtaining the image to be processed, the server calls the image parsing model to process the image to be processed, segments the image elements, and then obtains the position information and appearance information of the segmented image elements, that is, obtains the structured data corresponding to the image elements, wherein the position information represents the position characteristics of the image elements in the image to be processed, and the appearance information represents the appearance characteristics of the image elements in the image to be processed.

[0049] Specifically, for example, refer to Figure 3 The relationship diagram shown in the figure, the server calls the image parsing model to process the image to be processed, obtains the geometric center coordinates of the image elements, takes the lower left corner of the image to be processed as the coordinate origin of the two-dimensional rectangular coordinate system, and uses the geometric center coordinates corresponding to the geometric center of the image element to indicate the position characteristics of the image element in the image to be processed, and then based on Figure 3 The geometric center coordinates of the five image elements shown in FIG. 1 can determine the corresponding position information; further, refer to Figure 3 For the image element shown, the server calls the image parsing model to process the image to be processed and obtains the pixel information of the image element. Based on the pixel information of the pixel points corresponding to the image element, the appearance characteristics of the image element in the image to be processed can be obtained. For example, the first image element occludes the second image element. For example, the first image element is orange and the second image element is white. Then, at the boundary of the area where the first image element occludes the second image element, the values ​​of the three color channels corresponding to the pixel information are different. That is, the color of the image element and the outline of the image element are determined based on the pixel information, that is, the appearance characteristics of the image element in the image to be processed are determined. It can be understood that for a grayscale image, its pixel information corresponds to a color channel, and its implementation method is similar to the implementation method and implementation effect of this implementation step, which will not be repeated here.

[0050] In one possible implementation, the image parsing model includes a large language model, Figure 4 for Figure 2 The flowchart of the specific implementation method of step S102 in the embodiment shown is as follows: Figure 4 As shown, the specific implementation of step S102 includes:

[0051] Step S1021: extracting features from the image to be processed to obtain multi-level semantic information, where the multi-level semantic information represents semantic features of image elements in at least two receptive field dimensions.

[0052] Exemplarily, after obtaining the image to be processed, the server calls the visual encoder to perform feature extraction on the image to be processed in at least two receptive field dimensions to obtain multi-level semantic information, and the multi-level semantic information represents the semantic features of the image elements in at least two receptive field dimensions. Specifically, for example, the multi-level semantic information includes first-level semantic information, second-level semantic information, and third-level semantic information, the first-level semantic information corresponds to the semantic features of the image elements in the first receptive field dimension, the second-level semantic information corresponds to the semantic features of the image elements in the second receptive field dimension, and the third-level semantic information corresponds to the semantic features of the image elements in the third receptive field dimension; further, with reference to Figure 3 As shown in the relationship diagram, the image elements under the first receptive field dimension include the third image element, the fourth image element and the fifth image element, the image elements under the second receptive field dimension include the fourth image element and the fifth image element, and the image element under the third receptive field dimension is the fifth image element. Then, the first-level semantic information corresponds to the semantic features of the third image element, the fourth image element and the fifth image element, the second-level semantic information corresponds to the semantic features of the fourth image element and the fifth image element, and the third-level semantic information corresponds to the semantic features of the fifth image element.

[0053] In one possible implementation, Figure 5 for Figure 4 The flowchart of the specific implementation method of step S1021 in the embodiment shown is as follows: Figure 5 As shown, the specific implementation of step S1021 includes:

[0054] Step S1021A: Obtain at least two types of visual encoders.

[0055] Step S1021B: extract features of the image to be processed by using at least two types of visual encoders, and fuse the feature extraction results to obtain multi-level semantic information.

[0056] Exemplarily, after obtaining the image to be processed, the server obtains at least two types of visual encoders to implement feature extraction of the image to be processed through at least two types of visual encoders, obtain corresponding feature extraction results, and then fuse the feature extraction results to obtain multi-level semantic information of the image to be processed. It can be understood that when at least two types of visual encoders perform feature extraction on the image to be processed, it can be that based on the same receptive field dimension, at least two types of visual encoders perform feature extraction on the image to be processed or image elements, and then fuse the feature extraction results, or it can be that based on different receptive field dimensions, at least two types of visual encoders perform feature extraction on the image to be processed or image elements, and then fuse the feature extraction results.

[0057] Specifically, for example, after obtaining the image to be processed, the server calls two types of visual encoders, the first type of visual encoder and the second type of visual encoder. The first type of visual encoder is used to perform hierarchical feature processing on the image to be processed, and the second type of visual encoder is used to perform data denoising on the image to be processed. Figure 3 The image to be processed shown in the figure is subjected to feature extraction. Further, for example, feature extraction is performed on the fifth image element under the same receptive field dimension, and the fifth image element is a lightning icon. The first type of visual encoder generates a first type of feature extraction result after hierarchical feature enhancement, and the second type of visual encoder generates a second type of feature extraction result after data denoising. Further, by fusing the first type of feature extraction result and the second type of feature extraction result, the hierarchical semantic information corresponding to the fifth image element can be obtained, that is, the hierarchical semantic information is used to describe the image features of the lightning icon under the current receptive field dimension; that is, under the same receptive field dimension, the first type of visual encoder and the second type of visual encoder are used to extract the fifth image element. Figure 3 The image to be processed is subjected to feature extraction, and the feature extraction results are fused to obtain Figure 3 It is understandable that, in at least two receptive field dimensions, by extracting features from the image to be processed using the first type of visual encoder and the second type of visual encoder, and fusing the feature extraction results, the multi-level semantic information corresponding to the image to be processed can also be obtained.

[0058] In another possible implementation, for example, Figure 3 The image to be processed shown in the figure is subjected to feature extraction. The server calls the first type of visual encoder to perform feature extraction on the fifth image element under the second receptive field to generate a first type of feature extraction result. The server calls the second type of visual encoder to perform feature extraction on the fifth image element under the third receptive field to generate a second type of feature extraction result. Furthermore, by fusing the first type of feature extraction result with the second type of feature extraction result, the hierarchical semantic information corresponding to the fifth image element can be obtained.

[0059] Exemplarily, the visual encoder may include any of the following: an open source visual encoder (visual model) such as a visual encoder based on Sigmoid loss for language image pre-training and a language image pre-training visual encoder. Specifically, for example, after obtaining the image to be processed, the server performs feature processing on the image to be processed based on the visual encoder A, divides the image to be processed into a series of non-overlapping image blocks, and projects them into a low-dimensional linear embedding space to generate a series of block embeddings, and then captures the long-distance dependencies in the image through the self-attention mechanism to generate image features of the image to be processed, and then determines the correspondence between the image features of the image to be processed and the text description, and based on the correspondence of the text description, generates a first image embedding vector, that is, a first feature extraction result; performs feature processing on the image to be processed based on the visual encoder B, that is, extracts rich feature representations from the image to be processed, and generates a second image embedding vector corresponding to the text description by capturing the key details and global structure in the image, that is, generates a second feature extraction result; and then fuses the first feature extraction result and the second feature extraction result to obtain multi-level semantic information.

[0060] Step S1022: Processing multi-level semantic information through a large language model to obtain hierarchical data corresponding to the image to be processed, where the hierarchical data corresponds one-to-one to the image elements and is used to represent the layer content of the layer where the image elements in the image to be processed are located.

[0061] Step S1023: reconstruct the layered data to generate structured data corresponding to the image elements.

[0062] Exemplarily, after generating multi-level semantic information, the server calls the large language model to process the multi-level semantic information, and obtains hierarchical data corresponding to the image to be processed, wherein the hierarchical data corresponds one-to-one to each image element in the image to be processed, and the hierarchical data is used to represent the layer content of the layer where the image element in the image to be processed is located. Specifically, for example, refer to Figure 3 As shown in the relationship diagram, taking the fourth image element in the image to be processed as an example, the multi-level semantic information includes the fourth-level semantic information, and the fourth-level semantic information corresponds to the semantic feature of the fourth image element, that is, the fourth-level semantic information describes the fourth semantic feature of the word "door" corresponding to the fourth image element, and the large language model determines the fourth-level semantic information from the multi-level semantic information, and then the large language model generates the fourth layered data corresponding to the fourth image element according to the fourth-level semantic information. For example, the fourth semantic feature includes the color, height, width, and position information of the pixel points occupied by the word "door" in the image to be processed, and then the fourth layered data generated based on the fourth-level semantic information corresponding to the fourth semantic feature describes the information included in the above-mentioned fourth semantic feature, that is, corresponds to the layer content of the layer where the fourth image element is located.

[0063] Furthermore, the server generates structured data corresponding to the image elements based on the hierarchical data to achieve image reconstruction. Specifically, for example, the hierarchical data corresponds to the layer content of the layer where the image element is located in the image to be processed, and then the server parses the hierarchical data and matches the corresponding structured data master to generate structured data corresponding to the image element. For example, for Figure 3 In the relationship diagram shown, the fourth image element corresponds to the word "door", and its corresponding structured data master mas_text, and then generates structured data data_1 according to the hierarchical data and structured data master mas_text corresponding to the fourth image element. The fifth image element is a lightning icon, and its corresponding structured data master mas_imag, and then generates structured data data_2 according to the hierarchical data and structured data master mas_imag corresponding to the fifth image element.

[0064] In a possible implementation, the image parsing model further includes a vector quantization encoder and a vector quantization decoder. Figure 6 for Figure 4 The flowchart of the specific implementation method of step S1022 in the embodiment shown is as follows: Figure 6 As shown, the specific implementation of step S1022 includes:

[0065] Step S1022A: quantize and encode the multi-level semantic information through a vector quantization encoder to obtain an input encoding vector based on the quantization value.

[0066] Step S1022B: Process the input coding vector through the large language model to obtain hierarchical data based on the quantization value.

[0067] Exemplarily, a vector quantization encoder is used to quantize and encode analog quantities to generate an input coding vector based on a quantization value; specifically, for example, for a color image to be processed, the semantic features of the image elements corresponding to the multi-level semantic information are the color vectors of the image to be processed, and the server calls the vector quantization encoder to quantize and map the color vectors of each block of the image to be processed to the closest representative vector based on a pre-trained codebook (the codebook includes a finite number of representative vectors), and each representative vector has a corresponding code sub-index; then, the representative vector obtained by the quantization mapping is encoded based on the code sub-index, that is, an input coding vector based on the quantization value is obtained. It can be understood that in another possible implementation, the semantic features of the image elements corresponding to the multi-level semantic information are the image embedding vectors of the image to be processed, and the quantization mapping and encoding process of the image embedding vector by the vector quantization encoder are similar to the above-mentioned implementation and implementation effect, which will not be repeated here.

[0068] Furthermore, the server calls the large language model to process the input encoding vector to generate hierarchical data based on the quantized value. Specifically, for example, refer to Figure 3 As shown in the relationship diagram, taking the fourth image element in the image to be processed as an example, the multi-level semantic information includes the fourth-level semantic information, the vector quantization encoder quantizes and encodes the fourth-level semantic information to obtain a fourth input coding vector based on the quantization value, the large language model determines the fourth input coding vector from the input coding vector, and then based on the fourth input coding vector, the large language model generates corresponding hierarchical data based on the quantization value.

[0069] In the steps of this embodiment, multi-level semantic information is quantized and encoded through a vector quantization encoder, thereby achieving compression of the data volume of the image to be processed, that is, reducing the data processing volume of the large language model, and improving the data processing efficiency of the large language model and the accuracy of the output data.

[0070] Correspondingly, when the image parsing model further includes a vector quantization encoder and a vector quantization decoder, the specific implementation of step S1023 includes: quantizing and decoding the hierarchical data through the vector quantization decoder to generate structured data corresponding to the image elements.

[0071] Exemplarily, after the large language model generates hierarchical data based on quantization values, the server calls a vector quantization decoder to perform quantization decoding on the hierarchical data based on the quantization values ​​to generate corresponding representative vectors. Then, based on the representative vectors, the corresponding structured data master is matched to generate structured data corresponding to the image elements.

[0072] Figure 7 A schematic diagram of the structure of an image parsing model provided in the embodiment of the present disclosure is shown below. Figure 7The process of generating structured data is introduced. Exemplarily, the image parsing model includes a first visual encoder, a second visual encoder, a vector quantization encoder, a large language model, and a vector quantization decoder, and the image parsing model is deployed in a server; after the server receives the image to be processed sent by the terminal device, the server calls the image parsing model to parse the image to be processed; specifically, the first visual encoder performs feature processing on the image to be processed to generate a first image embedding vector, and the second visual encoder performs feature processing on the image to be processed to generate a second image embedding vector, and then the first feature extraction result and the second feature extraction result are fused to obtain a fused image embedding vector corresponding to multi-level semantic information; then, the vector quantization encoder performs quantization parsing encoding on the fused image embedding vector to generate an input encoding vector of a quantization value; then, the large language model generates corresponding hierarchical data based on the input encoding vector; then, the vector quantization decoder performs quantization decoding on the hierarchical data based on the quantization value to generate a corresponding representative vector, and matches the corresponding structured data master to generate structured data corresponding to the image element.

[0073] In the steps of this embodiment, the image parsing model parses the two-dimensional unstructured image information into structured data by segmenting each image element. At the same time, based on the superposition relationship and dependency relationship of the image elements, the structured data generated by the image parsing model can fill in the blocked parts of the image elements. Figure 1 In the illustrated schematic diagram of the application scenario, the obscured icon element “rectangle” in the image to be processed can have its obscured rectangular edges filled in based on the structured data.

[0074] Step S103: generating an output image including at least one image element in response to an editing instruction for the structured data.

[0075] Exemplarily, after obtaining the structured data corresponding to the image element, the server responds to the editing instruction for the structured data to implement the modification of the structured data, that is, the editing and modification of the image element, and then the server loads the structured data modified based on the editing instruction by calling the renderer to generate an output image containing at least one image element. Specifically, for example, the structured data corresponding to the image element provides data items that can be modified, for example, reference Figure 3 The image elements shown, the first image element is orange, the data items that can be modified include color channel data, then in response to the editing instruction for changing the color, the color of the first image element can be changed and the first image element with the changed color can be generated.

[0076] Furthermore, image elements include icon elements and text elements. Accordingly, structured data include text structured data and image structured data. Image structured data is used to characterize the appearance characteristics of icon elements; text structured data is used to characterize the text content and font attributes of text elements. Figure 8 A schematic diagram of a mapping relationship between image elements and structured data provided by an embodiment of the present disclosure, such as Figure 8 As shown, reference Figure 1 The image to be processed in the application scenario diagram shown in the figure includes two image elements. The first image element is an icon element "rectangle" whose interior is filled with grid lines, and the second image element is a text element "Cut Prices". Furthermore, for the mapping relationship between icon elements and image structured data, the image structured data data_imag includes "category, object length, position, size, and appearance features". Taking the icon element "rectangle" as an example, Figure 8 As shown, the "category" in the image structured data data_imag is "category": "ima", "object length" is "len": 144, "position" is "x": 204, "y": 15, "size" is "w": 652, "h": 223, and "appearance feature" is "quant": " <vt-01062> … <vt-15692>", where, in one possible implementation, "quant":" <vt-01062> … <vt-15692>" is a representative vector generated by the vector quantization decoder based on the hierarchical data of the quantized value in the above-mentioned embodiment steps; for the mapping relationship between text elements and text structured data, the text structured data data_text includes text content and font attributes, specifically, including "category, text content, position, size, font size, font, font color, arrangement mode", taking the text element "Cut Prices" as an example, Figure 8 As shown, in the text structured data data_text, the "category" is "category":"text", the "text content" is "content":"Cut Prices", the "position" is "x":204, "y":15, the "size" is "w":300, "h":600, the "font size" is "size":12, the "font" is "font":"Times New Roman", the "font color" is "color":[0,0,0], and the "arrangement" is "algin":left.

[0077] In one possible implementation, Fig. 9 for Figure 2 The flowchart of the specific implementation method of step S103 in the embodiment shown is as follows: Fig. 9 As shown, the specific implementation of step S103 includes:

[0078] Step S1031 : generating updated text structured data and / or updated icon elements in response to an editing instruction for text structured data and / or image structured data.

[0079] Step S1032: Generate an output image based on the updated text structured data and / or the rendering result of the updated icon element.

[0080] For example, Fig.10 A schematic diagram of editing structured data provided by an embodiment of the present disclosure, such as Fig.10 As shown, the server updates the text structured data in response to the editing instruction for the text structured data. For example, if the editing instruction is "change the font size of the text content to 15", then refer to Figure 8 In the text structured data data_text shown in FIG. 1 , before being updated, "size" corresponds to 12. Based on the editing instruction, in the updated text structured data generated, "size" corresponds to 15. The server updates the image structured data in response to the editing instruction for the image structured data. For example, if the editing instruction is "double the height of the rectangle", then refer to Figure 8 In the image structured data data_imag shown, before the update, "h" corresponds to 223. Based on the editing instruction, in the updated image structured data generated, "h" corresponds to 446, that is, the height of the generated updated icon element "rectangle" is doubled. Further, the server generates an output image by calling a renderer to load the updated text structured data and / or the updated icon element for rendering.

[0081] In the steps of this embodiment, based on the editable structured data, the server responds to the editing instructions for the image elements, and by changing the structured data, generates updated text structured data and / or updated icon elements, thereby realizing the editing and updating of the image elements in the processed image.

[0082] In this embodiment, by acquiring an image to be processed, the image to be processed includes at least two image elements; calling an image parsing model to process the image to be processed, and obtaining structured data corresponding to the image elements, wherein the structured data includes position information and appearance information of the image elements, the position information represents the position feature of the image element in the image to be processed, and the appearance information represents the appearance feature of the image element in the image to be processed; in response to an editing instruction for the structured data, an output image including at least one image element is generated. On the basis of segmenting each image element in the image to be processed by the image parsing model, the position feature and appearance feature of each image element in the image to be processed are extracted by the image parsing model to obtain the corresponding editable position information and appearance information, and the editable structured data corresponding to the image element can be generated. Further, based on the editing instruction for the structured data, the structured data is updated and edited, and an output image including at least one image element can be generated; the two-dimensional unstructured image information of the image to be processed is parsed into structured data, and the problem of inaccurate decomposition of image elements caused by decomposing and extracting image elements from a composite image based on the image element contour is solved.

[0083] refer to Fig.11 , Fig.11 The image processing method provided by the present invention is shown in FIG. Figure 2 In this embodiment Figure 2 On the basis of the illustrated embodiment, step S102 is further refined, and the image processing method includes:

[0084] Step S201: Acquire an image to be processed, where the image to be processed includes at least two image elements.

[0085] Step S202: extracting features from the image to be processed to obtain multi-level semantic information, where the multi-level semantic information represents semantic features of image elements in at least two receptive field dimensions.

[0086] Step S203: receiving a user operation instruction, where the user operation instruction is used to indicate a target image element in the image to be processed.

[0087] Step S204: Processing multi-level semantic information and user operation instructions through a large language model to obtain target hierarchical data corresponding to the target image element.

[0088] For example, Fig.12 for Fig.11 The flowchart of the specific implementation method of step S204 in the embodiment shown is as follows: Fig.12 As shown, the specific implementation of step S204 includes:

[0089] Step S2041: Generate model prompt words corresponding to the large language model based on the user operation instruction.

[0090] Step S2042, generating model input information according to the model prompt words and multi-level semantic information.

[0091] Step S2043, processing the model input information through the large language model to obtain target hierarchical data corresponding to the target image element.

[0092] Exemplarily, the terminal device generates a user operation instruction based on the interactive operation performed by the user, and the user operation instruction is used to indicate the target image element selected by the user in the image to be processed, and then the server obtains the user operation instruction sent by the terminal device based on the communication between the server and the client. Specifically, for example, the terminal device includes a touch display screen, and the user touches and presses a certain area of ​​the image to be processed, then the image element in the area is the target image element, and then the terminal device generates a user operation instruction corresponding to the target image element based on the user's touch operation. Furthermore, the server receives the user operation instruction sent by the terminal device, processes the user operation instruction, and generates a model prompt word that can be recognized by the large language model. Specifically, for example, taking the lower left corner of the image to be processed as the coordinate origin of the two-dimensional rectangular coordinate system, the position information of the pressing area of ​​the image to be processed where the user touches and presses can be determined based on the user operation instruction, for example, the position coordinate set of the pressing area, that is, the position information corresponding to the target image element, and then the server generates a model prompt word based on the position information, for example, "taking the lower left corner of this image as the coordinate origin of the two-dimensional rectangular coordinate system, obtain the image element in this image within the coordinate range determined by the position coordinate set {(x1, y1), (x2, y2), ..., (xn, yn)}".

[0093] Furthermore, the multi-level semantic information includes target multi-level semantic information corresponding to the target image elements, that is, the target multi-level semantic information is used to characterize the semantic features of the target image elements under at least two receptive field dimensions, and then the target multi-level semantic information is determined from the multi-level semantic information according to the model prompt words, thereby generating the model input information. Specifically, for example, the model prompt word is "take the lower left corner of this image as the coordinate origin of the two-dimensional rectangular coordinate system, and obtain the image elements in this image within the coordinate range determined by the position coordinate set {(x1, y1), (x2, y2), ..., (xn, yn)}", and then based on the position coordinate set, determine the semantic feature set related to the position coordinate set from the multi-level semantic information, and then determine the multi-level semantic information corresponding to the semantic feature set as the target multi-level semantic information; further, the target multi-level semantic information can be used as model input information for processing by the large language model to obtain target hierarchical data corresponding to the target image element; or the target multi-level semantic information is quantized and encoded by a vector quantization encoder to generate model input information based on the quantization value, and then the large language model processes the model input information based on the quantization value to obtain the target hierarchical data corresponding to the target image element, and the target hierarchical data is the hierarchical data based on the quantization value corresponding to the target image element.

[0094] Furthermore, in a possible implementation, the user operation instruction includes a first operation instruction or a second operation instruction, wherein the first operation instruction includes a coordinate value describing the position of the target image element; and the second operation instruction includes text describing the position of the target image element. Specifically, the first operation instruction is based on the coordinate value of the position of the target image element in the image to be processed determined by the user through the input / output (I / O) interface of the terminal device. For example, in the display interface of the terminal device, the user clicks / selects the position area of ​​the target image element in the image to be processed with the mouse, or, in the touch-controlled display interface of the terminal device, the user clicks / selects the position area of ​​the target image element in the image to be processed with a limb (such as a finger), and then the terminal device generates the first operation instruction based on the user's operation; the second operation instruction is generated based on the text input by the user through the I / O interface, such as keyboard input or microphone voice input. For example, the user inputs the text "output the icon in this image" according to the image to be processed displayed by the terminal device, then the coordinate position of the icon in this image corresponds to the text of the position. If the input text is "output the icon in the upper left corner of this image", then "upper left corner" is the text of the position, which is used to indicate the icon selected by the user.

[0095] In the present embodiment, the large language model, on the basis of obtaining multi-level semantic information, realizes personalized processing of the image to be processed based on the user operation instructions and the multi-level semantic information by responding to the user operation instructions, and generates the target hierarchical data of the target image elements corresponding to the user operation instructions in a targeted manner, thereby providing data support for subsequent image reconstruction and editing of the target image elements; at the same time, since the target hierarchical data of the target image elements can be generated in a targeted manner through the user operation instructions and the multi-level semantic information, the data processing amount of the large language model for processing the multi-level semantic information is reduced, thereby improving the data processing efficiency of the large language model.

[0096] Step S205: reconstruct the target hierarchical data to generate target structured data corresponding to the target image elements.

[0097] Step S206: In response to the editing instruction for the target structured data, an output image including at least one target image element is generated.

[0098] In this embodiment, the implementation methods of step S201, step S202, step S205, and step S206 are the same as those of the present disclosure. Figure 2 The implementation methods of step S101, the corresponding sub-steps in step S102, and step S103 in the illustrated embodiment are the same and will not be described in detail herein.

[0099] Furthermore, before all the steps, it also includes a training process for the image parsing model. The training process of the image parsing model through sample images includes: obtaining a sample image, and marking the sample image elements in the sample image, the sample image elements include sample icon elements and sample text elements, and further, marking the appearance features of the sample icon elements to generate corresponding sample icon element labels, marking the text content and font attributes of the sample text elements to generate corresponding sample text element labels; and then, inputting the sample image into the initial image parsing model to generate a process element label, and then adjusting the parameters of the image parsing model based on the residuals of the process element label and the sample text element label, and the residuals of the element label and the sample icon element label, until the residual value meets the preset standard, thereby obtaining the image parsing model.

[0100] Furthermore, when the sample image and the sample user operation instruction are used as two inputs of the model, the training process of the image parsing model is similar to the above training process and will not be repeated here.

[0101] Corresponding to the image processing method of the above embodiment, Fig.13 This is a block diagram of the structure of the image processing device provided in the embodiment of the present disclosure. The method introduced in the above embodiment can be executed by the image processing device, which can be implemented by software and / or hardware, and can be integrated in an electronic device with certain data processing functions. Among them, the electronic device can include but is not limited to mobile terminals with big data processing capabilities, as well as fixed terminals with big data processing capabilities such as desktop computers and supercomputers.

[0102] For the sake of convenience, only the parts related to the embodiments of the present disclosure are shown. Fig.13 , the image processing device 3 comprises:

[0103] An acquisition module 31 is used to acquire an image to be processed, wherein the image to be processed includes at least two image elements;

[0104] A processing module 32 is used to call an image parsing model to process the image to be processed, and obtain structured data corresponding to the image element, wherein the structured data includes position information and appearance information of the image element, the position information represents the position feature of the image element in the image to be processed, and the appearance information represents the appearance feature of the image element in the image to be processed;

[0105] The generating module 33 is used to generate an output image including at least one of the image elements in response to an editing instruction for the structured data.

[0106] According to one or more embodiments of the present disclosure, the image parsing model includes a large language model. When the processing module 32 calls the image parsing model to process the image to be processed and obtains structured data corresponding to the image elements, it is specifically used to: perform feature extraction on the image to be processed to obtain multi-level semantic information, and the multi-level semantic information represents the semantic features of the image elements under at least two receptive field dimensions; process the multi-level semantic information through the large language model to obtain hierarchical data corresponding to the image to be processed, and the hierarchical data corresponds to the image elements one-to-one and is used to represent the layer content of the layer where the image elements in the image to be processed are located; perform image reconstruction on the hierarchical data to generate structured data corresponding to the image elements.

[0107] According to one or more embodiments of the present disclosure, when the processing module 32 performs feature extraction on the image to be processed to obtain multi-level semantic information, it is specifically used to: obtain at least two types of visual encoders; perform feature extraction on the image to be processed through the at least two types of visual encoders, and fuse the feature extraction results to obtain the multi-level semantic information.

[0108] According to one or more embodiments of the present disclosure, the image parsing model also includes a vector quantization encoder and a vector quantization decoder. When the processing module 32 processes the multi-level semantic information through a large language model to obtain the hierarchical data corresponding to the image to be processed, it is specifically used to: quantize and encode the multi-level semantic information through the vector quantization encoder to obtain an input coding vector based on the quantization value; process the input coding vector through the large language model to obtain hierarchical data based on the quantization value; when the processing module 32 reconstructs the hierarchical data to generate the structured data corresponding to the image elements, it is specifically used to: quantize and decode the hierarchical data through the vector quantization decoder to generate the structured data corresponding to the image elements.

[0109] According to one or more embodiments of the present disclosure, the processing module 32 is also used to: receive user operation instructions, wherein the user operation instructions are used to indicate target image elements in the image to be processed; when the processing module 32 processes the multi-level semantic information through a large language model to obtain hierarchical data corresponding to the image to be processed, the processing module 32 is specifically used to: process the multi-level semantic information and the user operation instructions through a large language model to obtain target hierarchical data corresponding to the target image elements.

[0110] According to one or more embodiments of the present disclosure, when the processing module 32 processes the multi-level semantic information and the user operation instruction through a large language model to obtain the target hierarchical data corresponding to the target image element, it is specifically used to: generate a model prompt word corresponding to the large language model based on the user operation instruction; generate model input information according to the model prompt word and the multi-level semantic information; and process the model input information through a large language model to obtain the target hierarchical data corresponding to the target image element.

[0111] According to one or more embodiments of the present disclosure, the user operation instruction includes a first operation instruction or a second operation instruction, wherein the first operation instruction includes a coordinate value describing the position of the target image element; and the second operation instruction includes text describing the position of the target image element.

[0112] According to one or more embodiments of the present disclosure, the image elements include icon elements and text elements, the structured data include text structured data and image structured data, the image structured data is used to characterize the appearance characteristics of the icon elements; the text structured data is used to characterize the text content and font attributes of the text elements, and the generation module 33, when generating an output image containing at least one of the image elements in response to an editing instruction for the structured data, is specifically used to: generate updated text structured data and / or updated icon elements in response to an editing instruction for the text structured data and / or the image structured data; generate the output image based on the rendering results of the updated text structured data and / or the updated icon elements.

[0113] The acquisition module 31, the processing module 32 and the generation module 33 are connected in sequence. The image processing device 3 provided in this embodiment can execute the technical solution of the above method embodiment, and its implementation principle and technical effect are similar, which will not be described in detail in this embodiment.

[0114] Fig.14 A schematic diagram of the structure of an electronic device provided by an embodiment of the present disclosure is shown in FIG. Fig.14 As shown, the electronic device 4 includes:

[0115] A processor 41, and a memory 42 communicatively connected to the processor 41;

[0116] The memory 42 stores computer-executable instructions;

[0117] The processor 41 executes the computer-executable instructions stored in the memory 42 to implement the following Figure 2-Figure 12 The image processing method in the illustrated embodiment.

[0118] Optionally, the processor 41 and the memory 42 are connected via a bus 43 .

[0119] For related instructions, please refer to Figure 2-Figure 12 The relevant descriptions and effects corresponding to the steps in the corresponding embodiments can be understood, and no further elaboration is made here.

[0120] The present invention provides a computer-readable storage medium, wherein the computer-readable storage medium stores computer-executable instructions, which are used to implement the present invention when executed by a processor. Figure 2-Figure 12 The image processing method provided by any one of the corresponding embodiments.

[0121] The present invention provides a computer program product, including a computer program. When the computer program is executed by a processor, the present invention is realized. Figure 2-Figure 12 The image processing method provided by any one of the corresponding embodiments.

[0122] In order to implement the above embodiment, the embodiment of the present disclosure also provides an electronic device.

[0123] refer to Fig.15 , Fig.15 The hardware structure diagram of the electronic device provided in the embodiment of the present disclosure shows a schematic diagram of the structure of the electronic device 900 suitable for implementing the embodiment of the present disclosure, and the electronic device 900 can be a terminal device or a server. The terminal device can include but is not limited to mobile terminals such as mobile phones, laptops, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (such as vehicle-mounted navigation terminals), etc., and fixed terminals such as digital TVs, desktop computers, etc. Fig.15 The electronic device shown is only an example and should not bring any limitation to the functions and scope of use of the embodiments of the present disclosure.

[0124] like Fig.15 As shown, the electronic device 900 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 901, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 902 or a program loaded from a storage device 908 to a random access memory (RAM) 903. In the RAM 903, various programs and data required for the operation of the electronic device 900 are also stored. The processing device 901, the ROM 902, and the RAM 903 are connected to each other via a bus 904. An input / output (I / O) interface 905 is also connected to the bus 904.

[0125] Typically, the following devices may be connected to the I / O interface 905: input devices 906 including, for example, a touch screen, a touch pad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; output devices 907 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; storage devices 908 including, for example, a magnetic tape, a hard disk, etc.; and communication devices 909. The communication device 909 may allow the electronic device 900 to communicate with other devices wirelessly or by wire to exchange data. Although Fig.15 The electronic device 900 is shown with various devices, but it should be understood that it is not required to implement or possess all the devices shown. More or fewer devices may be implemented or possessed instead.

[0126] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program contains program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from a network through a communication device 909, or installed from a storage device 908, or installed from a ROM 902. When the computer program is executed by the processing device 901, the above-mentioned functions defined in the method of the embodiment of the present disclosure are executed.

[0127] It should be noted that the computer-readable medium disclosed above may be a computer-readable signal medium or a computer-readable storage medium or any combination of the above two. The computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or device, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium containing or storing a program that may be used by or in combination with an instruction execution system, device or device. In the present disclosure, a computer-readable signal medium may include a data signal propagated in a baseband or as part of a carrier wave, in which a computer-readable program code is carried. This propagated data signal may take a variety of forms, including but not limited to an electromagnetic signal, an optical signal, or any suitable combination of the above. The computer readable signal medium may also be any computer readable medium other than a computer readable storage medium, which may send, propagate or transmit a program for use by or in conjunction with an instruction execution system, apparatus or device. The program code contained on the computer readable medium may be transmitted using any suitable medium, including but not limited to: wires, optical cables, RF (radio frequency), etc., or any suitable combination of the above.

[0128] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0129] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0130] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as an independent software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In the case of a remote computer, the remote computer may be connected to the user's computer via any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0131] The flow chart and block diagram in the accompanying drawings illustrate the possible architecture, function and operation of the system, method and computer program product according to various embodiments of the present disclosure. In this regard, each square box in the flow chart or block diagram can represent a module, a program segment or a part of a code, and the module, the program segment or a part of the code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some implementations as replacements, the functions marked in the square box can also occur in a sequence different from that marked in the accompanying drawings. For example, two square boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each square box in the block diagram and / or flow chart, and the combination of the square boxes in the block diagram and / or flow chart can be implemented with a dedicated hardware-based system that performs a specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0132] The units or modules involved in the embodiments described in the present disclosure may be implemented by software or hardware, wherein the name of a unit or module does not, in some cases, constitute a limitation on the unit itself.

[0133] The functions described above herein may be performed at least in part by one or more hardware logic components. For example, without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), and the like.

[0134] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0135] In a first aspect, according to one or more embodiments of the present disclosure, there is provided an image processing method, comprising:

[0136] Acquire an image to be processed, wherein the image to be processed includes at least two image elements; call an image parsing model to process the image to be processed to obtain structured data corresponding to the image elements, wherein the structured data includes position information and appearance information of the image elements, the position information represents positional features of the image elements in the image to be processed, and the appearance information represents appearance features of the image elements in the image to be processed; and generate an output image including at least one of the image elements in response to an editing instruction for the structured data.

[0137] According to one or more embodiments of the present disclosure, the image parsing model includes a large language model, and calling the image parsing model to process the image to be processed to obtain structured data corresponding to the image elements includes: performing feature extraction on the image to be processed to obtain multi-level semantic information, and the multi-level semantic information represents the semantic features of the image elements under at least two receptive field dimensions; processing the multi-level semantic information through a large language model to obtain hierarchical data corresponding to the image to be processed, the hierarchical data corresponding to the image elements one-to-one and used to represent the layer content of the layer where the image elements in the image to be processed are located; performing image reconstruction on the hierarchical data to generate structured data corresponding to the image elements.

[0138] According to one or more embodiments of the present disclosure, feature extraction is performed on the image to be processed to obtain multi-level semantic information, including: obtaining at least two types of visual encoders; performing feature extraction on the image to be processed by using the at least two types of visual encoders, and fusing the feature extraction results to obtain the multi-level semantic information.

[0139] According to one or more embodiments of the present disclosure, the image parsing model also includes a vector quantization encoder and a vector quantization decoder, and the processing of the multi-level semantic information through a large language model to obtain hierarchical data corresponding to the image to be processed includes: quantizing and encoding the multi-level semantic information through the vector quantization encoder to obtain an input coding vector based on a quantization value; processing the input coding vector through the large language model to obtain hierarchical data based on a quantization value; and reconstructing the image of the hierarchical data to generate structured data corresponding to the image elements, including: quantizing and decoding the hierarchical data through the vector quantization decoder to generate structured data corresponding to the image elements.

[0140] According to one or more embodiments of the present disclosure, the method further includes: receiving a user operation instruction, wherein the user operation instruction is used to indicate a target image element in the image to be processed; and processing the multi-level semantic information through a large language model to obtain hierarchical data corresponding to the image to be processed, including: processing the multi-level semantic information and the user operation instruction through a large language model to obtain target hierarchical data corresponding to the target image element.

[0141] According to one or more embodiments of the present disclosure, the processing of the multi-level semantic information and the user operation instruction by a large language model to obtain the target hierarchical data corresponding to the target image element includes: generating a model prompt word corresponding to the large language model based on the user operation instruction; generating model input information according to the model prompt word and the multi-level semantic information; and processing the model input information by a large language model to obtain the target hierarchical data corresponding to the target image element.

[0142] According to one or more embodiments of the present disclosure, the user operation instruction includes a first operation instruction or a second operation instruction, wherein the first operation instruction includes a coordinate value describing the position of the target image element; and the second operation instruction includes text describing the position of the target image element.

[0143] According to one or more embodiments of the present disclosure, the image elements include icon elements and text elements, the structured data include text structured data and image structured data, the image structured data is used to characterize the appearance characteristics of the icon elements; the text structured data is used to characterize the text content and font attributes of the text elements, and the output image containing at least one of the image elements is generated in response to editing instructions for the structured data, including: generating updated text structured data and / or updated icon elements in response to editing instructions for the text structured data and / or the image structured data; generating the output image based on the rendering results of the updated text structured data and / or updated icon elements.

[0144] In a second aspect, according to one or more embodiments of the present disclosure, there is provided an image processing apparatus, comprising:

[0145] An acquisition module, used for acquiring an image to be processed, wherein the image to be processed includes at least two image elements;

[0146] A processing module, used for invoking an image parsing model to process the image to be processed, and obtaining structured data corresponding to the image element, wherein the structured data includes position information and appearance information of the image element, the position information represents the position feature of the image element in the image to be processed, and the appearance information represents the appearance feature of the image element in the image to be processed;

[0147] A generating module is used for generating an output image including at least one of the image elements in response to an editing instruction for the structured data.

[0148] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0149] The memory stores computer-executable instructions;

[0150] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the image processing method described in the first aspect and various possible designs of the first aspect.

[0151] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When a processor executes the computer execution instructions, the image processing method described in the first aspect and various possible designs of the first aspect is implemented.

[0152] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, including a computer program, which, when executed by a processor, implements the image processing method described in the first aspect and various possible designs of the first aspect.

[0153] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above disclosed concept. For example, the above features are replaced with the technical features with similar functions disclosed in the present disclosure (but not limited to) by each other to form a technical solution.

[0154] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details are included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0155] Although the subject matter has been described in language specific to structural features and / or methodological logical actions, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or actions described above. On the contrary, the specific features and actions described above are merely example forms of implementing the claims. < / vt-01062> < / vt-01062>

Claims

1. An image processing method, characterized in that: include: Acquire an image to be processed, wherein the image to be processed includes at least two image elements; Calling an image parsing model to process the image to be processed to obtain structured data corresponding to the image element, wherein the structured data includes position information and appearance information of the image element, the position information represents the position feature of the image element in the image to be processed, and the appearance information represents the appearance feature of the image element in the image to be processed; In response to an editing instruction for the structured data, an output image including at least one of the image elements is generated.

2. The method according to claim 1, characterized in that: The image parsing model includes a large language model, The calling of the image parsing model to process the image to be processed to obtain structured data corresponding to the image elements includes: Performing feature extraction on the image to be processed to obtain multi-level semantic information, wherein the multi-level semantic information represents semantic features of the image elements in at least two receptive field dimensions; Processing the multi-level semantic information through a large language model to obtain hierarchical data corresponding to the image to be processed, wherein the hierarchical data corresponds to the image elements one by one and is used to represent the layer content of the layer where the image elements in the image to be processed are located; Image reconstruction is performed on the layered data to generate structured data corresponding to the image elements.

3. The method according to claim 2, characterized in that Feature extraction is performed on the image to be processed to obtain multi-level semantic information, including: Obtain at least two types of visual encoders; The image to be processed is subjected to feature extraction by using the at least two types of visual encoders, and the feature extraction results are fused to obtain the multi-level semantic information.

4. The method according to claim 2, characterized in that: The image parsing model also includes a vector quantization encoder and a vector quantization decoder. The step of processing the multi-level semantic information by using a large language model to obtain hierarchical data corresponding to the image to be processed includes: The multi-level semantic information is quantized and encoded by the vector quantization encoder to obtain an input encoding vector based on the quantization value; Processing the input encoding vector through the large language model to obtain hierarchical data based on quantized values; The step of reconstructing the layered data to generate structured data corresponding to the image elements includes: The hierarchical data is quantized and decoded by the vector quantization decoder to generate structured data corresponding to the image elements.

5. The method according to claim 2, characterized in that: The method further comprises: receiving a user operation instruction, wherein the user operation instruction is used to indicate a target image element in the image to be processed; The step of processing the multi-level semantic information by using a large language model to obtain hierarchical data corresponding to the image to be processed includes: The multi-level semantic information and the user operation instruction are processed by a large language model to obtain target hierarchical data corresponding to the target image element.

6. The method according to claim 5, characterized in that The step of processing the multi-level semantic information and the user operation instruction by a large language model to obtain target hierarchical data corresponding to the target image element includes: Based on the user operation instruction, generating a model prompt word corresponding to the large language model; Generate model input information according to the model prompt words and the multi-level semantic information; The model input information is processed through a large language model to obtain target hierarchical data corresponding to the target image element.

7. The method according to claim 5, characterized in that The user operation instruction includes a first operation instruction or a second operation instruction, wherein the first operation instruction includes a coordinate value describing the position of the target image element; and the second operation instruction includes a text describing the position of the target image element.

8. The method according to claim 1, characterized in that The image element includes an icon element and a text element, the structured data includes text structured data and image structured data, and the image structured data is used to characterize the appearance characteristics of the icon element; The text structured data is used to represent the text content and font attributes of the text element. The step of generating an output image including at least one of the image elements in response to an editing instruction for the structured data comprises: In response to an editing instruction for the text structured data and / or the image structured data, generating updated text structured data and / or updated icon elements; The output image is generated based on the updated text structured data and / or the rendering result of the updated icon element.

9. An image processing device, characterized in that: include: An acquisition module, used for acquiring an image to be processed, wherein the image to be processed includes at least two image elements; A processing module, used for invoking an image parsing model to process the image to be processed, and obtaining structured data corresponding to the image element, wherein the structured data includes position information and appearance information of the image element, the position information represents the position feature of the image element in the image to be processed, and the appearance information represents the appearance feature of the image element in the image to be processed; A generating module is used for generating an output image including at least one of the image elements in response to an editing instruction for the structured data.

10. An electronic device, characterized in that: include: Processor and memory; The memory stores computer-executable instructions; The processor executes the computer-executable instructions stored in the memory, so that the processor performs the image processing method according to any one of claims 1 to 8.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer-executable instructions, and when the processor executes the computer-executable instructions, the image processing method according to any one of claims 1 to 8 is implemented.

12. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the image processing method according to any one of claims 1 to 8 is implemented.

Images