Method for reading pictures and writing, electronic equipment and computer readable storage medium

Abstract

The invention discloses a method for writing by looking at a picture, electronic equipment and a computer readable storage medium, and the method comprises the steps: collecting an image of a written question, the image comprising a question picture; according to whether the image contains characters or not and the positions of the characters in the image under the condition that the characters are contained, classifying the written questions to obtain the types of the written questions; and generating writing content by using a preset model based on a cue word template corresponding to the type and the image. By utilizing the scheme disclosed by the invention, the intentions of different types of writing questions can be accurately understood, the influence on the quality of the writing content caused by relatively high complexity of the content of the writing questions by watching a picture can be reduced, and the quality and the accuracy of the writing content generated by a large model can be improved, so that a better tutoring effect can be obtained.

Description

Technical Field

[0001] The present disclosure generally relates to the field of computer vision technology. More specifically, the present disclosure relates to a method for writing based on images, an electronic device, and a computer-readable storage medium. Background Art

[0002] As artificial intelligence (AI) is increasingly used in the field of education, its intelligent assistance has improved children's learning efficiency. In picture-based writing learning, AI also plays a relatively good auxiliary role. From the perspective of the industry's overall plan, the process of picture-based writing tutoring with the help of AI generally includes the following operations: users input writing topics by taking photos; use AI to understand the writing topics and generate writing content based on the writing topics; and output writing content.

[0003] The most important part of this tutoring process is AI's understanding of the writing topic. Only on the basis of a correct understanding of the writing topic can the subsequent generated writing content guarantee effectiveness and quality. However, picture-based writing topics are often composed of pictures and texts, and the content is relatively complex, which makes AI unable to accurately understand the intention of the writing topic, and thus unable to generate high-quality and accurate writing content, resulting in poor tutoring results.

[0004] In view of this, there is an urgent need to provide a solution for picture-based writing, so as to accurately understand the intention of the writing topic, thereby generating writing content with higher quality and accuracy and improving the tutoring effect. Summary of the invention

[0005] In order to at least solve one or more of the technical problems mentioned above, the present disclosure proposes solutions for writing by looking at pictures in the following aspects.

[0006] In a first aspect, the present disclosure provides a method for writing based on pictures, comprising: acquiring an image of a writing topic, the image containing a topic picture; classifying the writing topic according to whether the image contains text and, if so, the position of the text in the image to obtain the type of the writing topic; and generating writing content using a preset model based on a prompt word template corresponding to the type and the image.

[0007] In some embodiments, classifying the writing topic according to whether the image contains text and the position of the text in the image if the image contains text includes: performing optical character recognition on the image to determine whether the image contains text; if the image does not contain text, determining the type of the writing topic as a type without text; if the image contains text, obtaining the first coordinates of the text; and classifying the writing topic based on the text and its first coordinates.

[0008] In some embodiments, classifying the writing topic based on the text and its first coordinate includes: splitting the text based on the text and its first coordinate to obtain multiple sentences and second coordinates corresponding to the sentences; based on the image, using the preset model to identify the topic image contained in the image to obtain the topic image and the third coordinates corresponding to the topic image; comparing the second coordinate corresponding to each sentence with the third coordinate corresponding to the topic image in turn to obtain the coordinate comparison results of each sentence; and classifying the writing topic according to the coordinate comparison results of each sentence.

[0009] In some embodiments, classifying the writing topic according to the coordinate comparison results of the sentences includes: when all the coordinate comparison results of the sentences are located within the topic map, determining the type of the writing topic as an in-picture text type; when all the coordinate comparison results of the sentences are located outside the topic map, determining the type of the writing topic as an out-of-picture text type; and when part of the coordinate comparison results of the sentences are located within the topic map and the other part are located outside the topic map, determining the type of the writing topic as a mixed text type.

[0010] In some embodiments, based on the prompt word template corresponding to the type and the image, generating writing content using a preset model includes: based on the second coordinates corresponding to the sentences, aggregating multiple sentences with the same coordinate comparison results to obtain in-image sentences and / or out-image sentences; combining the image, the in-image sentences and / or the out-image sentences with the prompt word template corresponding to the type to obtain model prompt words; and based on the model prompt words, generating the writing content using the preset model.

[0011] In some embodiments, the prompt word template includes at least one input item, a special prompt part and a general prompt part, and the general prompt part in the prompt word templates corresponding to different types is the same, and the input item and the special prompt part are different; and when the type is an in-picture text type, the input item includes a title picture and an in-picture sentence, and the special prompt part is used to instruct the preset model to generate the writing content according to the title picture and the in-picture sentence; when the type is an out-picture text type, the input item includes a title picture and an out-picture sentence, and the special prompt part is used to instruct the preset model to generate the writing content according to the title picture and the out-picture sentence; and when the type is a mixed text type, the input item includes a title picture, an out-picture sentence and an in-picture sentence, and the special prompt part is used to instruct the preset model to generate the writing content according to the title picture, the out-picture sentence and the in-picture sentence.

[0012] In some embodiments, based on the text and its first coordinate, the text is split to obtain multiple sentences and the second coordinates corresponding to the sentences, including: based on the text and its first coordinate, the text with the same vertical coordinate is classified into one sentence, and the multiple texts in each sentence are sorted from small to large according to the horizontal coordinate to obtain multiple sentences and the second coordinates corresponding to the sentences.

[0013] In a second aspect, the present disclosure provides an electronic device, comprising: a processor; and a memory storing program instructions for writing by looking at pictures, wherein when the program instructions are executed by the processor, the method described in the first aspect and its multiple embodiments are implemented.

[0014] In a third aspect, the present disclosure provides a computer-readable storage medium having stored thereon program instructions for writing by looking at pictures, and when the program instructions are executed by a processor, the method described in the first aspect and its multiple embodiments are implemented.

[0015] Through the solution for picture-based writing provided above, when using AI to coach picture-based writing topics, the image of the writing topic is not directly input into the big model to generate writing content, but the writing topic is first classified according to whether the image includes text, and the position of the text contained in the image if it contains text, so as to obtain the type of writing topic. Then, based on the prompt word template corresponding to the type and the image, the big model is used to generate writing content. With the prompt of the prompt word template corresponding to the type, the big model can accurately understand the intention of different types of writing topics, can reduce the impact of the high content complexity of picture-based writing topics on the quality of writing content, and help improve the quality and accuracy of the writing content generated by the big model, so as to obtain better coaching results. BRIEF DESCRIPTION OF THE DRAWINGS

[0016] By reading the detailed description below with reference to the accompanying drawings, the above and other objects, features and advantages of the exemplary embodiments of the present disclosure will become readily understood. In the accompanying drawings, several embodiments of the present disclosure are shown in an exemplary and non-limiting manner, and the same or corresponding reference numerals represent the same or corresponding parts, wherein:

[0017] Figure 1 is an exemplary flow chart showing a method for writing based on pictures according to some embodiments of the present disclosure;

[0018] Figure 2 is an exemplary flow chart showing a method for writing based on pictures according to other embodiments of the present disclosure;

[0019] Figure 3 It is shown Figure 2 An exemplary flow chart of a step in the process;

[0020] Figure 4 It is shown Figure 2 An exemplary flow chart of another step in

[0021] Figure 5 is an exemplary flow chart showing a method for writing by looking at a picture implemented by a writing by looking at a picture system according to an embodiment of the present disclosure;

[0022] Figure 6 is an exemplary image showing a writing topic according to an embodiment of the present disclosure;

[0023] Figure 7 is a block diagram showing an exemplary structure of an electronic device according to an embodiment of the present disclosure. DETAILED DESCRIPTION

[0024] The following will be combined with the drawings in the embodiments of the present disclosure to clearly and completely describe the technical solutions in the embodiments of the present disclosure. Obviously, the described embodiments are part of the embodiments of the present disclosure, not all of the embodiments. Based on the embodiments in the present disclosure, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of the present disclosure.

[0025] It should be understood that when the terms "first", "second", "third" and "fourth" are used in the claims, the specification and the drawings of the present application, they are only used to distinguish different objects, rather than to describe a specific order. The terms "include" and "comprise" used in the specification and claims of the present disclosure indicate the presence of the described features, wholes, steps, operations, elements and / or components, but do not exclude the presence or addition of one or more other features, wholes, steps, operations, elements, components and / or their collections.

[0026] It should also be understood that the terms used in this disclosure are only for the purpose of describing specific embodiments and are not intended to limit the disclosure. As used in this disclosure and claims, the singular forms of "a", "an", and "the" are intended to include the plural forms unless the context clearly indicates otherwise. It should also be further understood that the term "and / or" used in this disclosure and claims refers to any combination of one or more of the associated listed items and all possible combinations, including these combinations.

[0027] As used in this specification and claims, the term "if" may be interpreted as "when" or "upon" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrase "if it is determined" or "if [described condition or event] is detected" may be interpreted as meaning "upon determination" or "in response to determining" or "upon detection of [described condition or event]" or "in response to detecting [described condition or event]," depending on the context.

[0028] The specific implementation of the present disclosure is described in detail below with reference to the accompanying drawings.

[0029] Figure 1 An exemplary flow chart of a method 100 for writing by looking at a picture according to an embodiment of the present disclosure is shown. It is understandable that the method 100 can be executed by any appropriate device with data processing capabilities, such as but not limited to a terminal device, a processor, and a server. Here, the terminal device may include but is not limited to a smart phone, an intelligent learning machine, a personal computer, a laptop, a tablet computer, and a portable wearable device.

[0030] like Figure 1 As shown, at step S101, method 100 can capture an image of a writing topic through a camera device, and the image includes a topic picture. In one embodiment, the writing topic may include at least one topic picture. In another embodiment, the writing topic may include at least one topic picture, topic requirements (also referred to as sentences outside the picture) and / or topic captions (also referred to as sentences inside the picture). Generally speaking, the topic requirements can be presented in text form outside the topic picture (for example, above the topic picture) to explain the formal requirements of the writing topic (for example, the word range, the words that cannot be written are replaced by pinyin). The topic caption can be presented in text form inside the topic picture (for example, the upper left corner of the topic picture, the upper corner of the topic picture, near the character in the topic picture). The topic caption can be a dialogue, inner monologue, descriptive text or any other form of text, which is a supplementary explanation to the topic picture and can help writers better understand the information or story expressed by the topic picture.

[0031] Next, at step S102, method 100 may classify the writing topic according to whether the image contains text and the position of the text in the image if the image contains text, so as to obtain the type of the writing topic.

[0032] In the disclosed solution, the position of the text in the image can be divided into being located outside the title image and being located inside the title image. The types of writing titles can at least include a type without text, a type of text inside the image, a type of text outside the image, and a mixed type of text.

[0033] The no-text type can indicate that the writing topic only includes at least one topic picture, and there is no text inside or outside the topic picture, that is, it does not include the topic requirements and topic captions. The in-picture text type can indicate that the writing topic includes at least one topic picture, and there is text inside the topic picture, but no text outside the topic picture, that is, the writing topic also includes the topic captions, but does not include the topic requirements. The out-picture text type can indicate that the writing topic includes at least one topic picture, and there is no text inside the topic picture, but there is text outside the topic picture, that is, the writing topic also includes the topic requirements, but does not include the topic captions. The mixed text type can indicate that the writing topic includes at least one topic picture, and there is text inside and outside the topic picture, that is, the writing topic also includes the topic requirements and topic captions.

[0034] Based on this, the writing topic is classified according to whether the image contains text, and the position of the text in the image if the image contains text. Specifically, when the image does not contain text, the type of the writing topic can be determined to be a text-free type. When the image contains text, if the positions of the text in the image are all located within the topic image, the type of the writing topic can be determined to be an in-image text type. When the image contains text, if the positions of the text in the image are all located outside the topic image, the type of the writing topic can be determined to be an out-of-image text type. When the image contains text, if part of the position of the text in the image is located within the topic image and the rest is located outside the topic image, the type of the writing topic can be determined to be a mixed text type.

[0035] Next, at step S103, the method 100 may generate writing content using a preset model based on the prompt word template and the image corresponding to the type.

[0036] In the scheme disclosed herein, the prompt word template may include at least one input item, a special prompt part, and a general prompt part. The general prompt part of the prompt word templates corresponding to different types is the same, and the number of input items and the content of the special prompt part of the prompt word templates corresponding to different types are different. The general prompt part may include task objectives and task steps. The task objective is used to indicate what task the preset model is to complete, for example, it can generate writing content based on the picture of the writing topic. The task steps are used to indicate the process of the preset model to complete the task, for example, it can include understanding the writing topic, disassembling the writing topic, generating writing ideas, generating model essays, and reading model essays carefully.

[0037] When the type is a textless type, the number of input items is 1, that is, the input items only include the title image. The special prompt part is used to instruct the preset model to generate writing content based on the title image. For example, the content of the special prompt part can be "understand and describe the title image, and generate writing content based on the description."

[0038] When the type is the text in the picture type, the number of input items is 2, that is, the input items include the title picture and the sentence in the picture. The special prompt part is used to instruct the preset model to generate writing content based on the title picture and the sentence in the picture. For example, the content of the special prompt part can be "understand and describe the title picture in combination with the sentence in the picture, and generate writing content based on the description."

[0039] When the type is the text outside the picture type, the number of input items is 2, that is, the input items include the title picture and the sentence outside the picture. The special prompt part is used to instruct the preset model to generate writing content based on the title picture and the sentence outside the picture. For example, the content of the special prompt part can be "understand and describe the title picture in combination with the sentence outside the picture, and generate writing content based on the description."

[0040] When the type is a mixed text type, the number of input items is 3, that is, the input items include the title image, sentences outside the image, and sentences inside the image. The special prompt part is used to instruct the preset model to generate writing content based on the title image, sentences outside the image, and sentences inside the image. For example, the content of the special prompt part can be "the result sentences outside the image and sentences inside the image understand and describe the title image, and generate writing content based on the description."

[0041] The combination of the prompt word template and the image corresponding to the type constitutes the prompt word prompt of the preset model. The prompt word is input into the preset model, so that the preset model can generate writing content under the prompt of the prompt word. In one embodiment, the writing content generated by the preset model may include but is not limited to the topic decomposition, writing ideas, reference model essays and model essay intensive reading comments. The topic decomposition is the information obtained after understanding the topic picture, topic caption and / or topic requirements, which may include but is not limited to time, place, characters and their relationships, actions, expressions and environment, etc.

[0042] It is understandable that although the vertical large model can improve the large model's understanding of writing topics, the time, data resources and computing resources required to train the vertical large model are huge, which is not achievable by ordinary teams or individuals, and its universality is poor. Therefore, in the scheme disclosed herein, the preset model uses a general large model, for example, a large multimodal model (Large Multimodal Models, LMMs). The multi-model large model is an artificial intelligence model that integrates various types of data input and processing capabilities. It can simultaneously understand and analyze data in multiple modes such as text, images, and audio. Compared with the investment in vertical large model training, the training of the general large model does not require a large amount of data resources and computing power resources to complete the effect upgrade of the model output content, and the training cost is low.

[0043] In addition, the general large model can process various types of data input, and then can generate writing content for various types of writing topics, with higher stability and universality. After generating the writing content, the method 100 can also output the writing content, and can collect user feedback on the writing content. Additionally or optionally, the method 100 can output the writing content to the user in an audible or visual manner. The user can input feedback content to the device in voice or text.

[0044] Combination of the above Figure 1 A method 100 for picture-based writing is described. When the method 100 uses AI to provide guidance on picture-based writing topics, the image of the writing topic is not directly input into a large model to generate writing content. Instead, the writing topic is first classified according to whether the image includes text, and if so, the position of the text contained in the image, so as to obtain the type of the writing topic. Then, based on the prompt word template corresponding to the type and the image, the large model is used to generate writing content. With the prompt of the prompt word template corresponding to the type, the large model can accurately understand the intention of different types of writing topics, can reduce the impact of the high content complexity of the picture-based writing topic on the quality of the writing content, and help improve the quality and accuracy of the writing content generated by the large model, thereby achieving better guidance results.

[0045] Figure 2 FIG. 2 shows an exemplary flow chart of a method 200 for writing based on a picture according to some other embodiments of the present disclosure. Figure 2 At step S201, the method 200 may collect an image of the writing topic. It should be noted that, in this embodiment, the content of step S201 is consistent with step S101 in the previous embodiment, and will not be repeated here.

[0046] Next, at step S202, method 200 may perform optical character recognition (OCR) on the image to determine whether the image contains text. Optical character recognition technology is a technology that converts text in an image (e.g., Chinese characters, letters, numbers, spaces, punctuation marks, etc.) into machine-readable text. It can identify whether an image contains text through image processing and pattern recognition algorithms, and if the image contains text, the coordinates of the text can be obtained. It can be understood that the coordinates of the text obtained by OCR usually use the upper left corner of the image as the reference origin, and are represented by the coordinates of the upper left corner point and the coordinates of the lower right corner point of the rectangular box of the image area where the text is located. In short, the coordinates of the text include the coordinates of the upper left corner point and the coordinates of the lower right corner point.

[0047] After performing optical character recognition on the image, if the recognition result is that the image does not contain text, method 200 can proceed to step S203 to determine that the type of the writing topic is a text-free type. On the contrary, if the recognition result is that the image contains text, method 200 can proceed to the following steps S204 and S205.

[0048] At step S204, method 200 may obtain the coordinates of the characters from the recognition results (for distinction, they may be referred to as first coordinates). Next, at step S205, method 200 may classify the writing topics based on the characters and their first coordinates.

[0049] In order to understand how to classify writing topics, Figure 3 The process of classifying writing topics based on words and their first coordinates is described in detail. Figure 3 As shown, at step S2051, method 200 can split the text based on the text and its first coordinates to obtain multiple sentences and the coordinates corresponding to the sentences (for distinction, they can be called second coordinates).

[0050] In actual operation, the text can be split according to the principle of one line and one sentence based on the text and its first coordinate to obtain multiple sentences and the second coordinates corresponding to the sentences. Specifically, the text with the same ordinate can be classified as one sentence, and the multiple texts in each sentence can be sorted from small to large according to the horizontal coordinate to obtain multiple sentences. The second coordinate corresponding to the sentence is represented by the coordinates of the text with the smallest horizontal coordinate in the sentence (for distinction, it can be called the first text) and the text with the largest horizontal coordinate (for distinction, it can be called the second text). Specifically, the second coordinate corresponding to the sentence can be represented by the coordinates of the upper left corner point of the first text and the coordinates of the lower right corner point of the second text. It can be understood that since the preset model in the previous text is a general large model, it can understand and analyze text data. In actual operation, the text can be split using the preset model based on the text and its first coordinate to obtain multiple sentences and the second coordinates corresponding to the sentences.

[0051] Then, at step S2052, the method 200 can identify the title image contained in the image based on the image using the above-mentioned preset model to obtain the title image and the coordinates corresponding to the title image (which can be called the third coordinates for the sake of distinction). The third coordinate of the title image is usually based on the upper left corner of the image as the reference origin, and is represented by the coordinates of the upper left corner point and the coordinates of the lower right corner point of the rectangular frame of the image area where the title image is located. In other words, the third coordinate includes the coordinates of the upper left corner point and the coordinates of the lower right corner point.

[0052] Next, at step S2053, the method 200 may sequentially compare the second coordinates corresponding to each sentence with the third coordinates corresponding to the topic map to obtain a coordinate comparison result of each sentence. Here, the coordinate comparison result may be located within the topic map or outside the topic map.

[0053] Here, when comparing the second coordinate corresponding to the sentence with the third coordinate corresponding to the title image, the coordinate of the upper left corner point in the second coordinate (for distinction, it can be called the second upper left corner point coordinate) can be compared with the coordinate of the upper left corner point in the third coordinate (for distinction, it can be called the third upper left corner point coordinate), and the coordinate of the lower right corner point in the second coordinate (for distinction, it can be called the second lower right corner point coordinate) can be compared with the coordinate of the lower right corner point in the third coordinate (for distinction, it can be called the third lower right corner point coordinate). In the case where the second upper left corner point coordinate is greater than or equal to the third upper left corner coordinate, and the second lower right corner point coordinate is less than or equal to the third lower right corner point coordinate, it can be determined that the coordinate result is located within the title image. On the contrary, in the case where the second upper left corner point coordinate is less than the third upper left corner coordinate, or the second lower right corner point coordinate is greater than the third lower right corner coordinate, it can be determined that the coordinate comparison result is located outside the title image.

[0054] When the number of topic images contained in the image is greater than 1, it is necessary to compare the second coordinate corresponding to each sentence with the third coordinate corresponding to each topic image to determine the coordinate comparison result of each sentence. Further, in the case where the sentence is located in any topic image, the coordinate comparison result can be determined to be located in the topic image. In the case where the sentence is not located in all topic images, the coordinate comparison result can be determined to be located outside the topic image.

[0055] After obtaining the coordinate comparison results of each sentence, the method 200 can classify the writing topic according to the coordinate comparison results of each sentence at step S2054. Specifically, when the coordinate comparison results of each sentence are all located within the topic map, the type of the writing topic can be determined to be the in-picture text type; when the coordinate comparison results of each sentence are all located outside the topic map, the type of the writing topic can be determined to be the out-picture text type; and when part of the coordinate comparison results of each sentence is located within the topic map and the other part is located outside the topic map, the type of the writing topic can be determined to be a mixed text type.

[0056] After determining the type of the writing topic, the method 200 may proceed to step S206 to generate writing content using a preset model based on the prompt word template and image corresponding to the type. Figure 4 The process of generating writing content using a preset model is described in detail. Figure 4 As shown, at step S2061, method 200 may aggregate multiple sentences with the same coordinate comparison results based on the second coordinates corresponding to each sentence to obtain in-graph sentences and / or out-graph sentences.

[0057] In one embodiment, multiple sentences with the same comparison results can be sorted from small to large according to the ordinate, and connected end to end so as to be aggregated into one sentence. Specifically, multiple sentences with the comparison results being located within the title map can be aggregated into the in-map sentences, and multiple sentences with the comparison results being located outside the title map can be aggregated into the out-map sentences. In actual operation, when sorting multiple sentences from small to large according to the ordinate, the multiple sentences can be sorted from small to large according to the ordinate based on the ordinate of the upper left corner point of the first word in each sentence, or the multiple sentences can be sorted from small to large according to the ordinate based on the ordinate of the lower right corner point of the second word in each sentence.

[0058] Then, at step S2062, the method 200 may combine the image, the sentence in the image, and / or the sentence outside the image with the prompt word template corresponding to the type to obtain the model prompt word. After obtaining the model prompt word, the method 200 may proceed to step S2063 to generate writing content based on the model prompt word using a preset model.

[0059] As can be seen from the foregoing description, the number of input items and the content of the special prompt part of the prompt word template corresponding to different types will be different. In actual operation, combining the image, the sentence in the picture and / or the sentence outside the picture with the prompt word template corresponding to the type is actually combining the specific value of the input item with the prompt word template corresponding to the type. Specifically, in the case where the type is a textless type, the image can be combined with the prompt word template corresponding to the textless type to obtain a model prompt word; in the case where the type is an in-picture text type, the image and the in-picture sentence can be combined with the prompt word template corresponding to the in-picture text type to obtain a model prompt word; in the case where the type is an out-picture text type, the image and the out-picture sentence can be combined with the prompt word model corresponding to the out-picture text type to obtain a model prompt word; and in the case where the type is a mixed text type, the image, the in-picture sentence and the out-picture sentence can be combined with the prompt word template corresponding to the mixed text type to obtain a model prompt word.

[0060] From the previous description, we can see that Figures 1 to 4 All steps of the described method for writing by looking at pictures are performed by a single device, which undoubtedly increases the computing pressure of the device. In order to balance the pressure on the device, in actual applications, some steps that directly interact with the user can be deployed on the client side, and other steps can be deployed on the server side to form a writing by looking at pictures system. Figure 5 The picture-based writing method 500 implemented based on the picture-based writing system is described in detail. Figure 5 As shown, at step S501, the client may capture an image of the writing topic through a camera device, and the image includes a topic map.

[0061] In step S502, the client may send a writing content acquisition request carrying an image with a writing title to the server, so that the server may generate writing content according to the writing content acquisition request and feed back the writing content to the client.

[0062] Correspondingly, at step S503, the server may receive a request for obtaining writing content, and may parse the request for obtaining writing content to obtain an image of the writing topic carried therein.

[0063] Next, at step S504, the server can classify the writing topic according to whether the image of the writing topic contains text and the position of the text in the image if the image contains text, so as to obtain the type of the writing topic. Then, at step S505, the server can generate writing content using a preset model based on the prompt word template and image corresponding to the type.

[0064] After the writing content is generated, at step S506, the server may send the writing content to the client. Correspondingly, at step S507, the client may receive the writing content. Further, at step S508, the client may also output the writing content in an audible or visual manner.

[0065] Combination of the above Figure 5 The picture-based writing method 500 implemented based on the picture-based writing system is described. It is understandable that the description of each embodiment in this disclosure emphasizes the differences between the embodiments, and the same or corresponding parts can be referenced to each other. For the purpose of brevity, this disclosure will not repeat them one by one.

[0066] To facilitate understanding, the method for picture-based writing provided by the present disclosure is described below with reference to an example.

[0067] Step A: Assume that the collected writing topic images are as follows Figure 6 As shown by Figure 6 It can be seen that the writing topic includes 4 title pictures and title captions.

[0068] Step B: Perform optical character recognition on the image to determine that the image contains text and obtain the first coordinates of the text. Here, the obtained texts include: for, not, up,!, no, relationship,!, put, clothes, on, of, soil, shoot, dry, clean.

[0069] Step C: Based on the text and its first coordinate, the text is split according to the principle of one line per sentence to obtain three sentences and the second coordinates corresponding to the sentences. Here, the sentences obtained are: Sentence 1-"Sorry! It's okay!", Sentence 2-"Put the dirt off the clothes", Sentence 3-"Clean up the dirt!".

[0070] Step D: Based on the image, use a preset model to identify the title image contained in the image to obtain the title image and the third coordinate corresponding to the title image.

[0071] Step E: Compare the second coordinates corresponding to each sentence with the third coordinates corresponding to the question map in turn to obtain the coordinate comparison results of each sentence. Sentence 1 - within the question map, sentence 2 - within the question map, sentence 3 - within the question map.

[0072] Step F: Based on the coordinate comparison results of the three sentences, it can be determined that the type of the writing topic is the in-picture text type.

[0073] Step G: Based on the second coordinates corresponding to each sentence, the three sentences with the same coordinate comparison results are aggregated to obtain the sentence in the graph “Sorry! It’s okay! Clean the dirt off your clothes!”.

[0074] Step H: Combine the image and the sentence in the image with the prompt word template corresponding to the text type in the image to obtain the model prompt word. The input item 1 of the prompt word template is the image, and the input item 2 is the sentence in the image "Sorry! It's okay! Clean the picture on the clothes!". The special prompt part is: understand and describe the title image in combination with the sentence in the image, and generate writing content based on the description.

[0075] Step I: Based on the model prompt words, use the preset model to generate writing content.

[0076] Next, combine Figure 7 An electronic device 700 provided by the embodiment of the present disclosure is exemplarily introduced. Figure 7 As shown, the electronic device 700 of the embodiment of the present disclosure may include a processor 701 , a memory 702 , and a communication bus 703 .

[0077] In the process of a specific embodiment, the processor 701 may be at least one of an application-specific integrated circuit (ASIC), a digital signal processor (DSP), a digital signal processing image processing device (DSPD), a programmable logic image processing device (PLD), a field programmable gate array (FPGA), a CPU, a controller, a microcontroller, and a microprocessor. It is understandable that for different devices, the electronic device used to implement the function of the processor may also be other, which is not specifically limited in this embodiment.

[0078] In the embodiment of the present disclosure, the communication bus 703 is used to realize the connection and communication between the processor 701 and the memory 702; the memory 702 stores program instructions for controlling the startup of the device; when the processor 701 executes the program instructions stored in the memory 702, the present disclosure is realized. Figures 1 to 5 The described method for writing from pictures.

[0079] Combination of the above Figure 7An electronic device for picture viewing and writing that can be used to perform the present disclosure is described. It should be understood that the device structure or architecture here is merely exemplary, and the implementation method and implementation entity of the present application are not limited thereto, but can be changed without departing from the spirit of the present application. It can be understood that the description of each embodiment in the present disclosure emphasizes the differences between the various embodiments, and the same or corresponding parts can be referenced to each other. For the purpose of brevity, the present disclosure will not go into details one by one.

[0080] According to the above description in combination with the accompanying drawings, those skilled in the art can also understand that the embodiments of the present application can also be implemented by a software program. Therefore, the present application also provides a computer-readable storage medium. The computer-readable storage medium stores program instructions for controlling the startup of the device, and the program instructions can be used to implement the present disclosure in combination with the software program. Figures 1 to 5 The described method for writing from pictures.

[0081] It should be noted that although the operations of the method of the present application are described in a specific order in the accompanying drawings, this does not require or imply that the operations must be performed in this specific order, or that all the operations shown must be performed to achieve the desired results. On the contrary, the steps depicted in the flow chart can be performed in a different order. Additionally or alternatively, some steps can be omitted, multiple steps can be combined into one step, and / or one step can be decomposed into multiple steps.

[0082] Although multiple embodiments of the present disclosure have been shown and described herein, it will be apparent to those skilled in the art that such embodiments are provided by way of example only. Those skilled in the art may think of many changes, modifications, and alternatives without departing from the thought and spirit of the present disclosure. It should be understood that in the process of practicing the present disclosure, various alternatives to the embodiments of the present disclosure described herein may be adopted. The attached claims are intended to define the scope of protection of the present disclosure, and therefore cover equivalents or alternatives within the scope of these claims.

Claims

1. A method for writing based on pictures, comprising: Collecting an image of a writing topic, wherein the image includes a topic picture; Classifying the writing topic according to whether the image contains text and, if the image contains text, the position of the text in the image to obtain the type of the writing topic; as well as Based on the prompt word template corresponding to the type and the image, a preset model is used to generate writing content.

2. The method according to claim 1, wherein: Classifying the writing topic according to whether the image contains text and, if the image contains text, the position of the text in the image includes: performing optical character recognition on the image to determine whether the image contains text; In the case where the image does not contain any text, determining the type of the writing topic as a text-free type; When the image includes text, obtaining first coordinates of the text; and Based on the text and its first coordinate, the writing topic is classified.

3. The method according to claim 2, wherein: Based on the text and its first coordinate, classifying the writing topic includes: Based on the text and its first coordinates, split the text to obtain a plurality of sentences and second coordinates corresponding to the sentences; Based on the image, using the preset model to identify the title image contained in the image to obtain the title image and the third coordinate corresponding to the title image; Comparing the second coordinates corresponding to each sentence with the third coordinates corresponding to the question graph in sequence to obtain coordinate comparison results for each sentence; and The writing topics are classified according to the coordinate comparison results of the sentences.

4. The method according to claim 3, wherein: Classifying the writing topic according to the coordinate comparison results of the sentences includes: When the coordinate comparison results of the sentences are all located within the topic map, determining the type of the writing topic as a text type within the map; When the coordinate comparison results of the sentences are all outside the topic map, determining the type of the writing topic as an out-of-map text type; and When part of the coordinate comparison results of the sentences is located within the topic map and the other part is located outside the topic map, it is determined that the type of the writing topic is a mixed text type.

5. The method according to claim 4, wherein: Based on the prompt word template corresponding to the type and the image, using a preset model to generate writing content includes: Based on the second coordinates corresponding to the sentences, a plurality of sentences having the same coordinate comparison results are aggregated to obtain in-graph sentences and / or out-graph sentences; combining the image, the sentence in the image and / or the sentence outside the image with the prompt word template corresponding to the type to obtain a model prompt word; and Based on the model prompt words, the writing content is generated using the preset model.

6. The method according to claim 5, wherein: The prompt word template includes at least one input item, a special prompt part and a general prompt part, and the general prompt part in the prompt word templates corresponding to different types is the same, and the input item and the special prompt part are different; as well as In the case where the type is a text-in-picture type, the input item includes a title picture and a sentence in the picture, and the special prompt part is used to instruct the preset model to generate the writing content according to the title picture and the sentence in the picture; In the case where the type is a non-picture text type, the input item includes a title picture and a non-picture sentence, and the special prompt part is used to instruct the preset model to generate the writing content according to the title picture and the non-picture sentence; In the case where the type is a mixed text type, the input items include a title image, sentences outside the image, and sentences inside the image, and the special prompt part is used to instruct the preset model to generate the writing content based on the title image, sentences outside the image, and sentences inside the image.

7. The method according to claim 3, wherein: Based on the text and its first coordinate, splitting the text to obtain a plurality of sentences and second coordinates corresponding to the sentences includes: Based on the characters and their first coordinates, characters with the same vertical coordinates are classified into one sentence, and multiple characters in each sentence are sorted from small to large according to the horizontal coordinates to obtain multiple sentences and the second coordinates corresponding to the sentences.

8. An electronic device comprising: processor; and a memory storing program instructions for writing by looking at pictures, wherein when the program instructions are executed by the processor, the electronic device executes the method according to any one of claims 1-7.

9. A computer-readable storage medium storing program instructions for writing by looking at pictures, wherein when the program instructions are executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images