Cboth generation method and electronic equipment
By fine-grained division of the target pictures in electronic devices and using the neural network model corresponding to the category to generate copywriting, the problem of low compatibility between copywriting and picture content in the prior art is solved, and a higher matching degree is achieved.
Patent Information
- Application Number
- CN202310595937.7
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2023-05-23
- Publication Date
- 2024-11-26
AI Technical Summary
In the prior art, the copywriting generated in the copywriting generation solution has little compatibility with the picture content and cannot meet user needs.
By fine-grained division of the target images in electronic devices and using neural network models corresponding to the image categories to generate copywriting, the matching degree between copywriting and image content is improved.
The generated copywriting is more in line with the image content, improving the compatibility between the copywriting and the image content, and meeting users' needs for the matching degree of copywriting and the image.
Smart Images

Figure CN119026574A_ABST
Abstract
Description
Technical Field
[0001] The embodiments of the present application relate to the field of electronic technology, and in particular to a document generation method and an electronic device. Background Art
[0002] With the rapid development of Internet technology, users can share their daily lives on various social platforms. For example, users can share music, movies, pictures, etc. on social platforms. Social platforms can be, for example, social websites, social applications, etc.
[0003] Users can share pictures on social platforms and attach a text related to the picture, or users can only share the text on social platforms. The text can be edited by the user. With the development of artificial intelligence, related technologies can automatically generate text based on the pictures provided by users through neural network models or databases, providing convenience for users.
[0004] However, in the copywriting generation solution provided by the prior art, the image category is first identified, and then the copywriting is generated based on the image category. In this solution, each category of image corresponds to a fixed copywriting, so the fit between the generated copywriting and the image content cannot meet the user's needs. Summary of the invention
[0005] The embodiments of the present application provide a text generation method and an electronic device, which are used to solve the problem that the text generated by the electronic device does not match the image, and to improve the fit between the text and the image content.
[0006] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:
[0007] In a first aspect, a method for generating a copy is provided, and the method is applied to an electronic device. The electronic device has a display screen. The method may include: the electronic device displays a first interface, the first interface includes a target image and a first button for triggering the electronic device to generate a copy of the target image; the electronic device responds to the user's first operation of the first button on the first interface, and the electronic device can classify the target image to obtain the category of the target image; thereafter, the electronic device can input the target image into a neural network model corresponding to the category of the target image to obtain the copy of the target image; wherein different categories correspond to different neural network models, and the neural network model has the ability to generate copy.
[0008] In this solution, before generating the copy, the electronic device can first perform fine-grained division of the target image used to generate the copy. Moreover, each type of image has a corresponding neural network model for outputting the copy of this type of image. The electronic device can input the target image into the neural network model corresponding to its category according to the category of the target image to obtain the copy of the target image. Since the neural network model corresponding to the category of the target image has the ability to generate the copy of this type of image, the copy generated in this way can be more in line with the target image, and the copy and the target image have a higher matching degree, which can improve the fit between the copy and the image content.
[0009] In a possible implementation of the first aspect, after the electronic device outputs the target image to the neural network model corresponding to the category of the target image and obtains the text of the target image, the electronic device can save the text of the target image in the clipboard of the electronic device. In this way, the electronic device can paste the text in the text editing area required by the user in response to the user's paste operation.
[0010] In a possible implementation of the first aspect, the electronic device may divide the target image into a foreground image or a background image. The foreground image may, for example, be an image that focuses more on the subject. The background image may, for example, be an image that focuses more on the whole. The electronic device may first divide the target image into a foreground image or a background image. Afterwards, the electronic device may input the target image into a neural network model corresponding to the foreground image or a neural network model corresponding to the background image to obtain the text of the target image. In this way, the fit between the text and the image content can be improved.
[0011] In a possible implementation of the first aspect, the foreground image corresponds to the first neural network model, and the background image corresponds to the second neural network model. The first neural network model has the ability to output the text of the foreground image. The second neural network model has the ability to output the text of the background image. After the electronic device classifies the target image, if the target image is a foreground image, the electronic device can input the target image into the first neural network model to obtain the text of the target image. If the target image is a background image, the electronic device can input the target image into the second neural network model to obtain the text of the target image.
[0012] In a possible implementation of the first aspect, the degree of fit between the text output by the first neural network model when recognizing a foreground image and the foreground image is higher than the degree of fit between the text output by the second neural network model when recognizing a foreground image and the foreground image. The degree of fit between the text output by the second neural network model when recognizing a background image and the background image is higher than the degree of fit between the text output by the first neural network model when recognizing a background image and the background image.
[0013] In a possible implementation manner of the first aspect, the electronic device may obtain the category of the target picture according to the position of at least one subject in the target picture and a preset position in the target picture.
[0014] Specifically, the electronic device may first identify the position of at least one subject in the target image. Afterwards, the electronic device may obtain the category of the target image based on one or more of the degree of overlap between the position of each subject in the at least one subject and the preset position, the distance between the position of each subject in the at least one subject and the preset position, and the area of each subject in the at least one subject in the target image. The implementation method provides a possible implementation manner for an electronic device to obtain the category of a target image.
[0015] In a possible implementation of the first aspect, the position of the subject may be the position of a minimum bounding box surrounding the subject, and the position of the minimum bounding box may represent the position of the subject in the target image. The preset position in the target image may be the position of a preset bounding box. In this way, the electronic device can obtain the category of the target image according to the position of the bounding box of at least one subject in the target image and the position of the preset bounding box.
[0016] Specifically, the electronic device may calculate the first index of each subject in the at least one subject by using the following formula according to the position of the bounding box of the at least one subject and the position of the preset bounding box.
[0017] The first indicator = a*IOU+b*D+c*AREA. IOU is the intersection-over-union ratio of the bounding box of the corresponding subject and the preset bounding box, and IOU indicates the overlap between the bounding box of the corresponding subject and the preset bounding box. D is the distance from the center point of the bounding box of the corresponding subject to the center point of the preset bounding box. AREA is the area of the bounding box of the corresponding subject.
[0018] If the first index of each of the at least one subject is less than the first threshold, the electronic device identifies the target image as a background image. If the first index of any one or more of the at least one subject is greater than the first threshold, the electronic device identifies the target image as a foreground image.
[0019] In a possible implementation of the first aspect, the electronic device may also receive a closed area circled by the user on the target image, and the electronic device may obtain the category of the target image according to the closed area circled by the user on the target image.
[0020] Specifically, the electronic device may receive a second operation of the user on the target image, where the second operation is used to circle a closed area on the target image. In response to the second operation of the user, the electronic device obtains the position of the closed area in the target image. Afterwards, the electronic device may obtain the category of the target image based on one or more of the degree of overlap between the position of each subject in at least one subject in the target image and the position of the closed area, the distance between the position of each subject in at least one subject and the position of the closed area, and the area of each subject in at least one subject in the target image. This implementation method provides a possible implementation manner in which an electronic device obtains the category of a target image.
[0021] In a possible implementation of the first aspect, the position of the subject may be the position of a minimum bounding box surrounding the subject, and the position of the minimum bounding box may represent the position of the subject in the target image. The position of the closed area in the target image may be the position of the custom bounding box. In this way, the electronic device can obtain the category of the target image according to the position of the bounding box of at least one subject in the target image and the position of the custom bounding box.
[0022] Specifically, the electronic device may calculate the second index of each subject in the at least one subject by using the following formula according to the position of the bounding box of the at least one subject and the position of the custom bounding box.
[0023] The second indicator = a*IOU'+b*D'+c*AREA, where IOU' is the intersection-over-union ratio of the bounding box of the corresponding subject and the custom bounding box, IOU' indicates the overlap between the bounding box of the corresponding subject and the custom bounding box, D' is the distance from the center point of the bounding box of the corresponding subject to the center point of the custom bounding box, and AREA is the area of the bounding box of the corresponding subject.
[0024] If the second index of each of the at least one subject is less than the first threshold, the electronic device identifies the target image as a background image. If the second index of any one or more of the at least one subject is greater than the first threshold, the electronic device identifies the target image as a foreground image.
[0025] In a possible implementation manner of the first aspect, the electronic device may also generate a text that meets the user's style requirements according to the style selected by the user.
[0026] Specifically, after the electronic device responds to the user's first operation on the first button on the first interface, the electronic device may also display the second interface. The second interface may include at least two style buttons. The electronic device may receive a click operation for a target button among the at least two style buttons. In response to the click operation, the electronic device may input the target image into a neural network model corresponding to the category of the target image and the target copywriting style to obtain the copywriting of the target image; wherein the target copywriting style corresponds to the target button. In this way, the electronic device can generate copywriting that meets the user's style requirements.
[0027] In a possible implementation of the first aspect, after the electronic device obtains the text of the target image, the electronic device may further display a third interface. The third interface may be used to preview the text. The third interface may include a regenerate button for triggering the electronic device to regenerate the text. In a text generation scheme provided in an embodiment of the present application, the electronic device is able to display the text on a text preview interface. The text preview interface may include a regenerate button. When a user is not satisfied with the current text, in response to the user clicking the button, the electronic device may regenerate the text.
[0028] In a possible implementation of the first aspect, before the electronic device displays the first interface, the electronic device may first display the fourth interface. The fourth interface includes thumbnails of multiple pictures, wherein the multiple pictures include multiple target pictures. In response to a user clicking or long pressing the thumbnail of the target picture in the fourth interface, the electronic device displays the first interface. This implementation provides a possible implementation of triggering the electronic device to jump from the fourth interface to the first interface.
[0029] In a possible implementation of the first aspect, the electronic device may first display a fifth interface, which is a picture and text sharing interface of a social software. In response to a user clicking on a picture and text sharing button in the fifth interface, the electronic device displays a fourth interface. Afterwards, in response to a user clicking or long pressing on a thumbnail of a target picture in the fourth interface, the electronic device displays the first interface. This implementation provides a possible implementation of triggering the electronic device to jump from the picture and text sharing interface of the social software to the first interface.
[0030] In a second aspect, the present application provides an electronic device, which includes: a memory, a display screen, and one or more processors. The memory, the display screen and the processor are coupled. The memory is used to store computer program code, and the computer program code includes computer instructions. When the computer instructions are executed by the processor, the electronic device performs the following steps: displaying a first interface, the first interface includes a target image and a first button, the first button is used to trigger the electronic device to generate a text for the target image; in response to the user's first operation of the first button in the first interface, classifying the target image to obtain the category of the target image; inputting the target image into a neural network model corresponding to the category of the target image to obtain the text of the target image; wherein different categories correspond to different neural network models, and the neural network model has the ability to generate text.
[0031] In conjunction with the second aspect, in a possible design, when the computer instruction is executed by the processor, the electronic device performs the following steps: after the electronic device inputs the target image into the neural network model corresponding to the category of the target image and obtains the text of the target image, the electronic device saves the text of the target image in the clipboard of the electronic device.
[0032] In conjunction with the second aspect, in a possible design, the category of the target image is a foreground image or a background image.
[0033] In conjunction with the second aspect, in a possible design method, the foreground image corresponds to the first neural network model, and the background image corresponds to the second neural network model. Among them, the first neural network model has the ability to output the text of the foreground image, and the second neural network model has the ability to output the text of the background image. When the above computer instructions are executed by the processor, the electronic device also performs the following steps: if the target image is a foreground image, the electronic device inputs the target image into the first neural network model to obtain the text of the target image; if the target image is a background image, the electronic device inputs the target image into the second neural network model to obtain the text of the target image.
[0034] In combination with the second aspect, in one possible design method, when the above-mentioned computer instructions are executed by the processor, the electronic device further performs the following steps: the electronic device responds to the first operation, and the electronic device identifies the position of at least one subject in the target picture; the electronic device identifies whether the target picture is a foreground picture or a background picture based on one or more of the degree of overlap between the position of each subject in at least one subject and the preset position in the target picture, the distance between the position of each subject in at least one subject and the preset position, and the area of each subject in at least one subject in the target picture.
[0035] In combination with the second aspect, in a possible design, the position of each subject in at least one subject is the position of the minimum bounding box surrounding the corresponding subject, the position of the bounding box is used to indicate the position of the subject in the target image, and the preset position in the target image is the position of the preset bounding box. When the above computer instructions are executed by the processor, the electronic device further performs the following steps: the electronic device calculates the first index of each subject in at least one subject according to the position of the bounding box of at least one subject and the position of the preset bounding box using the following formula; wherein the first index = a*IOU+b*D+c*AREA, IOU is the intersection-and-union ratio of the bounding box of the corresponding subject and the preset bounding box, IOU indicates the overlap between the bounding box of the corresponding subject and the preset bounding box, D is the distance from the center point of the bounding box of the corresponding subject to the center point of the preset bounding box, and AREA is the area of the bounding box of the corresponding subject; if the first index of each subject in at least one subject is less than the first threshold, the electronic device identifies the target image as a background image; if the first index of any one or more subjects in at least one subject is greater than the first threshold, the electronic device identifies the target image as a foreground image.
[0036] In combination with the second aspect, in one possible design method, when the above-mentioned computer instructions are executed by the processor, the electronic device further performs the following steps: the electronic device receives a second operation of the user on the target image, and the second operation is used to circle a closed area on the target image; the electronic device obtains the position of the closed area in the target image in response to the second operation; the electronic device identifies the position of at least one subject in the target image in response to the first operation; the electronic device identifies whether the target image is a foreground image or a background image based on one or more of the degree of overlap between the position of each subject in at least one subject and the position of the closed area, the distance between the position of each subject in at least one subject and the position of the closed area, and the area of each subject in at least one subject in the target image.
[0037] In combination with the second aspect, in a possible design, the position of each subject in at least one subject is the position of the minimum bounding box surrounding the corresponding subject, the position of the bounding box is used to indicate the position of the subject in the target image, and the position of the closed area is the position of the custom bounding box. When the above computer instructions are executed by the processor, the electronic device further performs the following steps: the electronic device calculates the second index of each subject in at least one subject according to the position of the bounding box of at least one subject and the position of the custom bounding box using the following formula; wherein the second index = a*IOU'+b*D'+c*AREA, IOU' is the intersection-over-union ratio of the bounding box of the corresponding subject and the custom bounding box, IOU' indicates the overlap between the bounding box of the corresponding subject and the custom bounding box, D' is the distance from the center point of the bounding box of the corresponding subject to the center point of the custom bounding box, and AREA is the area of the bounding box of the corresponding subject; if the second index of each subject in at least one subject is less than the first threshold, the electronic device identifies the target image as a background image; if the second index of any one or more subjects in at least one subject is greater than the first threshold, the electronic device identifies the target image as a foreground image.
[0038] In combination with the second aspect, in one possible design method, when the above-mentioned computer instructions are executed by the processor, the electronic device further performs the following steps: the electronic device displays a second interface in response to the first operation; wherein the second interface includes at least two style buttons, each style button corresponding to a copywriting style; the electronic device receives a user click operation on a target button among the at least two style buttons; the electronic device responds to the click operation, inputs the target image into a neural network model corresponding to the category of the target image and the target copywriting style, and obtains the copywriting of the target image; wherein the target copywriting style corresponds to the target button.
[0039] In combination with the second aspect, in one possible design method, when the above-mentioned computer instructions are executed by the processor, the electronic device also performs the following steps: the electronic device displays a third interface, the third interface is used to preview the copy, the third interface includes a regenerate button, and the regenerate button is used to trigger the electronic device to regenerate the copy.
[0040] In combination with the second aspect, in one possible design method, when the above-mentioned computer instructions are executed by the processor, the electronic device also performs the following steps: the electronic device displays a fourth interface; wherein the fourth interface includes thumbnails of multiple pictures, and the multiple pictures include multiple target pictures; in response to the user's third operation on the thumbnail of the target picture in the fourth interface, the electronic device displays the first interface; the third operation is a click or long press operation on the thumbnail of the target picture.
[0041] In combination with the second aspect, in one possible design method, when the above-mentioned computer instructions are executed by the processor, the electronic device also performs the following steps: the electronic device displays a fifth interface; wherein the fifth interface is a picture and text sharing interface of the social software; in response to the user clicking the picture and text sharing button in the fifth interface, the electronic device displays the fourth interface.
[0042] In a third aspect, the present application provides a chip system, which can be applied to an electronic device including a memory. The chip system includes one or more interface circuits and one or more processors. The interface circuit and the processor are interconnected by a line. The interface circuit is used to receive a signal from the above-mentioned memory and send the signal to the processor, and the signal includes a computer instruction stored in the memory. When the processor executes the computer instruction, the electronic device executes the method as in the first aspect and any possible design thereof.
[0043] In a fourth aspect, the present application provides a computer-readable storage medium, the computer-readable storage medium comprising computer instructions. When the computer instructions are executed on an electronic device, the electronic device executes the method of the first aspect and any possible design thereof.
[0044] In a fifth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the method of the first aspect and any possible design thereof.
[0045] It can be understood that the beneficial effects that can be achieved by the electronic device of any possible design of the second aspect, the chip system of the third aspect, the computer-readable storage medium of the fourth aspect, and the computer program product of the fifth aspect can be referred to as the beneficial effects in the first aspect and any possible design of the electronic device, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS
[0046] Figure 1 A schematic diagram of a picture and text provided in an embodiment of the present application;
[0047] Figure 2 A schematic diagram of the structure of an electronic device provided in an embodiment of the present application;
[0048] Figure 3 A schematic diagram of a software architecture of an electronic device provided in an embodiment of the present application
[0049] Figure 4 A schematic diagram of a human-computer interaction interface provided in an embodiment of the present application;
[0050] Figure 5 A schematic diagram of another human-computer interaction interface provided in an embodiment of the present application;
[0051] Figure 6 A schematic diagram of another human-computer interaction interface provided in an embodiment of the present application;
[0052] Figure 7 A flowchart of a method for generating a document provided in an embodiment of the present application;
[0053] Figure 8 A flowchart of another method for generating a document provided in an embodiment of the present application;
[0054] Fig. 9 A schematic diagram of a subject bounding box and a preset bounding box in a target image provided in an embodiment of the present application;
[0055] Fig.10 A schematic diagram of a human-computer interaction interface for publishing text in a social software provided in an embodiment of the present application;
[0056] Fig.11 A schematic diagram of another human-computer interaction interface for posting text on social software provided in an embodiment of the present application;
[0057] Fig.12 A schematic diagram of another human-computer interaction interface provided in an embodiment of the present application;
[0058] Fig.13 A schematic diagram of a document preview interface provided in an embodiment of the present application;
[0059] Fig.14 A schematic diagram of a human-computer interaction interface for regenerating text based on a picture provided in an embodiment of the present application;
[0060] Fig.15 A schematic diagram of a human-computer interaction interface for selecting a copywriting style provided in an embodiment of the present application;
[0061] Fig.16 A flowchart of another method for generating a document provided in an embodiment of the present application;
[0062] Fig.17 A schematic diagram of a human-computer interaction interface for generating text according to a style provided in an embodiment of the present application;
[0063] Fig.18 A schematic diagram of a human-computer interaction interface for regenerating text according to a style provided in an embodiment of the present application;
[0064] Fig.19 A flowchart of another method for generating a document provided in an embodiment of the present application;
[0065] Fig. 20A schematic diagram of a subject bounding box and a custom bounding box in a target image provided in an embodiment of the present application;
[0066] Fig.21 A schematic diagram of another human-computer interaction interface provided in an embodiment of the present application;
[0067] Fig. 22 A schematic diagram of another method for generating text provided in an embodiment of the present application;
[0068] Fig.23 A schematic diagram of another method for generating text provided in an embodiment of the present application;
[0069] Fig.24 A schematic diagram of another human-computer interaction interface provided in an embodiment of the present application;
[0070] Fig.25 A schematic diagram of another human-computer interaction interface provided in an embodiment of the present application. DETAILED DESCRIPTION
[0071] With the rapid development of Internet technology, users can share their daily lives on various social platforms. For example, users can share music, movies, pictures, etc. on social platforms. Social platforms can be, for example, social websites, social applications, etc. Terminals such as mobile phones can have built-in social applications. Users can share their daily lives in social applications anytime and anywhere through their mobile phones.
[0072] Usually, users can share pictures on social platforms and attach text corresponding to the pictures. Users can also share only the text of the pictures on social platforms. The text can be a paragraph of text related to the picture. For example, the text can be a paragraph of text description related to the image content displayed in the picture. Figure 1 As shown, Figure 1 Two pictures and the texts corresponding to the pictures are shown. Picture 1 shows flowers, and the text corresponding to picture 1 can be a paragraph of text related to flowers. Picture 2 shows a landscape including a river, and the text corresponding to picture 2 can be a paragraph of text related to rivers and scenery.
[0073] The copy can be edited by the user. With the development of artificial intelligence, related technologies can automatically generate copy based on the pictures provided by the user through neural network models or databases, providing convenience for users.
[0074] As an example, the database stores a variety of edited texts. For example, the database stores texts corresponding to the types of objects in a variety of pictures, wherein each type corresponds to a fixed number of texts.
[0075] After an electronic device such as a mobile phone obtains a picture provided by a user, the picture can be sent to a classification network. The classification network is used to obtain the type of object shown in the picture based on the picture.
[0076] After that, the mobile phone can match and search in the database according to the type of object in the picture and the auxiliary information input by the user, obtain relatively matching copy, and provide it to the user. Among them, the auxiliary information input by the user can be, for example, the category information that the user wants. Since the copy is selected from the edited copy in the database, and the number of copy in the database is limited, there may be a problem that the copy does not match the picture.
[0077] As another example, after an electronic device such as a mobile phone obtains a picture provided by a user, the picture can be input into a neural network model for generating text. The neural network model has the ability to generate text based on the picture, so that the text can be automatically generated. However, the text output by the neural network model may also have the problem of not matching the picture. This is because the image content displayed by the picture is diverse. In order to obtain text that matches the picture, a large number of training samples are required to train the neural network model, which requires a lot of manpower and material resources.
[0078] That is to say, although the related technology provides a solution for automatically generating text using a neural network model or a database, there is a problem in the related technology that the text is not consistent with the image content in the picture.
[0079] To this end, an embodiment of the present application provides a method for generating copy and an electronic device. The method is applied to an electronic device. Before generating the copy, the electronic device can first perform fine-grained division on the target image used to generate the copy. Moreover, each type of image has a corresponding neural network model for outputting the copy of this type of image. The electronic device can input the target image into the neural network model corresponding to the category according to the category of the target image to obtain the copy of the target image. Since the neural network model has the ability to output the copy of the category of images, the copy generated in this way can be more in line with the target image, the copy and the target image have a higher degree of matching, and the fit between the copy and the image content can be improved.
[0080] The method provided in the embodiments of the present application can be applied to electronic devices with data processing capabilities. The above-mentioned electronic devices may include servers, mobile phones, tablet computers, laptops, personal computers (PCs), ultra-mobile personal computers (UMPCs), handheld computers, netbooks, smart home devices (e.g., smart TVs, smart screens, large screens, smart speakers, smart air conditioners, etc.), personal digital assistants (PDAs), wearable devices (e.g., smart watches, smart bracelets, etc.), vehicle-mounted devices, virtual reality devices, etc., and the embodiments of the present application do not impose any restrictions on this. In the embodiments of the present application, the above-mentioned electronic devices are electronic devices that can run operating systems and install applications. Optionally, the operating system running on the electronic device may be system, system, System, etc.
[0081] For example, please refer to Figure 2 , which shows a schematic diagram of the structure of an electronic device 200. The electronic device 200 may include a processor 210, an external memory interface 220, an internal memory 221, an audio module 230, a speaker 230A, a microphone 230B, a display screen 240, a communication module 250, a power module 260, an input device 270, a sensor module 280, etc. Among them, the sensor module 280 may include a pressure sensor, a touch sensor, etc.
[0082] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.
[0083] The processor 210 may include one or more processing units. For example, the processor 210 may include an application processor (AP), a modem processor, a graphics processor (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), and / or a neural-network processing unit (NPU). Different processing units may be independent components or integrated into one or more processors. In some embodiments, the electronic device 200 may also include one or more processors 210.
[0084] The controller is the nerve center and command center of the electronic device 200. It can generate operation control signals according to the instruction operation code and timing signal to complete the control of fetching and executing instructions.
[0085] The application processor can run the operating system of the electronic device 200, which is used to manage the hardware and software resources of the electronic device 200. For example, it manages and configures memory, determines the priority of system resource supply and demand, controls input and output devices, operates the network, manages the file system, manages drivers, etc. The operating system can also be used to provide an operation page for users to interact with the system. Among them, various types of software can be installed in the operating system, such as drivers, applications (applications, Apps), etc.
[0086] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. NPU can realize applications such as intelligent cognition of electronic device 200. For example, in an embodiment of the present application, the electronic device can use a neural network model to automatically generate text based on a picture.
[0087] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.
[0088] In some embodiments, the processor 210 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a USB interface, etc.
[0089] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present application is only a schematic illustration and does not constitute a structural limitation on the electronic device 200. In other embodiments of the present application, the electronic device 200 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.
[0090] The external memory interface 220 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 200. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function, such as storing music, video and other files in the external memory card.
[0091] The internal memory 221 can be used to store one or more computer programs, which include instructions. The processor 210 can enable the electronic device 200 to execute the application operation method provided in some embodiments of the present application, as well as various applications and data management, etc. by running the above instructions stored in the internal memory 221. The internal memory 221 may include a code storage area and a data storage area. Among them, the data storage area can store data created during the use of the electronic device 200, etc. In addition, the internal memory 221 may include a high-speed random access memory, and may also include a non-volatile memory, such as one or more disk storage components, flash memory components, universal flash storage (universal flash storage, UFS), etc. In some embodiments, the processor 210 can enable the electronic device 200 to execute the application operation method provided in the embodiment of the present application, as well as other applications and data management by running instructions stored in the internal memory 221, and / or instructions stored in a memory provided in the processor 210. For example, in an embodiment of the present application, after the electronic device 200 generates the text of the target image, the text can be stored in the memory.
[0092] The electronic device 200 can implement audio functions through the audio module 230, the speaker 230A, the microphone 230B, and the application processor, etc. For example, music playing, recording, etc. The audio module 230 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 230 can also be used to encode and decode audio signals. In some embodiments, the audio module 230 can be arranged in the processor 210, or some functional modules of the audio module 230 can be arranged in the processor 210.
[0093] The speaker 230A, also called a "horn", is used to convert audio electrical signals into sound signals.
[0094] The microphone 230B, also called a "microphone" or "microphone", is used to convert a sound signal into an electrical signal. A user can make a sound by approaching the microphone 230B with his or her mouth, and input the sound signal into the microphone 230B.
[0095] The communication function of the electronic device 200 can be realized through the antenna 1, the antenna 2 and the communication module 250.
[0096] The communication module 250 can provide solutions for wireless communications including cellular, Wi-Fi, Bluetooth (BT), wireless data transmission modules (e.g., 433MHz, 868MHz, 915MHz) applied to the electronic device 200. The communication module 250 can be one or more devices integrating at least one communication processing module. The communication module 250 receives electromagnetic waves via antenna 1 or antenna 2, filters and frequency modulates the electromagnetic wave signals, and sends the processed signals to the processor 210. The communication module 250 can also receive the signal to be sent from the processor 210, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through antenna 1 or antenna 2.
[0097] The electronic device 200 implements the display function through a GPU, a display screen 240, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 240 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs, which execute program instructions to generate or change display information.
[0098] The display screen 240 is used to display images, videos, etc. The display screen 240 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode or an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), Miniled, MicrOLED, Micro-OLED, quantum dot light-emitting diodes (QLED), etc. In some embodiments, the electronic device 200 may include 1 or N display screens 240, where N is a positive integer greater than 1. In the embodiment of the present application, the display screen 240 can be used to display a UI and receive user operations on the UI.
[0099] In some embodiments, a pressure sensor, a touch sensor, etc. are provided on the display screen 240. The pressure sensor is used to sense the pressure signal and can convert the pressure signal into an electrical signal. When a touch operation acts on the display screen 240, the electronic device 200 detects the strength of the touch operation according to the pressure sensor. The electronic device 200 can also calculate the position of the touch according to the detection signal of the pressure sensor. The touch sensor, also known as a "touch panel", can form a touch screen, also known as a "touch screen", with the display screen 240. The touch sensor is used to detect a touch operation acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can also be provided through the display screen 240. For example, in an embodiment of the present application, the display screen 240 can display a first interface and receive a first operation of a user on a first button on the first interface. The display screen 240 can also display a second interface and receive a click operation of a user on a target style button. The display screen 240 can also display a third interface, and the third interface includes a text generated by the electronic device 200. The display screen 240 can also display a fourth interface, which includes thumbnails of multiple pictures and accepts a user's click or long press operation on a thumbnail of a target picture. The display screen 240 can also display a fifth interface, which can be a picture and text sharing interface of a social software, and the display screen 240 can receive a user's click operation on a picture and text sharing button on the fifth interface.
[0100] The power module 260 can be used to supply power to various components included in the electronic device 200. In some embodiments, the power module 260 can be a battery, such as a rechargeable battery.
[0101] The input device 270 may include a keyboard, a mouse, etc. The keyboard is used to input English letters, numbers, punctuation marks, etc. into the electronic device 200, thereby issuing commands to the electronic device 200, inputting data, etc. The mouse is an indicator for the vertical and horizontal coordinates of the display system of the electronic device 200, and is used to input instructions, etc. to the electronic device 200. Among them, the input device 270 can be connected to the electronic device 200 through a wired connection, for example, the input device 270 is connected to the electronic device 200 through a GPIO interface, a USB interface, etc. The input device 270 can also be connected to the electronic device 200 wirelessly, for example, the input device 270 is connected to the electronic device 200 through Bluetooth, infrared, etc.
[0102] Take the above electronic device 200 as a mobile phone as an example. The software system of the electronic device 200 can adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. The embodiment of the present invention takes the Android system of the layered architecture as an example to illustrate the software structure of the electronic device 200.
[0103] Figure 3It is a software structure block diagram of the electronic device 200 according to an embodiment of the present invention.
[0104] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom, namely, the application layer, the application framework layer, the Android runtime (Android runtime), the system library, and the kernel layer.
[0105] The application layer can include a series of application packages.
[0106] like Figure 3 As shown, the application package may include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, social and other applications.
[0107] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.
[0108] like Figure 3 As shown, the application framework layer may include a content provider, a view system, a resource manager, a notification manager, an input system, a text generation module, an image classification module, and the like.
[0109] The input system is used to monitor the input module of the mobile phone (such as the touch screen driver), and convert the parameters input by the input module into usable events and pass them to the relevant modules in the upper layer. For example, the input system is used to monitor the touch screen of the mobile phone through the touch screen driver, and convert the touch parameters generated by the touch operation input by the touch screen into usable events and pass them to the upper layer APP.
[0110] Content providers are used to store and retrieve data and make it accessible to applications. The data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.
[0111] The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. A display page can be composed of one or more views. For example, a display page including a text notification icon can include a view for displaying text and a view for displaying pictures. For example, in the present application, the visual system can be used to construct a first interface, which includes a target image and a first button. The visual system can also be used to construct a second interface, which includes at least two style buttons. The visual system can also be used to construct a third interface, in which text generated by the electronic device 200 is displayed.
[0112] The copy generation module is used to generate copy in response to the call of the application. Taking the gallery application as an example, the input system of the electronic device 200 generates a first event corresponding to the operation of triggering the electronic device 200 to generate copy in response to the user inputting the touch screen to trigger the electronic device 200 to generate copy (such as the first operation described in the embodiment of the present application), and sends the first event to the gallery application. In response to the first event, the gallery application calls the copy generation module to generate copy.
[0113] Specifically, the copy generation module may include multiple submodules, one of which has the ability to generate copy for a category of images. The gallery application may first call the image classification module to classify the target image and obtain the category of the target image. Afterwards, the gallery application may call the copy generation submodule corresponding to the category of the target image according to the category of the image to generate the copy for the target image.
[0114] In some embodiments, the image classification module may output the category of the target image based on the target detection network model. The text generation module may output the text of the target image based on the neural network model for generating image text.
[0115] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.
[0116] The notification manager enables applications to display notification information in the status bar. It can be used to convey notification-type messages and can disappear automatically after a short stay without user interaction. For example, the notification manager is used to notify download completion, message reminders, etc. The notification manager can also be a notification that appears in the system top status bar in the form of a chart or scroll bar text, such as notifications of applications running in the background, or a notification that appears on the display in the form of a dialog window. For example, a text message is displayed in the status bar, a prompt sound is emitted, an electronic device vibrates, an indicator light flashes, etc.
[0117] Android Runtime includes core libraries and virtual machines. Android Runtime is responsible for scheduling and management of the Android system.
[0118] The core library consists of two parts: one part is the function that needs to be called by the Java language, and the other part is the Android core library.
[0119] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.
[0120] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.
[0121] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.
[0122] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.
[0123] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.
[0124] A 2D graphics engine is a drawing engine for 2D drawings.
[0125] The kernel layer is the layer between hardware and software. The kernel layer can contain display drivers, audio drivers, sensor drivers, etc.
[0126] In conjunction with the accompanying drawings, a method for generating text provided by an embodiment of the present application is described below, taking the electronic device as a mobile phone as an example. In the embodiment of the present application, the mobile phone can generate text corresponding to the picture based on the picture. The picture can be a photo taken by the user, a picture downloaded by the user from the Internet, a picture stored in the cloud by the user, or a screenshot of the user.
[0127] As mentioned above, the mobile phone may include a gallery application. The gallery application may include display interfaces in various forms, and one of the display interfaces may include a text generation button and a target image. In response to the user's operation on the text generation button on the interface, the mobile phone may generate a text for the target image. The text generation button is used to trigger the mobile phone to generate a text for the target image.
[0128] For example, the large picture preview interface of the gallery application may include a copy generation button. Specifically, the mobile phone may display a main interface, which may include an application icon of the gallery application. In response to the user clicking the application icon of the gallery application, the mobile phone may display a text generation button. Figure 4The list display interface 401 shown. The list display interface 401 may include thumbnails of one or more pictures. For example, the list display interface 401 may include a thumbnail of picture 1. In response to a user clicking on the thumbnail of picture 1, the mobile phone may jump to a large picture preview interface 402. The large picture preview interface 402 may display picture 1 (target picture) in large picture mode. The large picture preview interface 402 may include a copy generation button 403 (first button). That is, the large picture preview interface 402 includes a target picture and a first button.
[0129] For another example, the list selection interface of the gallery application may include a copy generation button. Figure 5 As shown, in response to the user long pressing the thumbnail of picture 1 on the list display interface 401, the mobile phone can display the list selection interface 501. The list selection interface 501 includes thumbnails of multiple pictures, among which the thumbnail of picture 1 has been selected (that is, picture 1 has been selected). The list selection interface 501 can also include a text generation button 502. Since picture 1 has been selected, the mobile phone receives the user clicking the text generation button 502, and the mobile phone can generate a text corresponding to picture 1. That is, the list selection interface 501 can include the target picture and the first button.
[0130] It should be understood that in response to a user long pressing the thumbnail of any picture on the list display interface 401, the mobile phone can jump from the list display interface 401 to the list selection interface 501. For example, in response to a user long pressing the thumbnail of picture 2 on the list display interface 401, the mobile phone can display the list selection interface 501. At this time, the thumbnail of picture 2 is selected on the list selection interface 501. That is, the target picture is picture 2 at this time, and in response to the user clicking the copy generation button, the mobile phone can generate the copy for picture 2. If the mobile phone responds to the user clicking the thumbnail of picture 1 on the list selection interface 501, the mobile phone can reselect picture 1. In addition, when the mobile phone receives the user clicking the copy generation button, the mobile phone can generate the copy for picture 1.
[0131] That is to say, the mobile phone can display a first interface including a target image and a first button. The target image is a picture used to generate a copy. The first button, such as a copy generation button, is used to trigger the mobile phone to generate a copy for the target image. The first interface can be, for example, a large picture preview interface 402 or a list selection interface 501 of a gallery application. In this way, when the user browses pictures in the gallery application, he or she interacts with the mobile phone and triggers the mobile phone to display the above interface. After receiving a user click on the copy generation button, the mobile phone can automatically generate the copy to provide convenience for the user.
[0132] Furthermore, after the user completes the photo shooting, the user can also trigger the mobile phone to display the first interface including the target image and the first button. In this way, the user can directly trigger the mobile phone to generate the copy after shooting.
[0133] like Figure 6 As shown, the mobile phone can display a main interface, which includes a camera application icon. In response to the user clicking the camera application icon, the mobile phone displays a photo preview interface 601. In response to the user clicking the capture button 602, the mobile phone can capture an image to obtain picture 1. The mobile phone can save picture 1 in the gallery application and display a thumbnail of picture 1 in the photo preview interface 601. In response to the user clicking the thumbnail of picture 1, the mobile phone can display a large picture preview interface 402 of picture 1, i.e., the first interface.
[0134] Combined with the previous article Figure 4 , Figure 5 , Figure 6 Three methods are introduced to trigger the mobile phone to display the first interface including the target picture and the first button. Figure 7 A copywriting generation method provided in an embodiment of the present application is introduced.
[0135] In S701, the mobile phone displays a fourth interface including a thumbnail of a first picture, and receives a third operation of the user on the thumbnail of the first picture, wherein the third operation is used to trigger the mobile phone to display a target picture.
[0136] The fourth interface can be as follows Figure 4 or Figure 5 The list display interface 401 shown in FIG. 401 includes at least one picture thumbnail. The first picture is any one of the at least one picture, for example, the first picture can be picture 1. The fourth interface can also be as follows: Figure 6 The photo preview interface 601 shown includes a thumbnail of picture 1.
[0137] The third operation is used to trigger the mobile phone to display the target picture. The third operation may be, for example, clicking on the thumbnail of the first picture. The third operation may also be, for example, long pressing on the thumbnail of the first picture.
[0138] For example, in response to the user clicking on the thumbnail of picture 1 in the list display interface 401 , the mobile phone may display a large picture preview interface 404 , in which picture 1 (target picture) is displayed.
[0139] For example, in response to a user long pressing the thumbnail of picture 1 in the list display interface 401, the mobile phone can display the list selection interface 501, in which picture 1 (target picture) is displayed. Alternatively, in response to a user long pressing the thumbnail of picture 2 in the list display interface 401, the mobile phone can display the list selection interface 501, in which picture 1 (target picture) is displayed.
[0140] In S702, in response to the third operation, the mobile phone displays a first interface, the first interface includes a target image and a first button, and the first button is used to trigger the mobile phone to generate a text for the target image.
[0141] In response to the user's third operation on the thumbnail of the first picture on the fourth interface, the mobile phone can display the first interface including the target picture and the first button. The target picture and the first picture can be the same picture or different pictures.
[0142] like Figure 4 As shown, in response to the user clicking on the thumbnail of picture 1 in the list display interface 401, the mobile phone can display the large picture preview interface 402 (first interface). Figure 6 As shown, in response to the user clicking on the thumbnail of picture 1 in the photo preview interface 601, the mobile phone can display a large picture preview interface 402 (first interface). The large picture preview interface 402 includes picture 1 (target picture) and a copy generation button 403 (first button).
[0143] like Figure 5 As shown, in response to the user long pressing the thumbnail of picture 1 in the list display interface 401, the mobile phone can display the list selection interface 501 (first interface). The list selection interface 501 includes picture 1 (target picture) and a copy generation button 502 (first button). In response to the user long pressing the thumbnail of picture 2 in the list display interface 401, the mobile phone can display the list selection interface 501 (first interface). The list selection interface 501 includes picture 1 (target picture) and a copy generation button 502 (first button).
[0144] Among them, the target image is the image used to generate the copy, that is, the image used to generate the copy.
[0145] When the mobile phone receives the first operation of the user on the first button on the first interface, the mobile phone can generate the text of the target image. Specifically, the mobile phone can execute S703-S704 to generate the text of the target image.
[0146] In S703, in response to the user's first operation on the first button in the first interface, the mobile phone classifies the target image to obtain a category of the target image.
[0147] The first operation may be, for example, an operation in which the user clicks a copy generation button (i.e., the first button) on the first interface. For example, the first operation may be a user clicking a copy generation button 403 on the large image preview interface 402, triggering the mobile phone to generate a copy for image 1. The first operation may also be a user clicking a copy generation button 502 on the list selection interface 501 (image 2 is selected on the interface), triggering the mobile phone to generate a copy for image 2.
[0148] In some embodiments, the mobile phone can classify the target image according to the type of the subject shown in the target image to obtain the category of the target image. For example, the mobile phone can classify the target image into animal images, plant images, landscape images, person images, food images, etc.
[0149] In other embodiments, the mobile phone may classify the target image as a foreground image or a background image.
[0150] The foreground image may be an image that includes the subject and occupies a larger proportion of the image. In other words, the foreground image focuses more on displaying the subject and pays more attention to the subject. The subject in the foreground image may be closer to the middle of the image, and the subject may occupy most of the image area. Foreground photos can be, for example, close-ups of people, animals, buildings, etc.
[0151] The background image may not include the subject or the subject may occupy a small proportion of the image. In other words, the background image focuses more on displaying the overall environment and does not highlight the subject. The background image may not include the subject. The background image may also include the subject, but the subject only occupies a small part of the image area, and the subject may be relatively scattered in the image. For example, the background image can be a landscape photo.
[0152] After the mobile phone classifies the target image, it can input the target image into the corresponding neural network model according to the category of the target image to obtain the text of the target image. Each category of target image has a corresponding neural network model, and the neural network model can have the ability to output the text of this type of image. The process of the mobile phone classifying the target image is not introduced here, and it will be introduced in detail below in conjunction with the accompanying drawings and specific embodiments.
[0153] In S704, the mobile phone inputs the target image into a neural network model corresponding to the category of the target image to obtain the text of the target image.
[0154] For example, the animal pictures, plant pictures, landscape pictures, person pictures, and food pictures mentioned above all have their corresponding neural network models. The mobile phone can input animal pictures into a neural network model that has the ability to output text for animal pictures to generate text. The mobile phone can input food pictures into a neural network model that has the ability to output text for food pictures to generate text.
[0155] For another example, if the mobile phone recognizes that the target image is a foreground image, the mobile phone can input the target image into the first neural network model corresponding to the foreground image. The first neural network model has the ability to generate text according to the foreground image, that is, the first neural network model has the ability to output the text of the foreground image.
[0156] If the mobile phone recognizes that the target image is a background image, the mobile phone can input the target image into the second neural network model corresponding to the background image. The second neural network model has the ability to generate text according to the background image, that is, the second neural network model has the ability to output the text of the background image.
[0157] The first neural network model and the second neural network model may not be the same neural network model.
[0158] Specifically, the first neural network model is trained using a foreground image and a preset text corresponding to the foreground image. The second neural network model is trained using a background image and a preset text corresponding to the background image. Although both the first neural network model and the second neural network model have the ability to output text of images. For example, inputting a foreground image into the first neural network model can obtain a first text of the foreground image, and inputting a foreground image into the second neural network model can obtain a second text of the foreground image. However, the degree of fit between the first text and the foreground image is higher than the degree of fit between the second text and the foreground image. In other words, the degree of fit between the first text output by the first neural network when recognizing the foreground image and the foreground image is higher than the degree of fit between the second text output by the second neural network when recognizing the foreground image and the foreground image.
[0159] Similarly, inputting the background image into the first neural network model can obtain the third copy of the background image. Inputting the background image into the second neural network model can obtain the fourth copy. However, the degree of fit between the fourth copy and the background image is higher than the degree of fit between the third copy and the background image. In other words, the degree of fit between the fourth copy output by the second neural network model when recognizing the background image and the background image is higher than the degree of fit between the third copy output by the first neural network model when recognizing the background image and the background image.
[0160] The first neural network model and the second neural network model can be pre-configured in the mobile phone. The first neural network model and the second neural network model can also be deployed in a cloud server. The mobile phone can input the target image into the first neural network model or the second neural network model deployed in the cloud, and the mobile phone can receive the text of the target image returned by the cloud.
[0161] In summary, the embodiment of the present application provides a method for generating copywriting. Before generating the copywriting, the mobile phone can first perform fine-grained division on the target image used to generate the copywriting. Moreover, each type of image has a corresponding neural network model for outputting the copywriting of such images. The mobile phone can input the target image into the corresponding neural network model according to the category of the target image to obtain the copywriting of the target image. The embodiment of the present application classifies the target image in advance, and uses the neural network model corresponding to the target image category to generate the copywriting of the target image. Since the neural network model has the ability to output the copywriting of the image of this category, the copywriting generated in this way can be more in line with the target image, and the copywriting and the target image have a higher matching degree, which can improve the fit between the copywriting and the image content.
[0162] The following takes the example of a mobile phone classifying a target image into a foreground image and a background image to introduce the process of a mobile phone classifying a target image. Figure 8 As shown, S703 may include S801 - S802 .
[0163] In S801, in response to a first operation performed by a user on a first interface, the mobile phone identifies a position of at least one subject in a target image.
[0164] The subject is the subject of the photo, which can be a person, animal, building, plant, etc.
[0165] The phone can identify the position of the subject in the picture. When the picture includes multiple subjects, the phone can identify the position of each subject.
[0166] The position of the subject may be the position of the subject in the target image. For example, the mobile phone may indicate the position of the subject by the position of the minimum bounding box surrounding the subject in the target image. The bounding box may be the minimum rectangle surrounding the subject, and the subject is within the minimum rectangle. Of course, the bounding box may also be a circle, an ellipse, or other shapes. The position of the above-mentioned bounding box may be the position of the center point of the bounding box in the image, or may also be the position of the upper left corner vertex and the lower right corner vertex of the bounding box in the image.
[0167] In some embodiments, the mobile phone can also identify the subject in the target image and identify the type of the subject. For example, if the mobile phone identifies that the subject in the target image is a person, the mobile phone can identify that the type of the subject is a person. If the mobile phone identifies that the subject in the target image is a cat, the mobile phone can identify that the type of the subject is an animal.
[0168] Exemplarily, the mobile phone can use a target detection network to identify the type of subject and the location of the subject in the target image. The target detection network is a neural network model. After the target image is input into the target detection network, the target detection network can output the type of subject and the location of the subject in the target image. When the target image includes multiple subjects, the target detection model can output the type of each subject and the location of each subject.
[0169] For example, for the subject in the target image, the target detection network can output the detection value of one or more subjects. The detection value can be, for example, [L, x, y, w, h]. Wherein, L can represent the type of subject, and L can be a preset number used to represent the type of subject. For example, 1 represents a person, 2 represents an animal, 3 represents food, etc. [x, y, w, h] is the position of the subject in the target image, specifically the position of the bounding box surrounding the subject in the target image. Wherein, (x, y) is the coordinate of the center point of the bounding box, and (w, h) is the width and height of the bounding box.
[0170] After the mobile phone recognizes the position of the subject in the target picture, the mobile phone can determine whether the target picture is a foreground picture that focuses more on the subject or a background picture that focuses more on the whole according to the position of the subject in the target picture. The mobile phone can continue to execute S802.
[0171] In S802, the mobile phone determines whether the target image is a foreground image or a background image according to the position of at least one subject in the target image and a preset position in the target image.
[0172] The preset position in the target image may be, for example, the center position in the target image. The preset position may also be, for example, the upper left or lower right or lower left or upper right position of the target image. The preset position may also be, for example, located at the golden section point of the target image. The preset position may be represented by [ax, ay, w, ah], where (ax, ay) is the coordinate of the center point of the preset position, and (aw, ah) is the width and height of the preset position.
[0173] The mobile phone can identify whether the target image is a foreground image or a background image based on one or more of the degree of overlap between the position of at least one subject in the target image and a preset position, the distance between the position of at least one subject and the preset position, and the area of at least one subject in the target image.
[0174] For example, if the target image includes a subject, the mobile phone can calculate the degree of overlap between the subject's position and the preset position. If the degree of overlap between the subject's position and the preset position is high, the mobile phone recognizes the target image as a foreground image. The mobile phone can calculate the distance between the subject's position and the preset position. If the subject's position is close to the preset position, the mobile phone recognizes the target image as a foreground image. The mobile phone can calculate the area of the subject. If the subject occupies a large area of the target image, the mobile phone recognizes the target image as a foreground image.
[0175] Taking the target image including multiple subjects as an example, the mobile phone can calculate the overlap between each subject and the preset position respectively, and then average or sum the multiple overlaps to obtain the overlap between the multiple subjects and the preset position. The mobile phone can compare the overlap value with the threshold value to further identify whether the target image is a foreground image or a background image. The mobile phone can calculate the distance between each subject and the preset position respectively, and then average or sum the multiple distances. The mobile phone can compare the value with the threshold value to further identify whether the target image is a foreground image or a background image. The mobile phone can calculate the area of each subject respectively. Then, average or sum the multiple area values, and the mobile phone can identify whether the target image is a foreground image or a background image based on the average area value or the sum value.
[0176] Furthermore, the position of the subject may be represented by the position of the subject's bounding box. The preset position in the target image may also be represented by the position of the preset bounding box. The position of the preset bounding box in the target image may be, for example, the position of the center point of the preset bounding box on the target image.
[0177] For example, the center point of the preset bounding box may be located at the relative center point of the target image. The preset bounding box may be, for example, a rectangular area whose center point is located at the relative center position of the target image.
[0178] In some embodiments, the position of the preset bounding box can also be located in the upper left or lower right or lower left or upper right area of the target image. That is to say, the embodiment of the present application does not limit the position of the preset bounding box in the target image. In some embodiments, the center point of the preset bounding box can also be located at the golden section point of the target image.
[0179] In some embodiments, the locations of the preset bounding boxes of target images of different categories may be different.
[0180] The mobile phone can calculate the first index of at least one subject in the target image according to the position of the bounding box of at least one subject and the position of the preset bounding box in the target image. The mobile phone can determine whether the target image is a foreground image or a background image through at least one first index. If at least one first index is less than a threshold, the mobile phone determines that the target image is a background image; if at least one first index is greater than a threshold, the mobile phone determines that the target image is a foreground image.
[0181] Specifically, the first index of the subject may be calculated using the following formula.
[0182] The first indicator = a*IOU+b*D+c*AREA.
[0183] Among them, the Intersection over Union (IOU) can be used to indicate the degree of overlap between the subject position and the preset position. The larger the IOU, the higher the degree of overlap between the subject position and the preset position. When the position of the bounding box is used to represent the subject position and the position of the preset bounding box is used to represent the preset position, the IOU can indicate the degree of overlap between the subject's bounding box and the preset bounding box. The larger the IOU value, the higher the degree of overlap between the subject's bounding box and the preset bounding box. The smaller the IOU value, the lower the degree of overlap between the subject's bounding box and the preset bounding box.
[0184] D is the distance between the center point of the subject's bounding box and the center point of the preset bounding box. The smaller D is, the closer the subject's position is to the preset position. The larger D is, the farther the subject's position is from the preset position.
[0185] AREA is the area of the bounding box of the subject. The larger the AREA, the more area the subject occupies in the target image. The smaller the AREA, the smaller the area the subject occupies in the target image.
[0186] a, b, and c are weights respectively. In the present embodiment, the values of a, b, and c are not specifically limited. For example, a may be 0.5, b may be 0.3, and c may be 0.2.
[0187] Take the case where the center point of the preset bounding box is located at the relative center of the target image as an example. The larger the IOU, the higher the overlap between the subject's bounding box and the preset bounding box, and most of the subject's area is within the preset bounding box. The smaller D is, the closer the subject's position is to the relative center of the target image, that is, the subject is closer to the relative center of the target image.
[0188] If the target image includes only one subject, the mobile phone can calculate the first index of the subject. The mobile phone can compare the first index with the first threshold. If the first index is greater than the first threshold, the mobile phone identifies the target image as a foreground image. If the first index is less than the first threshold, the mobile phone identifies the target image as a background image.
[0189] by Fig. 9Taking 9a in FIG. 1 as an example, the position of the subject 1 can be represented by the position of the bounding box 1. The area of the bounding box 1 is S1. The target image also includes a preset bounding box located at a relative center position of the target image. The area of the preset bounding box is S3. The overlapping area of the bounding box 1 and the preset bounding box is S4. The distance from the center point of the bounding box 1 to the center point of the preset bounding box is D1.
[0190]
[0191] After calculating the first indicator of subject 1, the mobile phone can compare the first indicator of subject 1 with the first threshold. If the first indicator of subject 1 is greater than the first threshold, the mobile phone recognizes Fig. 9 The target image shown in 9a is the foreground image. The first index of the subject 1 is less than the first threshold value, and the mobile phone recognizes Fig. 9 The target picture shown in 9a is the background picture.
[0192] If the target image includes multiple subjects, the mobile phone can calculate the first indicators of the multiple subjects respectively. The mobile phone compares the calculated first indicators of the multiple subjects with the first threshold. If the first indicators of the multiple subjects are all less than the first threshold, the mobile phone identifies the target image as a background image. If the first indicators of the multiple subjects include one or more first indicators greater than the first threshold, the mobile phone identifies the target image as a foreground image.
[0193] by Fig. 9 Take 9b in as an example, Fig. 9 The target image shown in 9b in FIG. 1 also includes a subject 2. The position of the subject 2 can be represented by the position of the bounding box 2. The area of the bounding box 2 is S2. The overlapping area of the bounding box 2 and the preset bounding box is S5. The distance from the center point of the bounding box 2 to the center point of the preset bounding box is D2.
[0194]
[0195] After calculating the first index of subject 1 and the first index of subject 2, the mobile phone can compare the two first indexes with the threshold. If the first index of subject 1 is greater than the first threshold, or the first index of subject 2 is greater than the first threshold, or the first index of subject 1 and the first index of subject 2 are both greater than the first threshold, the mobile phone determines Fig. 9 The target image shown in 9b is a foreground image. If the first index of subject 1 is less than the threshold and the first index of subject 2 is less than the first threshold, the mobile phone determines that the target image is a background image.
[0196] After the mobile phone recognizes that the target image is a foreground image or a background image, the mobile phone can input the target image into the neural network model to generate the text of the target image. Specifically, S704 can include S803 and S804. After S703, if the target image is a foreground image, the mobile phone can execute S803; after S703, if the target image is a background image, the mobile phone can execute S804.
[0197] At S803, the mobile phone inputs the target image into a first neural network capable of outputting the text of the foreground image to obtain the text of the target image. In some embodiments, the mobile phone outputs the target image and the subject type of at least one subject in the target image to the first neural network to obtain the text of the target image.
[0198] At S804, the mobile phone inputs the target image into a second neural network capable of outputting the text of the background image to obtain the text of the target image. In some embodiments, the mobile phone outputs the target image and the subject type of at least one subject in the target image into the second neural network to obtain the text of the target image.
[0199] That is, in response to the user clicking the copy generation button 403 or the copy generation button 502 in the large image preview interface 402 or the list selection interface 501, the mobile phone can execute Figure 7 In S703, the mobile phone can first identify the category of the picture 1, and then the mobile phone can input the picture 1 into the corresponding neural network model to obtain the text of the picture 1.
[0200] Furthermore, the mobile phone can also execute S801-S802 as shown in Figure 8 to identify whether Image 1 is a foreground image or a background image. Then, based on the recognition result, the mobile phone executes S803-S804 to input Image 1 into the first neural network model or the second neural network model to obtain the text of Image 1.
[0201] After the mobile phone generates the copy, it can store the copy in a buffer in the mobile phone's memory. The buffer can be, for example, a clipboard. Afterwards, in response to the user's paste operation, the mobile phone can paste the copy in a text editing area in the mobile phone. The text editing area can be, for example, a text editing area for a text message, a text editing area for an instant messaging software, a text editing area for a document, or a text editing area for a social software, etc. In other words, after the mobile phone automatically generates the copy, it can first store it in a buffer, and in response to the user's paste operation, the mobile phone can copy the copy in the user's text editing area. In this way, the user can share the copy via text messages, instant messaging software, or social software, or add the copy to a document. Fig.10 Give an example of sharing copy via social software.
[0202] like Fig.10 As shown, the mobile phone displays a large picture preview interface 402 including picture 1. In response to the user clicking the copy generation button 403 on the large picture display interface, the mobile phone executes the following Figure 7 S703 in the figure automatically generates the text of picture 1. The mobile phone copies the text to the clipboard of the mobile phone.
[0203] When the mobile phone receives the user clicking the application icon of the social application on the mobile phone main interface 1001 , the mobile phone can display the picture and text sharing interface 1002 of the social application topic.
[0204] In some embodiments, when the mobile phone receives that the user clicks the application icon of the social application on the mobile phone main interface 1001, the mobile phone may first display other interfaces of the social application, such as the main interface (not shown in the figure). The main interface includes a button for triggering the mobile phone to display the picture and text sharing interface 1002. After receiving that the user clicks the button, the mobile phone may display the picture and text sharing interface 1002.
[0205] The picture and text sharing interface 1002 includes a create button 1003. After the mobile phone receives a click operation of the user on the create button 1003, the mobile phone can display a picture and text editing interface 1004. In some embodiments, after the mobile phone receives a click operation of the user on the create button 1003, the mobile phone can display the following Fig.12 In response to the user selecting picture 1 on the list selection interface 1101, the mobile phone may display the following Fig.10 The graphic and text editing interface 1004 is shown.
[0206] The picture and text editing interface 1004 may include a picture 1, a text editing area 1007, and a publish button 1005. The text editing area 1007 is used to display the edited text. The publish button 1005 is used to publish the edited text in the picture and text sharing interface 1002.
[0207] When the mobile phone receives the user's long press on the text editing area 1007, the mobile phone may display a paste button 1006. In response to the user clicking the paste button 1006, the mobile phone may paste the text of the picture 1 in the text editing area 1007 to complete the picture and text editing. Fig.10 Interface 1008. When the mobile phone receives that the user clicks the publish button 1005 in interface 1008, the mobile phone can display a picture and text sharing interface 1009. The picture and text sharing interface 1009 includes picture 1 and corresponding text.
[0208] Fig.10The figure shows the jump process of the mobile phone display interface when the mobile phone creates a picture and a text corresponding to the picture in the picture and text sharing interface of the social application in response to the user's operation. The mobile phone can not only create pictures and texts in the picture and text sharing interface of the social application, but also create texts only in the picture and text sharing interface of the social application.
[0209] like Fig.11 As shown in FIG. 1 , when the mobile phone receives a message that the user clicks the create button 1003 in the picture and text sharing interface 1002, the mobile phone may display a pop-up window 1101. The pop-up window 1101 is used to prompt the user to select the format of the content to be published. The pop-up window 1101 may include a picture and text sharing button, a text sharing button, and a video sharing button. In response to the user clicking the picture and text sharing button, the mobile phone may display the following message: Fig.10 In response to the user clicking the text button, the mobile phone may display the following Fig.11 The text editing interface 1102 is shown.
[0210] The text editing interface 1102 includes a text editing area 1103 and a publish button. When the mobile phone receives that the user has long pressed the text editing area 1103, the mobile phone may display a paste button. In response to the user clicking the paste button, the mobile phone may paste the text of picture 1 in the text editing area 1103 to complete the text editing. The mobile phone displays an interface 1104. When the mobile phone receives that the user has clicked the publish button in the interface 1104, the mobile phone may display a picture and text sharing interface 1105. The picture and text sharing interface 1105 only includes the text of picture 1.
[0211] That is to say, users can trigger the copy generated by the mobile phone in the gallery application or camera application. After that, users can trigger the mobile phone to paste the copy in the picture and text editing interface or text editing interface of the social software. In this way, users can avoid editing the copy themselves. However, the mobile phone needs to switch between the two applications, which is a cumbersome process. In order to further improve the user experience, the copy generation button can be located on a certain interface of the social application. Users can trigger the mobile phone to generate copy in the social application, avoiding switching between multiple programs and improving the user experience.
[0212] like Fig.12 As shown, the copy generation button can be located on the large picture preview interface 1202 of the social application or the list selection interface 1204 of the social software. For example, the mobile phone displays the fifth interface, and the fifth interface can be the picture and text sharing interface 1002 of the social software. The picture and text sharing interface 1002 can include a picture and text sharing button. In response to the user clicking the picture and text sharing button, the mobile phone displays the fourth interface. The fourth interface includes thumbnails of multiple pictures. In response to clicking or long pressing the thumbnail of the target picture on the fourth interface, the mobile phone displays the first interface.
[0213] Specifically, the mobile phone displays the picture and text sharing interface 1002 of the social software, and the mobile phone receives the user clicking the picture and text sharing button in the pop-up window 1101, and the mobile phone can display the list selection interface 1201 (fourth interface) of the social software. The list selection interface 1201 may include thumbnails of pictures in the mobile phone. Among them, the list selection interface 1201 and the list selection interface 501 of the gallery application can be the same interface or different interfaces, and the embodiment of the present application does not make specific limitations on this.
[0214] In response to the user selecting picture 1 in the list selection interface 1201 and clicking the OK button, the mobile phone can display the large picture preview interface 1202 (first interface) of the social application. The large picture preview interface 1202 and the large picture preview interface 402 of the gallery application can be the same interface or different interfaces, and the embodiment of the present application does not make specific limitations on this. The large picture preview interface 1202 includes a copy generation button 1203. In other words, the fourth interface can also be the list selection interface 1201, and the first interface can also be the large picture preview interface 1202.
[0215] like Fig.12 As shown, after the mobile phone receives the user clicking the picture and text sharing button in the pop-up window 1101, the mobile phone can display the list selection interface 1201. In response to the user clicking the thumbnail of picture 1 in the list selection interface 1201, the mobile phone can display the following Fig.12 The list selection interface 1204 (first interface) is described. The list selection interface 1204 may include a text generation button 1205. In response to the user selecting picture 1 in the list selection interface 1204 and clicking the text generation button 1205, the mobile phone may generate a text for picture 1. The list selection interface 1204 and the list selection interface 501 of the gallery application may be the same interface or different interfaces, and the embodiment of the present application does not make specific limitations on this. In other words, the first interface may also be the list selection interface 1204.
[0216] In response to the user clicking the text generation button 1203 on the large picture preview interface 1202, or selecting picture 1 on the list selection interface 1204 and clicking the text generation button 1205, the mobile phone may display the following: Fig.10 The picture and text editing interface 1004 of the social software shown in the figure. Afterwards, in response to the user's related operations in the picture and text editing interface 1004, the mobile phone can create picture 1 and the text corresponding to picture 1 in the picture and text sharing interface. The process of the mobile phone creating picture 1 and text is not repeated here, and can be referred to Fig.10 And previous text.
[0217] In other words, the phone can also display Fig.12The large image preview interface 1202 (first interface) or the list selection interface 1204 (first interface) shown in FIG. 1 receives a user's click operation on the copy generation button and executes the following steps: Figure 7 As shown in S703-S704, the text is automatically generated.
[0218] In some embodiments, in response to the user clicking the copy generation button, the mobile phone may also display a copy preview interface (third interface).
[0219] The text preview interface is used to preview the text generated by the mobile phone. Fig.13 Two forms of copy preview interfaces are shown. Copy preview interface 1301 can only preview the copy generated by the mobile phone. Copy preview interface 1302 can also preview the picture used to generate the copy and the copy generated by the mobile phone. Copy preview interface 1301 or copy preview interface 1302 can include a return button, an apply button, and a regenerate button.
[0220] The return button is used to trigger the mobile phone to return from the copy preview interface to the interface including the copy generation button. The interface including the copy generation button can be, for example, the large picture preview interface 402, the list selection interface 501, the large picture preview interface 1202 or the list selection interface 1204 mentioned above. In this way, it is convenient for the user to reselect the picture for generating the copy.
[0221] The Apply button is used to trigger the mobile phone to cache the generated text in the buffer. That is, the mobile phone can automatically cache the text in the buffer after generating the text, or cache the text in the buffer in response to the user clicking the Apply button on the text preview interface.
[0222] The Regenerate button is used to regenerate the copy based on the image of the copy to be generated. Fig.14 As shown, the mobile phone displays a copy preview interface 1301 of picture 1. When the mobile phone receives a message that the user clicks the regenerate button on the interface, the mobile phone can regenerate the copy according to picture 1. Specifically, in response to the user clicking the regenerate button on the copy preview interface, the mobile phone inputs picture 1 into the neural network model corresponding to the category of picture 1 again. The neural network model can regenerate the copy according to picture 1 again. The regenerated copy can be different from or the same as the copy generated last time.
[0223] Further, after the mobile phone receives the user clicking the copy generation button on the display interface including the target image and the copy generation button, the mobile phone can display a style selection interface. The style selection interface is used for the user to select the style of the generated copy. As an example, the style of the copy can include fresh, literary, elegant, funny, etc.
[0224] As an example, Fig.15 In response to the user clicking the text generation button 403 in the large picture preview interface 402, the mobile phone can display a style selection pop-up window 1501, or the mobile phone can display a style selection interface 1502. The style selection pop-up window 1501 or the style selection interface 1502 includes a plurality of preset style selection buttons. In response to the user clicking one of the style buttons, the mobile phone can generate a text according to the picture 1 and the style selected by the user.
[0225] Specifically, Fig.16 As shown, the mobile phone executes S702 to display a first interface including a target image and a first button. The mobile phone executes S703, and in response to the user's first operation on the first button in the first interface, the mobile phone classifies the target image to obtain a category of the target image. Afterwards, the mobile phone may also execute S1601.
[0226] In S1601, in response to a user's first operation on the first interface, the mobile phone displays a second interface, where the second interface includes at least two style buttons. The second interface may be as follows: Fig.15 A style selection pop-up window 1501 or a style selection interface 1502 is shown.
[0227] In 1602, the mobile phone receives a click operation of a user on a target button among at least two style buttons.
[0228] After the mobile phone receives the user's click operation on the target button among the at least two style buttons, the mobile phone may execute S704 to generate a text of the target image. Specifically, S704 may include S1603.
[0229] In S1603, in response to the click operation, the mobile phone inputs the target image into a neural network model corresponding to the category of the target image and the target text style to obtain the text of the target image; wherein the target text style corresponds to the target button.
[0230] That is to say, taking Picture 1 as an example, the mobile phone recognizes that the category of Picture 1 is plant, and the mobile phone can Figure 1 Input a neural network model capable of outputting text for plant images to obtain text for Image 1. Further, the mobile phone can output Image 1 to a neural network model corresponding to the plant category and the style selected by the user according to the category of Image 1 and the text style selected by the user to obtain text for Image 1. The neural network model has the ability to output text for plant images and the style of the text is consistent with the style selected by the user. In this way, text that is more in line with the image and meets the style requirements of the user can be obtained.
[0231] For example, the preset styles in the mobile phone are small fresh and funny. For the neural network model that can generate copy based on a category of pictures, it can be further divided into a neural network model that can generate small fresh style copy and a neural network model that can generate funny style copy. These two neural network models may not be the same neural network model. The mobile phone can selectively input the target picture into the neural network model corresponding to the category and target style of the target picture according to the category and target style of the target picture, and obtain a copy that is more in line with the picture and meets the user's style requirements.
[0232] For another example, the two neural network models mentioned above can be the same neural network model. The neural network model has the ability to generate texts with different styles for the same picture. As an example, take Picture 1 as an example, in the neural network model training stage. The training sample can be [Picture 1, style information, predefined text 1]. Picture 1 and style information are input into the neural network model, and the neural network model outputs the text of Picture 1. According to the text output by the neural network model and the predefined text 1, the parameters of the neural network model are corrected so that the neural network model outputs the text corresponding to the style information. In other words, in the neural network model training stage, the style can be used as auxiliary information and input into the neural network model together with the picture, so that the neural network can be trained to have the ability to generate different texts according to different styles for the same picture.
[0233] Fig.17 The following figure shows the interface jump diagram for generating copywriting according to different styles. Fig.17 As shown, the mobile phone displays a large picture preview interface 402, and in response to the user clicking the copy generation button 403, the mobile phone displays a style selection interface 1502. In response to the user clicking the small fresh button in the style selection interface, the mobile phone can display the copy preview interface 1301a. The copy preview interface 1301a includes copy 1. In response to the user clicking the elegant button in the style selection interface, the mobile phone can display the copy preview interface 1301b. The copy preview interface 1301b includes copy 2. Copy 1 and copy 2 can be different copies.
[0234] Furthermore, the back button on the copy preview interface can also trigger the mobile phone to display the style selection interface, so that the user can reselect the style of the copy. Fig.18 The text preview interface 1301a includes the generated text and a return button. In response to the user clicking the return button, the mobile phone can display the style selection interface 1502. In response to the user selecting the target style in the style selection interface 1502, the mobile phone can execute the following operations: Fig.16 What is shown is S1602-S1603, regenerating the copy. After the copy is generated, the mobile phone can display the copy preview interface 1301b. The copy preview interface 1301b includes the copy generated again by the mobile phone.
[0235] The regenerate button on the text preview interface can also be used to trigger the mobile phone to generate text with the same style but different content based on the target image and the currently selected target style.
[0236] In some cases, a user may only want to generate text based on a portion of the content of a picture. To this end, a text generation method provided in an embodiment of the present application can also generate text only for the area of the picture that the user is interested in. Specifically, Fig.19 As shown, after the mobile phone executes S720 and displays the first interface in response to the user's first operation, the mobile phone can also receive a second operation of the user on the target image. The second operation can be that the user circles a closed area on the target image. The closed area can be a rectangle, a circle, a heart or other irregular shapes.
[0237] In S1901, the mobile phone receives a second operation performed by the user on the target image, where the second operation is used to circle a closed area on the target image.
[0238] At S1902 , the mobile phone obtains the position of the closed area in the target image in response to the second operation.
[0239] In response to receiving the second operation of the user on the target image on the first interface, the mobile phone can obtain the position of the closed area circled by the user on the target image.
[0240] Afterwards, the mobile phone may execute S703, in response to the first operation of the user on the first interface, the mobile phone may classify the target image. Afterwards, the mobile phone may input the target image into a neural network model corresponding to the category of the target image to generate a copy.
[0241] Taking the example of a mobile phone dividing a target image into a foreground image and a background image, S703 may also include S1903-S1904.
[0242] At S1903 , in response to a first operation of a user on a first button, the mobile phone identifies a position of at least one subject in a target picture.
[0243] In S1904, the mobile phone determines whether the target image is a foreground image or a background image according to the position of at least one subject in the target image and the position of a closed area circled by the user on the target image.
[0244] The mobile phone can identify whether the target image is a foreground image or a background image based on one or more of the degree of overlap between the position of at least one subject in the target image and the position of a closed area, the distance between the position of at least one subject and the position of the closed area, and the area of at least one subject in the target image.
[0245] For example, if the target image includes a subject, the mobile phone can calculate the degree of overlap between the subject's position and the closed area. If the subject's position and the closed area have a high degree of overlap, the mobile phone will identify the target image as a foreground image. The mobile phone can calculate the distance between the subject's position and the closed area. If the subject's position is close to the closed area, the mobile phone will identify the target image as a foreground image. The mobile phone can calculate the area of the subject. If the subject occupies a large area of the target image, the mobile phone will identify the target image as a foreground image.
[0246] Taking the target image including multiple subjects as an example, the mobile phone can calculate the overlap between each subject and the closed area respectively, and then average or sum the multiple overlaps to obtain the overlap between the multiple subjects and the closed area. The mobile phone can compare the overlap value with the threshold value to identify whether the target image is a foreground image or a background image. The mobile phone can calculate the distance between each subject and the closed area respectively, and then average or sum the multiple distances. The mobile phone can compare the value with the threshold value to identify whether the target image is a foreground image or a background image. The mobile phone can calculate the area of each subject respectively. Then, average or sum the multiple area values, and the mobile phone can identify whether the target image is a foreground image or a background image based on the average area value or the sum value.
[0247] Further, the position of the subject can be represented by the position of the subject's bounding box. The position of the target enclosed area can be represented by a custom bounding box. The custom bounding box can be the minimum area including the enclosed area circled by the user on the target image. The custom bounding box can be a rectangle, a circle, a heart or other irregular shapes.
[0248] The mobile phone can calculate the second index of the subject based on the bounding box of the subject and the custom bounding box.
[0249] Specifically, the second index of the subject may be calculated using the following formula.
[0250] The second index = a*IOU'+b*D'+c*AREA.
[0251] Where IOU' represents the degree of overlap between the subject's bounding box and the custom bounding box. D' represents the distance from the center point of the subject's bounding box to the center point of the custom bounding box. AREA represents the area of the subject's bounding box. a, b, and c are weights respectively.
[0252] If the target image only includes one subject, the mobile phone can calculate the second index of the subject. The mobile phone can compare the second index with the first threshold. If the second index is greater than the first threshold, the mobile phone identifies the target image as a foreground image. If the second index is less than the first threshold, the mobile phone identifies the target image as a background image.
[0253] by Fig. 20 Taking 20a in FIG. 1 as an example, the position of subject 3 can be represented by the position of bounding box 3. The area of bounding box 3 is S7. The area of the custom bounding box in the target image is S10. The overlapping area of bounding box 3 of subject 3 and the custom bounding box is S9. The distance from the center point of bounding box 3 to the center point of the custom bounding box is D7. Then:
[0254]
[0255] After calculating the second index of subject 3, the mobile phone can compare the second index of subject 3 with the first threshold. If the second index of subject 3 is greater than the first threshold, the mobile phone recognizes Fig. 20 The target image shown in 20a is the foreground image. The second index of the subject 3 is less than the first threshold value, and the mobile phone recognizes Fig. 20 The target picture shown in 20a is the background picture.
[0256] If the target image includes multiple subjects, the mobile phone can calculate the second indicators of the multiple subjects respectively. The mobile phone compares the calculated second indicators of the multiple subjects with the first threshold. If the second indicators of the multiple subjects are all less than the first threshold, the mobile phone identifies the target image as a background image. If the second indicators of the multiple subjects include one or more second indicators greater than the first threshold, the mobile phone identifies the target image as a foreground image.
[0257] by Fig. 20 Taking 20b in FIG. 1 as an example, the target image may also include a subject 4. The position of the subject 4 may be represented by the position of the bounding box 4. The area of the bounding box 4 is S6. The overlapping area of the bounding box 3 of the subject 4 and the custom bounding box is S8. The distance from the center point of the bounding box 4 to the center point of the custom bounding box is D6. Then:
[0258]
[0259] After calculating the second index of subject 3 and the second index of subject 4, the mobile phone can compare the two second indexes with the first threshold. If the second index of subject 3 is greater than the first threshold, or the second index of subject 3 is greater than the first threshold, or the second thresholds of subject 3 and subject 4 are both greater than the first threshold, the mobile phone determines Fig. 20 The target image shown in 20b is the foreground image. If the second index of subject 3 is less than the first threshold and the second index of subject 4 is less than the first threshold, the mobile phone determines Fig. 20 The target image shown in 20b is the background image
[0260] After that, the mobile phone can input the target image into the corresponding neural network model to obtain the text of the target image.
[0261] In S704, the mobile phone inputs the target image into a neural network model corresponding to the category of the target image to obtain the text of the target image.
[0262] In some embodiments, the mobile phone may automatically receive a second operation of the user to circle a closed area on the target image, and in response to the second operation, automatically obtain the location of the closed area circled by the user.
[0263] In some other embodiments, the first interface may further include a custom button. The custom button is used to trigger the mobile phone to receive the second operation of the user on the target image, and obtain the position of the closed area circled by the user on the target image in response to the second operation. Fig.21 As shown, the large image preview interface 402 may include not only a copywriting generation button 403, but also a customization button 404. In response to the user clicking the customization button 404, the mobile phone may receive the user's second operation and obtain the position of the closed area circled by the user.
[0264] That is to say, the mobile phone can determine the category of the target image based on the position of at least one subject in the target image and the preset position in the target image or the position of the closed area circled by the user. The target image is input into the neural network model corresponding to its category to generate copy. Fig. 22 A flowchart of a method for generating text provided in an embodiment of the present application.
[0265] In S2201, in response to a first operation of a user on a first button, the mobile phone inputs a target image into a target detection network to obtain a position of at least one subject in the target image.
[0266] In S2202, the mobile phone calculates a first index or a second index of at least one subject in the target image according to the position of the at least one subject.
[0267] In S2203, the mobile phone determines the magnitude of the first indicator or the second indicator of at least one subject and the first threshold α.
[0268] If the first indicator or the second indicator of at least one subject includes an indicator greater than the first threshold α, the mobile phone executes S2204.
[0269] In S2204, the mobile phone inputs the target image into the foreground target text generation network, that is, the first neural network model, to obtain the text of the target image.
[0270] If the first indicator or the second indicator of at least one subject does not include an indicator greater than the first threshold α, the mobile phone executes S2205.
[0271] In S2205, the mobile phone inputs the target image into the background target text generation network, that is, the second neural network model, to obtain the text of the target image.
[0272] Furthermore, the text generation method provided in the embodiment of the present application can also generate text consistent with the style selected by the user according to the style selected by the user. Fig.23 shown. Fig.23 A flowchart of a method for generating text provided in an embodiment of the present application.
[0273] In S2301, the mobile phone displays a first interface including a target image and a first button, wherein the first button is used to trigger the mobile phone to generate a text of the target image.
[0274] In S2302, in response to the user clicking the first button, the mobile phone displays a second interface, wherein the second interface includes at least two style buttons.
[0275] In S2303, in response to the user clicking a target button among the at least two style buttons on the second interface, the mobile phone identifies whether the user performs a second operation on the target image. If the mobile phone does not identify that the user performs the second operation on the target image, the mobile phone may proceed to S2305.
[0276] If the mobile phone recognizes that the user performs the second operation on the target image, the mobile phone may execute S2304.
[0277] In S2304, the mobile phone responds to the second operation on the target image by the user and obtains the closed area circled by the user. Afterwards, the mobile phone can execute S2305 to generate a text for the target image.
[0278] At S2305, the mobile phone generates text for the target image.
[0279] Specifically, the mobile phone can first identify the position of at least one subject in the target image. The mobile phone can identify the category of the target image based on the position of at least one subject and the preset position in the target image or the position of the closed area circled by the user. If the target image is a foreground image, the mobile phone inputs the target image into the first neural network model, and the first neural network model has the ability to output the text of the foreground image. If the target image is a background image, the mobile phone inputs the target image into the second neural network model, and the second neural network model has the ability to display the text of the background image.
[0280] After the copy is generated, the mobile phone may execute S2306.
[0281] At S2306, the mobile phone displays a third interface, which includes a return button, an apply button, and a regenerate button.
[0282] In response to the user clicking the Apply button, the mobile phone can also cache the text in the clipboard. In this way, in response to the user's paste operation, the mobile phone can paste the text in the required document editing area.
[0283] In response to the user clicking the back button, the mobile phone may further display a second interface, where the second interface includes at least two style buttons.
[0284] In response to the user clicking the regenerate button, the mobile phone can also re-input the target image into the neural network model corresponding to the category of the target image and the target copy style, and regenerate the copy of the target image.
[0285] In some embodiments, after the mobile phone executes S701 and receives the first operation of the user on the thumbnail of the first picture, the mobile phone may first display the fifth interface. The sixth interface may be, for example, Fig.24 The large picture preview interface 2401 is shown. The large picture preview interface 2401 includes a copy button. In response to the user clicking the copy button, the mobile phone can display a first interface 2402. The first interface 2402 includes a pop-up window 2403 and a picture 1. The pop-up window 2403 may include a copy generation button and a custom button. Alternatively, in response to the user clicking the copy button, the mobile phone may display a first interface 2404. The first interface 2404 may include a picture 1, a copy generation button and a custom button.
[0286] like Fig.25 As shown, the sixth interface can also be Fig.25 The list selection interface 2501 shown in FIG. 2501 includes a text button. In response to the user clicking the text button, the mobile phone can display the first interface 2502. The first interface 2502 includes a pop-up window 2503 and thumbnails of multiple pictures, including a thumbnail of picture 1. The pop-up window 2503 can include a text generation button and a custom button. Alternatively, in response to the user clicking the text button, the mobile phone can display the first interface 2504. The first interface 2504 can include thumbnails of multiple pictures, a text generation button, and a custom button.
[0287] The embodiment of the present application does not limit the type of target detection network. The target detection network can be any neural network model that has the ability to output the type and position of at least one subject in the target image. The target detection network can be, for example, a YOLO series target detection network, or, for example, a transform neural network model.
[0288] The embodiment of the present application does not limit the type of neural network model used to generate the document. In the embodiment of the present application, the neural network model has the ability to output the text of the picture that is consistent with the type of the target picture, and the neural network model can be any neural network model. The neural network model can be, for example, a convolutional neural network, or, for example, a seq2seq neural network model.
[0289] The present application embodiment provides an electronic device, which includes: a memory, a display screen, and one or more processors. The display screen is coupled to the processor. The memory is used to store computer program code. The computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device can execute the various functions or steps executed by the mobile phone in the above method embodiment. The structure of the electronic device can refer to Figure 2 The structure of the electronic device 200 is shown.
[0290] The present application also provides a computer storage medium, which includes computer instructions. When the computer instructions are in the above electronic device (such as Figure 2 When the method is executed on the electronic device 200 shown in the figure, the electronic device executes each function or step in the above method embodiment.
[0291] The embodiment of the present application also provides a computer program product. When the computer program product is run on a computer, the computer is enabled to execute each function or step in the above method embodiment.
[0292] The embodiment of the present application also provides a chip system, which includes at least one processor and at least one interface circuit. The processor and the interface circuit can be interconnected by lines. For example, the interface circuit can be used to receive signals from other devices (such as a memory of an electronic device). For another example, the interface circuit can be used to send signals to other devices (such as a processor). Exemplarily, the interface circuit can read instructions stored in the memory and send the instructions to the processor. When the instruction is executed by the processor, the electronic device can perform the various steps in the above embodiments. Of course, the chip system can also include other discrete devices, which is not specifically limited in the embodiment of the present application.
[0293] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.
[0294] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.
[0295] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.
[0296] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.
[0297] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.
[0298] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims
1. A copywriting generation method, characterized in that: Applied to electronic equipment, the method comprises: The electronic device displays a first interface, the first interface includes a target image and a first button, the first button is used to trigger the electronic device to generate a text for the target image; In response to a first operation of the user on the first button in the first interface, the electronic device classifies the target image to obtain a category of the target image; The electronic device inputs the target image into a neural network model corresponding to the category of the target image to obtain the text of the target image; wherein different categories correspond to different neural network models, and the neural network model has the ability to generate text.
2. The method according to claim 1, characterized in that After the electronic device inputs the target image into the neural network model corresponding to the category of the target image to obtain the text of the target image, the method further includes: The electronic device saves the text of the target image in a clipboard of the electronic device.
3. The method according to claim 1 or 2, characterized in that: The category of the target image is: foreground image or background image.
4. The method according to claim 3, characterized in that The foreground image corresponds to a first neural network model, and the background image corresponds to a second neural network model; wherein the first neural network model has the ability to output the text of the foreground image, and the second neural network model has the ability to output the text of the background image; The electronic device inputs the target image into a neural network model corresponding to the category of the target image to obtain the text of the target image, including: The target image is a foreground image, and the electronic device inputs the target image into the first neural network model to obtain a text of the target image; The target image is a background image, and the electronic device inputs the target image into the second neural network model to obtain the text of the target image.
5. The method according to claim 3 or 4, characterized in that: In response to a first operation of the user on the first button on the first interface, the electronic device classifies the target image to obtain a category of the target image, including: In response to the first operation, the electronic device identifies a position of at least one subject in the target picture; The electronic device identifies whether the target image is a foreground image or a background image based on one or more of the degree of overlap between the position of each of the at least one subject and a preset position in the target image, the distance between the position of each of the at least one subject and the preset position, and the area of each of the at least one subject in the target image.
6. The method according to claim 5, characterized in that The position of each subject in the at least one subject is the position of a minimum bounding box surrounding the corresponding subject, the position of the bounding box is used to represent the position of the subject in the target image, and the preset position in the target image is the position of the preset bounding box; The electronic device identifies whether the target picture is a foreground picture or a background picture according to one or more of the following: a degree of overlap between the position of each of the at least one subject and a preset position in the target picture, a distance between the position of each of the at least one subject and the preset position, and an area of each of the at least one subject in the target picture, including: The electronic device calculates a first index of each of the at least one subject according to the position of the bounding box of the at least one subject and the position of the preset bounding box by using the following formula; wherein the first index=a*IOU+b*D+c*AREA, the IOU is the intersection-and-union ratio of the bounding box of the corresponding subject and the preset bounding box, the IOU indicates the overlap between the bounding box of the corresponding subject and the preset bounding box, the D is the distance from the center point of the bounding box of the corresponding subject to the center point of the preset bounding box, and the AREA is the area of the bounding box of the corresponding subject; If the first indicator of each of the at least one subject is less than a first threshold, the electronic device identifies the target image as a background image; If the first indicator of any one or more of the at least one subject is greater than the first threshold, the electronic device identifies the target image as a foreground image.
7. The method according to claim 3 or 4, characterized in that: Before the electronic device, in response to a first operation of the user on the first button on the first interface, classifies the target image to obtain a category of the target image, the method further includes: The electronic device receives a second operation performed by a user on the target image, where the second operation is used to circle a closed area on the target image; In response to the second operation, the electronic device obtains the position of the closed area in the target image; The electronic device, in response to a first operation of the user on the first button on the first interface, classifies the target image to obtain a category of the target image, including: In response to the second operation, the electronic device identifies a position of the at least one subject in the target picture; The electronic device identifies whether the target image is a foreground image or a background image based on one or more of the following: a degree of overlap between the position of each of the at least one subject and the position of the closed area, a distance between the position of each of the at least one subject and the position of the closed area, and an area of each of the at least one subject in the target image.
8. The method according to claim 7, characterized in that The position of each subject in the at least one subject is the position of the minimum bounding box surrounding the corresponding subject, the position of the bounding box is used to represent the position of the subject in the target image, and the position of the closed area is the position of the custom bounding box. The electronic device identifies whether the target picture is a foreground picture or a background picture according to one or more of the following: a degree of overlap between the position of each of the at least one subject and the position of the closed area, a distance between the position of each of the at least one subject and the position of the closed area, and an area of each of the at least one subject in the target picture, including: The electronic device calculates the second index of each of the at least one subject according to the position of the bounding box of the at least one subject and the position of the custom bounding box by using the following formula; wherein the second index = a*IOU'+b*D'+c*AREA, the IOU' is the intersection-and-union ratio of the bounding box of the corresponding subject and the custom bounding box, the IOU' indicates the overlap between the bounding box of the corresponding subject and the custom bounding box, D' is the distance from the center point of the bounding box of the corresponding subject to the center point of the custom bounding box, and AREA is the area of the bounding box of the corresponding subject; If the second indicator of each of the at least one subject is less than the first threshold, the electronic device identifies the target image as a background image; If the second indicator of any one or more of the at least one subject is greater than a first threshold, the electronic device identifies the target image as a foreground image.
9. The method according to any one of claims 1 to 8, characterized in that: The method further comprises: The electronic device displays a second interface in response to the first operation; wherein the second interface includes at least two style buttons, each style button corresponding to a copywriting style; The electronic device receives a click operation of a user on a target button among the at least two style buttons; The electronic device inputs the target image into a neural network model corresponding to the category of the target image to obtain the text of the target image, including: In response to the click operation, the electronic device inputs the target image into a neural network model corresponding to the category of the target image and the target text style to obtain the text of the target image; wherein the target text style corresponds to the target button.
10. The method according to any one of claims 1 to 9, characterized in that: After the electronic device inputs the target image into a neural network model corresponding to the category of the target image to obtain the text of the target image, the method further includes: The electronic device displays a third interface, where the third interface is used to preview the document. The third interface includes a regenerate button, where the regenerate button is used to trigger the electronic device to regenerate the document.
11. The method according to any one of claims 1 to 10, characterized in that: The electronic device displays a first interface, including: The electronic device displays a fourth interface; wherein the fourth interface includes thumbnails of a plurality of pictures, and the plurality of pictures include a plurality of target pictures; In response to a third operation performed by the user on the thumbnail of the target image in the fourth interface, the electronic device displays the first interface; the third operation is a click or long press operation on the thumbnail of the target image.
12. The method according to claim 11, characterized in that Before the electronic device displays the first interface in response to the user's third operation on the fourth interface, the method further includes: The electronic device displays a fifth interface; wherein the fifth interface is a picture and text sharing interface of a social software; In response to a user clicking on a picture and text sharing button in the fifth interface, the electronic device displays the fourth interface.
13. An electronic device, characterized in that: The electronic device comprises: a memory, a display screen, and one or more processors; the memory and the display screen are coupled to the processor; wherein the memory is used to store computer program code, and the computer program code comprises computer instructions; when the computer instructions are executed by the processor, the electronic device executes the method as described in any one of claims 1-12.
14. A computer-readable storage medium, characterized in that: The method comprises computer instructions, which, when executed on an electronic device, cause the electronic device to execute the method as claimed in any one of claims 1 to 12.
