Image processing method and electronic device

Abstract

An image processing method and an electronic device can improve the convenience of secondary editing of a picture in an application program. The method comprises: obtaining skeleton information of a to-be-processed image, the skeleton information comprising a contour point set of each object in the to-be-processed image; and determining a sketch image corresponding to the to-be-processed image based on the skeleton information.

Description

Technical Field

[0001] This application relates to the field of image processing, and more specifically, to an image processing method and an electronic device. Background Technology

[0002] Currently, some text editing applications can insert images, thus enriching the content displayed by the application.

[0003] For example, note-taking applications, as a common type of text editing application, can insert images while displaying text information, enriching the note-taking interface. In some cases, users may want to further edit the images inserted into their notes. Currently, this cannot be done directly through note-taking applications; image processing software (such as Photoshop) is required. TM After editing the image, reinsert it into the Notes application.

[0004] Therefore, improving the ease of secondary image editing within applications has become an urgent problem to be solved. Summary of the Invention

[0005] This application provides an image processing method and apparatus that can improve the convenience of secondary editing of images in applications, thereby enhancing the user experience.

[0006] Firstly, an image processing method is provided, including:

[0007] Obtain the skeleton information of the image to be processed. The skeleton information includes the set of contour points of each object in the image to be processed.

[0008] Based on the skeleton information, the ink image corresponding to the image to be processed is determined.

[0009] It should be understood that an inkblot image refers to an image where the lines simulate hand-drawn lines by the user. In an inkblot image, each line can include multiple brushstrokes. Unlike RGB images, inkblot images present a handwritten or drawing style and retain brushstrokes, while RGB images have smoother, more natural color transitions, more closely resembling real-world scenes. Furthermore, since each line in an inkblot image is composed of multiple brushstrokes, brushstrokes in an inkblot image can be directly selected for editing. In contrast, RGB images are stored as vector layers, locking pixel data by default; therefore, secondary editing of RGB images requires rasterizing the layer first.

[0010] Contour points may refer to points that describe the shape and boundary of an element's contour in an image. One element in an image may correspond to a set of contour points, which can describe the shape and boundary of the element. For example, as Figure 2 shown in (a), an image includes the character "王", which includes three horizontal strokes and one vertical stroke, totaling four elements. Extraction of contour points from the image can obtain as Figure 2 shown in (b), a plurality of contour points. Wherein, each element may include a plurality of contours.

[0011] It should be noted that the set of contour points corresponding to one element can be stored in the form of a list or an array. Each list or array includes coordinate information corresponding to a plurality of contour points. Multiple sets of contour points corresponding to multiple elements in an image can describe the boundaries and shapes of each object in the image, therefore, multiple sets of contour points in the image can be regarded as the skeleton information of the image.

[0012] In some possible cases, converting an image to be processed into an ink image provides users with a new image style and enriches the types of images.

[0013] The image processing method provided by the embodiments of the present application obtains the skeleton information of the image to be processed by performing skeleton extraction on the image to be processed, then obtains the ink image corresponding to the image to be processed based on the skeleton information, wherein the skeleton information includes a set of contour points of each object in the image to be processed. That is, the image processing method provided by the embodiments of the present application can extract contour points indicating the boundary and shape of each object from the image to be processed, then perform image drawing and image rendering based on these contour points to obtain the ink image of the image to be processed. It can be understood that each line in the ink image includes a plurality of strokes, therefore, the electronic device can directly select partial strokes from the ink image for secondary editing, avoiding the situation that secondary editing can only be performed after rasterizing the image to be processed through a special image processing application, and improving the convenience of secondary editing of the image to be processed. In addition, since the ink image is an image that simulates user hand-drawn lines, converting the image to be processed into an ink image can also realize image style conversion and improve user experience.

[0014] With reference to the first aspect, in some embodiments of the first aspect, the step of obtaining skeleton information of the image to be processed, wherein the skeleton information includes a set of contour points of each object in the image to be processed, comprises: performing grayscale processing on the image to be processed to obtain a grayscale image corresponding to the image to be processed; performing binarization conversion based on the grayscale image to obtain a binary image; and performing skeleton extraction on the binary image to obtain the skeleton information of the image to be processed.

[0015] The image to be processed is typically an image containing a color matrix. Converting the color matrix to a grayscale matrix yields the corresponding grayscale image. In a grayscale image, the grayscale value of each pixel varies between 0 and 255. Pixels with a grayscale value of 0 are pure black, and pixels with a grayscale value of 255 are pure white.

[0016] Converting the image to grayscale simplifies the matrix containing color information into a matrix containing only grayscale information, reducing the amount of data in the image and facilitating subsequent processing.

[0017] The image processing method provided in this application embodiment can preprocess the image to be processed before extracting the skeleton. This preprocessing includes converting the image to grayscale to obtain a grayscale image corresponding to the image to be processed, and performing binarization conversion based on the grayscale image to obtain a binary image. Through these preprocessing, information in the image to be processed that affects the accuracy of skeleton extraction can be removed, improving the accuracy of the extracted skeleton information, thereby making the ink image corresponding to the image to be processed obtained based on the skeleton information more accurate.

[0018] Electronic devices can use bilateral filtering, Gaussian blurring, or median filtering to filter noise information in grayscale images to obtain filtered grayscale images. In conjunction with the first aspect, in some embodiments of the first aspect, a binary image is obtained by binarizing the grayscale image, including: filtering the grayscale image to obtain a filtered grayscale image; and performing a binarization transformation on the filtered grayscale image to obtain a binary image.

[0019] Noise in a grayscale image can affect the accuracy of obtaining a binary image from the grayscale image, and consequently the accuracy of obtaining skeleton information from the binary image. Therefore, noise in the grayscale image can be filtered first to obtain a filtered grayscale image, thus avoiding the impact of noise in the grayscale image on accuracy.

[0020] It should be understood that preprocessing also includes filtering the grayscale image to obtain a filtered grayscale image; and performing binarization on the filtered grayscale image to obtain a binary image.

[0021] Electronic devices can use bilateral filtering, Gaussian blurring, or median filtering to filter noise information in grayscale images to obtain filtered grayscale images.

[0022] In conjunction with the first aspect, in some embodiments of the first aspect, the above-described skeleton extraction of a binary image to obtain skeleton information of the image to be processed includes: performing an erosion operation on the binary image to obtain an eroded binary image, and using the eroded binary image as the updated binary image; performing a dilation operation on the eroded binary image to obtain a dilated image; determining the difference information between the binary image and the dilated image; merging the difference information and the skeleton information to obtain the updated skeleton information, and returning to execute "performing an erosion operation on the binary image to obtain an eroded binary image" until the updated binary image meets the preset conditions.

[0023] The image processing method provided in this application, after extracting the skeleton of a binary image to obtain the skeleton information of the image to be processed, can further perform erosion and dilation operations on the binary image to obtain a dilated image. Then, the dilated image and the binary image are compared to obtain difference information indicating that the binary image is missing detailed information in the image to be processed. The difference information and the skeleton information are then merged to obtain updated skeleton information. Since the difference information indicates that the binary image is missing detailed information in the image to be processed, merging the difference information and the skeleton information is equivalent to supplementing the skeleton information with the missing detailed information in the image to be processed. This allows the updated skeleton information to include more comprehensive skeleton information in the image to be processed, thereby making the ink image obtained based on the updated skeleton information more similar to the image to be processed, and improving the user experience.

[0024] In conjunction with the first aspect, in some embodiments of the first aspect, the preset condition includes that the updated binary image contains no non-zero pixels.

[0025] In conjunction with the first aspect, in some embodiments of the first aspect, the method further includes: determining whether the average gray value of the grayscale image is less than a preset threshold; if the average gray value of the grayscale image is less than the preset threshold, performing color inversion processing on the grayscale image to obtain a color-inverted grayscale image; and performing binarization conversion based on the color-inverted grayscale image to obtain a binary image.

[0026] Understandably, when the background area of ​​the image to be processed has a high proportion of white, it's advisable to invert the image before converting it to grayscale, so that the background area has a higher proportion of black. This avoids the problem of small differences between objects and the background area in the grayscale image after conversion.

[0027] In conjunction with the first aspect, in some embodiments of the first aspect, the above-described determination of the ink image corresponding to the image to be processed based on skeleton information includes: dividing multiple contour points in the skeleton information into N batches to obtain N batches of contour points; sequentially performing image drawing and image rendering on the N batches of contour points to obtain the ink image corresponding to the image to be processed.

[0028] Let's take a whiteboard image as an example. A whiteboard image usually contains multiple characters, so the skeleton information can indicate the boundaries and shapes of the strokes in each character of the whiteboard image.

[0029] Electronic devices can divide the contour points in the skeleton information into multiple batches according to the number of strokes. For example, an electronic device can divide the contour points into multiple batches, each containing 200 contour points. The list storing the path information of the skeleton information in the electronic device can be called the path, and the contour points in the skeleton information are usually stored in lists divided by different strokes. Each list typically includes all the contour points of a single stroke.

[0030] The image processing method provided in this application, in the process of obtaining an ink image based on the skeleton information of the image to be processed, can divide multiple contour points in the skeleton information into N batches, and then perform image drawing and image rendering based on the contour points in batches to obtain the ink image. This can control the number of contour points for each image drawing and image rendering within a reasonable range, avoiding the situation where image drawing and image rendering of a large number of contour points at the same time occupy a large amount of rendering engine, thereby causing excessive consumption of electronic device resources, and improving the efficiency of obtaining an ink image based on the skeleton information of the image to be processed.

[0031] In conjunction with the first aspect, in some embodiments of the first aspect, the method is applied to an electronic device, the electronic device including an image processing module and a rendering module, the image processing module being used to acquire skeleton information of an image to be processed; the rendering module being used to obtain an ink image corresponding to the image to be processed based on the skeleton information.

[0032] In conjunction with the first aspect, in some embodiments of the first aspect, the rendering module is further configured to divide multiple contour points in the skeleton information into N batches to obtain N batches of contour points; and sequentially perform image drawing and image rendering on the N batches of contour points to obtain the ink image corresponding to the image to be processed.

[0033] In a second aspect, an image processing apparatus is provided, including a unit for performing any of the methods in the first aspect. The apparatus may be a terminal device or a chip within a terminal device. The apparatus may include an input unit and a processing unit.

[0034] When the device is a terminal device, the processing unit may be a processor, and the input unit may be a communication interface; the terminal device may also include a memory for storing computer program code, which, when the processor executes the computer program code stored in the memory, causes the terminal device to perform any of the methods in the first aspect.

[0035] When the device is a chip within a terminal device, the processing unit can be an internal processing unit of the chip, and the input unit can be an output interface, pin, or circuit, etc.; the chip may also include a memory, which can be an internal memory of the chip (e.g., a register, cache, etc.) or an external memory (e.g., a read-only memory, random access memory, etc.); the memory is used to store computer program code, and when the processor executes the computer program code stored in the memory, the chip performs any of the methods in the first aspect.

[0036] In one possible implementation, the memory is used to store computer program code; the processor executes the computer program code stored in the memory, and when the computer program code stored in the memory is executed, the processor is used to perform: acquiring skeleton information of the image to be processed, the skeleton information including the set of contour points of each object in the image to be processed; and determining the ink image corresponding to the image to be processed based on the skeleton information.

[0037] Fourthly, a computer-readable storage medium is provided, the computer-readable storage medium storing computer program code, which, when executed by an image processing apparatus, causes the image processing apparatus to perform any of the image processing methods in the first aspect.

[0038] Fifthly, a computer program product is provided, the computer program product comprising: computer program code, which, when executed by an image processing apparatus, causes the image processing apparatus to perform any of the image processing methods in the first aspect.

[0039] The image processing method and electronic device provided in this application extract the skeleton of the image to be processed to obtain its skeleton information. Then, based on the skeleton information, an ink blot image corresponding to the image to be processed is obtained. The skeleton information includes a set of contour points of each object in the image to be processed. That is, the image processing method provided in this application can extract contour points indicating the boundaries and shapes of each object from the image to be processed, and then perform image drawing and rendering based on these contour points to obtain an ink blot image of the image to be processed. It is understood that each line in the ink blot image includes multiple brushstrokes. Therefore, the electronic device can directly select some brushstrokes from the ink blot image for secondary editing, avoiding the need for a dedicated image processing application to rasterize the image before secondary editing, thus improving the convenience of secondary editing of the image to be processed. Furthermore, since the ink blot image simulates lines drawn by the user, converting the image to be processed into an ink blot image can also achieve image style conversion, improving the user experience. Attached Figure Description

[0040] Figure 1 This is a schematic diagram of an RGB image and an ink blot image;

[0041] Figure 2 It is a schematic diagram of contour points and skeleton information in an image;

[0042] Figure 3 This is a schematic diagram of a hardware system applicable to the apparatus of this application;

[0043] Figure 4 This is a schematic diagram of a software system applicable to the apparatus of this application;

[0044] Figure 5 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0045] Figure 6 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0046] Figure 7 This is a schematic diagram illustrating an application scenario provided in an embodiment of this application;

[0047] Figure 8 This is a schematic diagram of an image processing method provided in an embodiment of this application;

[0048] Figure 9 This is a schematic diagram of another image processing method provided in an embodiment of this application;

[0049] Figure 10 This is a schematic diagram of an interface for entering a note-taking application provided in an embodiment of this application;

[0050] Figure 11 This is a schematic diagram of an interface for converting ink marks in an embodiment of this application;

[0051] Figure 12 This is a schematic diagram of another interface for converting ink images provided in an embodiment of this application;

[0052] Figure 13 This is a schematic diagram of another image processing method provided in an embodiment of this application;

[0053] Figure 14 This is a schematic diagram of a reminder processing device provided in this application;

[0054] Figure 15 This is a schematic diagram of an electronic device for image processing provided in this application. Detailed Implementation

[0055] To make the objectives, technical solutions, and advantages of this application clearer, the embodiments of this application will be described in further detail below with reference to the accompanying drawings.

[0056] It should be understood that "multiple" as mentioned in this application refers to two or more. In the description of this application, unless otherwise stated, " / " means "or", for example, A / B can mean A or B; "and / or" in this document is merely a description of the relationship between related objects, indicating that there can be three relationships, for example, A and / or B can mean: A exists alone, A and B exist simultaneously, and B exists alone.

[0057] Furthermore, to facilitate a clear description of the technical solutions of this application, terms such as "first" and "second" are used to distinguish identical or similar items with substantially the same function and effect. The terms "first" and "second," etc., in the specification and claims of the embodiments of this application are used to distinguish different objects, not to describe a specific order of objects. For example, "first target object" and "second target object," etc., are used to distinguish different target objects, not to describe a specific order of target objects.

[0058] In the embodiments of this application, the words "exemplarily" or "for example" are used to indicate examples, illustrations, or explanations. Any embodiment or design described as "exemplarily" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design solutions. Specifically, the use of the words "exemplarily" or "for example" is intended to present the relevant concepts in a specific manner.

[0059] In the description of the embodiments in this application, unless otherwise stated, "multiple" means two or more. For example, multiple processing units means two or more processing units; multiple systems means two or more systems.

[0060] Since the embodiments of this application involve a large number of image processing applications, for ease of understanding, the relevant terms and concepts of image processing that may be involved in the embodiments of this application will be introduced below.

[0061] (1) Image rendering.

[0062] Image rendering is an image processing method that generates images by simulating the interaction between light and objects. During image rendering, an image rendering engine is typically invoked to execute the corresponding image rendering tasks. It's understandable that image rendering is a complex image processing method; every line in the image is drawn by the image rendering engine. Therefore, image rendering typically consumes a significant amount of the image rendering engine's resources, meaning that image rendering tasks usually consume a large amount of computing resources in electronic devices.

[0063] (2) Bitmap image.

[0064] Bitmap images are a format that represents an image using an array of pixels. Each pixel in a bitmap records color information at a specific location within the image, thus forming a complete image. Images inserted into some applications are typically in bitmap format. For example, images inserted into a note-taking application for text editing are in bitmap format.

[0065] (3) RGB image.

[0066] RGB images are a type of digital image formed by combining the three primary color channels of red, green, and blue. By adjusting the brightness values ​​(0-255) of the three independent color channels of red, green, and blue and mixing them together, a rich color image visible to the human eye can be obtained.

[0067] (4) Grayscale image.

[0068] A grayscale image is an image representation that contains only brightness information and no color information. In a grayscale image, white and black are divided into 256 levels according to a logarithmic relationship, where 0 represents pure black and 255 represents pure white. In a grayscale image, each pixel retains only its brightness value and does not contain color information. Unlike black and white images (which contain only pure black and pure white), grayscale images have continuous transitional color levels between black and white.

[0069] (5) Binary Image.

[0070] A binary image is a digital image that contains only two color values. In a binary image, each pixel has only two possible values: a pixel value of 0 represents pure black, and a pixel value of 1 represents pure white. In some possible cases, a binary image can be regarded as a special grayscale image. Different from a grayscale image which includes continuous brightness levels from 0 to 255, a binary image only retains two values, 0 and 1, which can form a strong contrast in pixel brightness. Therefore, using a binary image can simplify complex backgrounds.

[0071] (6) Ink image.

[0072] An ink image may refer to an image in which the lines simulate hand-drawn lines by a user. In an ink image, each line may include multiple brush strokes. Different from an RGB image, an ink image presents a handwritten or painting style and retains brush strokes, while the color transition of an RGB image is smoother and more natural, and closer to real-life scenes. In addition, each line in an ink image is composed of multiple brush strokes, so brush strokes in a partial area of the ink image can be directly selected for editing. Whereas an RGB image is stored in the form of a vector layer, and pixel data is locked by default, so secondary editing of an RGB image requires rasterizing the layer before the secondary editing can be performed.

[0073] For example, an RGB image can be as shown in Figure 1 (a), and an ink image can be as shown in Figure 1 (b). The color transition in an RGB image is smoother and more natural, and closer to real-life scenes; an ink image presents a handwritten or painting style and retains brush strokes of lines, as shown in Figure 1 (b). It can be understood that an ink image can be a color image or a black-and-white image, which is not limited in the embodiments of the present application.

[0074] (7) Contour point.

[0075] A contour point in an image may refer to a point that describes the shape and boundary of the contour of an element in the image. One element in an image can correspond to a set of contour points, and the set of contour points can describe the shape and boundary of the element.

[0076] For example, as shown in Figure 2 (a), an image includes the character "王", which comprises three horizontal strokes and one vertical stroke, a total of four elements. Extracting contour points in the image can obtain as shown in Figure 2the plurality of contour points shown in (b) of . Wherein, each element may include a plurality of contours. It can be understood that the number of contour points included in each element may be the same or different, which is not limited in the embodiments of the present application. A set of contour points corresponding to each element can describe the shape and boundary of the corresponding element. For example, the shape and boundary of a vertical stroke can be described through the coordinate positions of a set of contour points corresponding to the vertical stroke.

[0077] It should be noted that a set of contour points corresponding to one element can be stored in the form of a list or an array. Each list or array includes coordinate information corresponding to a plurality of contour points.

[0078] (8) Skeleton information.

[0079] It should be understood that contour points in an image may be points describing the contour shape and boundary of an element in the image, and these contour points are usually stored in the form of a list or an array. A set of contour points can represent one element in the image. Multiple sets of contour points corresponding to multiple objects in an image can describe the boundaries and shapes of the objects in the image, so these multiple sets of contour points can be regarded as skeleton information of the image.

[0080] For example, as shown in Figure 2 (a) of , the skeleton information of an image may refer to lines formed by connecting a plurality of contour points in the character "王".

[0081] At present, secondary editing of an image usually requires special image processing software, and applications for editing text usually cannot directly perform secondary editing on the image, which results in complicated user operations and affects user experience.

[0082] In view of this, the image processing method provided in this application extracts the skeleton of the image to be processed to obtain the skeleton information of the image to be processed, and then obtains the ink image corresponding to the image to be processed based on the skeleton information. The skeleton information includes a set of contour points of each object in the image to be processed. That is to say, the image processing method provided in this application can extract contour points indicating the boundaries and shapes of each object from the image to be processed, and then perform image drawing and rendering based on these contour points to obtain the ink image of the image to be processed. It is understood that each line in the ink image includes multiple brushstrokes. Therefore, electronic devices can directly select some brushstrokes from the ink image for secondary editing, avoiding the need for a dedicated image processing application to rasterize the image before secondary editing, thus improving the convenience of secondary editing of the image to be processed. Furthermore, since the ink image simulates lines drawn by the user, converting the image to be processed into an ink image can also achieve image style conversion, improving the user experience.

[0083] The image processing method provided in this application can be applied to electronic devices. Optionally, the electronic device includes a terminal device, which can also be called a terminal, user equipment (UE), mobile station (MS), mobile terminal (MT), etc. The terminal device can be a mobile phone, smart TV, wearable device, tablet computer, computer with wireless transceiver function, virtual reality (VR) terminal device, augmented reality (AR) terminal device, wireless terminal in industrial control, wireless terminal in self-driving, wireless terminal in remote medical surgery, wireless terminal in smart grid, wireless terminal in transportation safety, wireless terminal in smart city, wireless terminal in smart home, etc. The embodiments of this application do not limit the specific technology or device form used in the terminal device.

[0084] For example, Figure 3A schematic diagram of the structure of electronic device 100 is shown. Electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0085] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0086] Processor 110 may include one or more processing units, such as: application processor (AP), modem processor, graphics processing unit (GPU), image signal processor (ISP), controller, memory, video codec, digital signal processor (DSP), baseband processor, and / or neural network processing unit (NPU), etc. Different processing units may be independent devices or integrated into one or more processors.

[0087] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0088] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0089] Electronic device 100 implements display functions through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0090] Display screen 194 is used to display images, videos, etc. Display screen 194 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, electronic device 100 may include one or N displays 194, where N is a positive integer greater than 1.

[0091] Electronic device 100 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0092] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0093] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then passed to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard RGB, YUV, or other formats. In some embodiments, the electronic device 100 may include one or N cameras 193, where N is a positive integer greater than 1.

[0094] Digital signal processors (DSPs) are used to process digital signals. Besides digital image signals, they can also process other digital signals. For example, when electronic device 100 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0095] Video codecs are used to compress or decompress digital video. Electronic device 100 may support one or more video codecs. Thus, electronic device 100 can play or record videos in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG2, MPEG3, MPEG4, etc.

[0096] An NPU (Neural Processing Unit) is a computational processor for neural networks (NNs). By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in electronic devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0097] It should be noted that any electronic device mentioned in the embodiments of this application may include more or fewer modules in electronic device 100.

[0098] The software system of electronic device 100 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This application embodiment uses the layered architecture Android system as an example to exemplify the software structure of electronic device 100.

[0099] Figure 4 This is a software structure block diagram of the electronic device 100 according to an embodiment of this application.

[0100] The layered architecture of the electronic device 100 divides the software into several layers, each with a clear role and division of labor. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0101] The application layer can include a series of application packages.

[0102] like Figure 4 As shown, the application package can include applications such as notes, gallery, and camera.

[0103] Gallery applications can store RGB images or bitmaps.

[0104] Camera apps can capture RGB images using the camera and store them in the gallery app.

[0105] Note-taking apps can be used for word processing. In some cases, images can also be inserted into note-taking apps, and electronic devices can convert the inserted images into ink images, making it easy for users to edit the ink images.

[0106] For example, a note-taking application can insert a photograph taken by the user of a teacher's writing on a blackboard. This image can be a bitmap or an RGB image. The electronic device can convert the blackboard image into an ink image, allowing the user to circle or write important information on the ink image.

[0107] In some cases, the application layer may also include modules such as a stroke control module, a rendering module, a network module, and an image processing module.

[0108] The stroke control module can be used to perform lasso processing, movement processing, deletion processing, color change processing, scaling processing, etc. on each stroke in the ink image.

[0109] The rendering module can be used to render image data and generate ink images that are displayed on the screen.

[0110] The network module can be used to retrieve bitmaps from a gallery application and insert them into a note-taking application.

[0111] The image processing module can be used to convert RGB images into ink images.

[0112] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications within the application layer. The application framework layer includes predefined functions. Various applications and modules within the application layer can interact with underlying modules through these interfaces to implement their respective functionalities.

[0113] like Figure 4 As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, etc.

[0114] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0115] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0116] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0117] The phone manager is used to provide communication functions for electronic device 100. For example, it manages call status (including connection and disconnection).

[0118] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0119] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating electronic devices, and flashing indicator lights.

[0120] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0121] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0122] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0123] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0124] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0125] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0126] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0127] A 2D graphics engine is a graphics engine for 2D drawing.

[0128] The kernel layer is the layer between hardware and software. The kernel layer includes at least display drivers, camera drivers, audio drivers, sensor drivers, Wi-Fi drivers, etc.

[0129] It should be noted that the electronic device mentioned in the embodiments of this application may include more or fewer modules of the above-mentioned electronic device.

[0130] The application scenarios provided by the embodiments of this application are described below with reference to the accompanying drawings.

[0131] For example, such as Figure 5 As shown, the image processing method of this application can be applied to the field of image style conversion. RGB images can be input into the suitable image processing module provided in the embodiments of this application for image processing to obtain ink images.

[0132] Application Scenario 1: Converting captured images of blackboard writing into ink images for secondary editing.

[0133] For example, during a lesson, users can use their mobile phones, laptops, or smart glasses to photograph the content written on the whiteboard by the teacher, obtaining an image of the whiteboard. Figure 6 As shown in (a) above. Then, the image processing method provided in this application embodiment can perform image processing on the blackboard image to obtain an ink image, such as... Figure 6 As shown in (b) of the image. Each line in the inkblot image can include multiple strokes, and strokes within a selected area can be directly edited. Therefore, users can select portions of the whiteboard content converted into inkblot images for further editing. For example, as shown in... Figure 6 As shown in (c), users can select and move parts of the text, or, as... Figure 6 As shown in (d), users can zoom in or out on the selected text (not shown in the image). Alternatively, users can also change the color of the selected text.

[0134] Application Scenario 2: Converting image styles.

[0135] The image processing method provided in this application embodiment can also be applied to convert RGB images into ink images. For example, landscape images taken by users are usually images recorded from real-world scenery, with a high degree of fidelity to the real scene, such as... Figure 7 As shown in (a) of this application, the image processing method provided in this embodiment can convert photographs with high fidelity to real-world scenes into handwritten or painted ink images, such as... Figure 7 As shown in (b) of the diagram.

[0136] Furthermore, since photographic images are typically in RGB format, which are usually stored as vector layers and have their pixel data locked by default, further editing of RGB images requires the use of image processing applications (such as Photoshop). TM Rasterizing the layer before secondary editing is inconvenient for users. Converting the RGB image to an inkblot image allows users to directly edit the inkblot image. For example, a user can select a portion of the scene in the photo and move it around. Figure 7 As shown in (c), users can select and move trees within an image. Here, a tree is just one object in the image, not the entire image. Alternatively, users can zoom in or out on the selected portion of the image, such as... Figure 7As shown in (d), the user can select a tree in the ink image for zooming in, wherein the tree is only an object in the image, not the entire image. Alternatively, the user can also change the color of the partially circled object, for example, the user can select the tree canopy in the image to separately change the color of the tree canopy, while the color of other areas in the image remains unchanged.

[0137] It should be understood that the above is an illustrative example of application scenarios, and does not impose any limitation on the application scenarios of the present application.

[0138] The following combines Figures 8 to 13 to describe in detail the image processing method provided by the embodiments of the present application.

[0139] Figure 8 is a schematic flowchart of an image processing method provided by an embodiment of the present application, as Figure 8 shown, the method includes:

[0140] S101, performing skeleton extraction on an image to be processed to obtain skeleton information of the image to be processed, wherein the skeleton information comprises a set of contour points of each object in the image to be processed.

[0141] Wherein, the image to be processed may be an image captured by a user, for example, the image to be processed may be a blackboard writing image obtained by the user taking a photo of the blackboard writing on a blackboard. The image to be processed may also be an image obtained by the user capturing a frame from a video, for example, the image to be processed may be an image obtained by the user capturing a frame from a teaching video. The image to be processed may also be a photo obtained by the user taking pictures of scenery and people.

[0142] It should be understood that the image to be processed may be an image including a color matrix, such as an RGB image. In some possible cases, an image inserted in a note application is a bitmap, and in this case, the bitmap also carries a color matrix of an RGB image.

[0143] Contour points may refer to points that describe the contour shape and boundary of an element in an image. One element in an image may correspond to a set of contour points, and this set of contour points may describe the shape and boundary of the element. For example, as Figure 2 shown in (a), the character "王" is included in an image, wherein the character "王" comprises three horizontal strokes and one vertical stroke, totaling four elements. Extracting contour points in the image can obtain results as shown in Figure 2The multiple contour points shown in (b) are illustrated. Each element may include multiple contours. It is understood that the number of contour points included in each element may be the same or different, and this embodiment does not impose any limitation on this. A set of contour points corresponding to each element can describe the shape and boundary of the corresponding element. For example, by drawing the coordinate positions of a set of contour points corresponding to a vertical line, the shape and boundary of the vertical line can be described.

[0144] It should be noted that a set of contour points corresponding to an element can be stored as a list or an array. Each list or array includes the coordinate information of multiple contour points. Multiple sets of contour points corresponding to multiple elements in an image can describe the boundaries and shapes of various objects in the image; therefore, multiple sets of contour points in an image can be regarded as the skeleton information of the image. The image processing method provided in this application embodiment can use a skeleton extraction algorithm to extract the skeleton information in the image to be processed. It can be understood that the boundary of a contour is usually a region where pixel values ​​change. The skeleton extraction algorithm can determine whether a pixel is part of a contour by detecting the changes between each pixel in the image and its corresponding neighboring pixels. If the pixel is part of a contour, it is treated as a contour point. After traversing the entire image, a set of multiple contour points can be obtained, which is the skeleton information.

[0145] For example, let's take an image containing text as an example. After obtaining multiple contour points, these points can be stored in a list. Then, based on a preset contour retrieval mode, it can be determined which stroke in the image these contour points belong to. The preset contour retrieval method can be either the RETR_EXTERNAL function or the RETR_LIST function. The RETR_EXTERNAL function can be used to determine whether a contour point is the outer contour of a stroke, while the RETR_LIST function can be used to extract all contour points and determine the hierarchical relationship of the strokes to which these contour points belong.

[0146] After obtaining multiple contour points, redundant contour points can be removed. Electronic devices can remove redundant contour points using an approximation method. This approximation method can refer to the `CHAIN_APPROX_SIMPLE` or `CHAIN_APPROX_NONE` function. The `CHAIN_APPROX_SIMPLE` function removes redundant contour points, while the `CHAIN_APPROX_NONE` function retains all contour points except the redundant ones.

[0147] S102. Obtain the ink image corresponding to the image to be processed based on the skeleton information.

[0148] As described above, skeleton information can refer to a set of multiple contour points in the image to be processed. These contour points indicate the boundaries and shapes of various objects in the image. Therefore, the electronic device performs image drawing and rendering based on the boundaries and shapes of various objects in the image indicated by the contour points, thereby obtaining the ink image corresponding to the image to be processed.

[0149] It is understood that in the process of drawing and rendering images based on the contour points in the skeleton information to obtain the ink image corresponding to the image to be processed, all contour points included in the skeleton information can be drawn and rendered at once to obtain the ink image, or the contour points in the skeleton information can be divided into multiple batches and drawn and rendered in batches to obtain the ink image. This application embodiment does not limit this.

[0150] The image processing method provided in this application extracts the skeleton of the image to be processed to obtain its skeleton information, and then obtains an ink blot image corresponding to the image to be processed based on the skeleton information. The skeleton information includes a set of contour points of each object in the image to be processed. In other words, the image processing method provided in this application can extract contour points indicating the boundaries and shapes of each object from the image to be processed, and then perform image drawing and rendering based on these contour points to obtain an ink blot image of the image to be processed. It is understood that each line in the ink blot image includes multiple brushstrokes. Therefore, electronic devices can directly select some brushstrokes from the ink blot image for secondary editing, avoiding the need for a dedicated image processing application to rasterize the image before secondary editing, thus improving the convenience of secondary editing of the image to be processed. Furthermore, since the ink blot image simulates lines drawn by the user, converting the image to be processed into an ink blot image can also achieve image style conversion, improving the user experience.

[0151] In the process of converting an image into an ink image, the number of strokes in the ink image is usually large. Taking a captured image of whiteboard writing as an example, a whiteboard writing image typically includes 2,000 to 3,000 strokes. In this case, if all the contour points in the skeleton information are drawn and rendered at once, it will occupy multiple rendering engines in the electronic device, resulting in excessive consumption of electronic device resources and causing users to wait for a period of time before the ink image is displayed.

[0152] Optionally, the electronic device can divide multiple contour points in the skeleton information into N batches, and then perform image drawing and image rendering based on the contour points in batches to obtain the ink image.

[0153] Let's take a whiteboard image as an example. A whiteboard image usually contains multiple characters, so the skeleton information can indicate the boundaries and shapes of the strokes in each character of the whiteboard image.

[0154] Electronic devices can divide the contour points in the skeleton information into multiple batches according to the number of strokes. For example, an electronic device can divide the contour points into multiple batches, each containing 200 contour points. The list storing the path information of the skeleton information in the electronic device can be called the path, and the contour points in the skeleton information are usually stored in lists divided by different strokes. Each list typically includes all the contour points of a single stroke.

[0155] The electronic device can iterate through all the contour points in the skeleton information's `path` using a `for()` loop, performing image drawing and rendering on a set of 200 stroke contour points at a time. The electronic device can also use the `subList(i, end)` method to extract a sublist (denoted as `subList`) from the `paths` list for the current batch. The `subList(i, end)` method takes a starting index `i` and an ending index `end` as parameters and returns a list containing all elements (contour points) between these two indices. This allows the electronic device to perform image drawing and rendering on the contour points between the two indices, thus implementing the function of batch image drawing and rendering.

[0156] During each iteration, the electronic device can calculate the end index (denoted as end) of the current sublist based on the starting index (i) of the current batch and the preset batch size (e.g., 200). The electronic device also needs to verify whether the end index exceeds the total length of the paths list to avoid out-of-bounds access.

[0157] The image processing method provided in this application, in the process of obtaining an ink image based on the skeleton information of the image to be processed, can divide multiple contour points in the skeleton information into N batches, and then perform image drawing and image rendering based on the contour points in batches to obtain the ink image. This can control the number of contour points for each image drawing and image rendering within a reasonable range, avoiding the situation where image drawing and image rendering of a large number of contour points at the same time occupy a large amount of rendering engine, thereby causing excessive consumption of electronic device resources, and improving the efficiency of obtaining an ink image based on the skeleton information of the image to be processed.

[0158] Understandably, in some cases, preprocessing can be performed on the image to be processed before skeleton extraction to remove information that affects the accuracy of skeleton extraction and improve the accuracy of the extracted skeleton information. The following will demonstrate... Figure 9 The embodiments shown will be described in detail below.

[0159] Figure 9 A flowchart illustrating another image processing method provided in this application embodiment is shown below. Figure 9 As shown, the method includes:

[0160] S201. Perform grayscale processing on the image to be processed to obtain a grayscale image.

[0161] The image to be processed is typically an image containing a color matrix. Converting the color matrix to a grayscale matrix yields the corresponding grayscale image. In a grayscale image, the grayscale value of each pixel varies between 0 and 255. Pixels with a grayscale value of 0 are pure black, and pixels with a grayscale value of 255 are pure white.

[0162] Converting the image to grayscale simplifies the matrix containing color information into a matrix containing only grayscale information, reducing the amount of data in the image and facilitating subsequent processing.

[0163] It should be understood that a color matrix typically includes information from three color channels: red (R), green (G), and blue (B), with each channel's value usually between 0 and 255. A grayscale matrix, on the other hand, contains only one channel, the grayscale value, which represents the brightness information of a pixel.

[0164] In some cases, a weighted average method can be used to convert the color matrix into a grayscale matrix.

[0165] For example, the color matrix can be processed using formula (1) to obtain the grayscale matrix corresponding to the color matrix, wherein formula (1) includes:

[0166] Gray value = Red coefficient × R + Green coefficient × G + Blue coefficient × B Formula (1).

[0167] Based on the sensitivity of the human visual system to different colors, the coefficient for red can be set to 0.299, the coefficient for green to 0.587, and the coefficient for blue to 0.114.

[0168] It should be noted that if the gray value obtained by formula (1) exceeds the range of 0-255, the gray value can be determined by normalization.

[0169] The gray value of each pixel can be calculated using formula (1), thereby constructing a gray matrix of the same size as the color matrix, and then obtaining the gray image corresponding to the image to be processed based on the gray matrix.

[0170] S202. Filter the noise information in the grayscale image to obtain the filtered grayscale image.

[0171] Noise in a grayscale image can affect the accuracy of obtaining a binary image from the grayscale image, and consequently the accuracy of obtaining skeleton information from the binary image. Therefore, noise in the grayscale image can be filtered first to obtain a filtered grayscale image, thus avoiding the impact of noise in the grayscale image on accuracy.

[0172] Electronic devices can use bilateral filtering, Gaussian blurring, or median filtering to filter noise information in grayscale images to obtain filtered grayscale images.

[0173] Bilateral filtering, a non-linear filtering method, is a compromise that combines spatial proximity and pixel similarity, considering both spatial information and grayscale similarity to achieve edge-preserving noise reduction. Bilateral filtering effectively preserves edges. The method demonstrates good results in filtering noise from the grayscale images in this embodiment.

[0174] In some cases, where the background area of ​​the image to be processed has a high proportion of white, it is advisable to invert the image before converting it to grayscale, so that the background area has a higher proportion of black. This can avoid the problem of small differences between objects and the background area in the grayscale image after conversion.

[0175] S203. Perform binarization on the filtered grayscale image to obtain a binary image.

[0176] As we can understand, the grayscale value of each pixel in a grayscale image is between 0 and 255. Electronic devices divide the pixels in a grayscale image into two categories by setting a threshold, resulting in a binary image. That is to say, the pixel values ​​in a binary image usually include two pixel values, such as 0 and 255.

[0177] For example, the pixel value of a pixel in a grayscale image that is greater than or equal to a preset threshold is set to 255, which means that the pixel is set to a white pixel, and the pixel value of a pixel that is less than the preset threshold is set to 0, which means that the pixel is set to a black pixel, thus obtaining a binary image.

[0178] The preset threshold can be a value selected by the user based on experience, or a value determined by the neural network model based on machine learning. This application does not limit this.

[0179] S204. Extract the skeleton from the binary image to obtain the skeleton information of the image to be processed.

[0180] Electronic devices can use a first algorithm to extract the skeleton from a binary image to obtain the skeleton information of the image to be processed. The skeleton information refers to multiple sets of contour points corresponding to multiple objects in the image, describing the boundaries and shapes of the objects.

[0181] For example, the first algorithm can refer to the zhang_suen_thinning algorithm, which is a classic method for thinning the edges of a binary image into a single-pixel skeleton. By iteratively deleting non-skeleton pixels in the image, the topological structure and key morphological features of the objects in the image are preserved, thereby obtaining skeleton information.

[0182] S205. Obtain the ink image corresponding to the image to be processed based on the skeleton information.

[0183] The image processing method provided in this application embodiment can preprocess the image to be processed before extracting the skeleton, remove information in the image to be processed that affects the accuracy of skeleton extraction, improve the accuracy of the extracted skeleton information, and thus make the ink image corresponding to the image to be processed based on the skeleton information more accurate.

[0184] During the process of extracting skeleton information using the first algorithm, some details in the image to be processed may be lost. In order to reduce the problem of inaccurate ink image caused by the loss of details, the embodiments of this application can also process the extracted skeleton information through steps S2041 to S2044 to obtain skeleton information with more complete image details.

[0185] S2041. Perform an erosion operation on the binary image to obtain the eroded binary image, and use the eroded binary image as the updated binary image.

[0186] Erosion is an operation that involves traversing a binary image through a sliding window and assigning the minimum value of the area covered by the window to the center pixel. If background pixels (such as the value 0 in the binary image) exist in the neighborhood, the center pixel is set to the background value. Erosion can shrink the target area, eliminate small noise in a binary image, separate adhered objects, and refine the edges of target objects.

[0187] After obtaining the eroded binary image, the eroded binary image can be used as the updated binary image. The next time the binary image needs to be called for erosion operation, the updated binary image is called, which is the eroded binary image obtained after the current erosion operation.

[0188] S2042. Perform a dilation operation on the eroded binary image to obtain a dilated image.

[0189] In contrast to erosion, dilation involves assigning the maximum value of the area covered by a sliding window to the center pixel. If a foreground pixel exists in the neighborhood (e.g., the value 255 in a binary image), the center point is set as the foreground value. Dilation can expand a target region, fill in internal holes, connect broken areas, and enhance edge continuity.

[0190] It should be noted that performing an erosion operation on the binary image first to obtain an eroded binary image, and then performing a dilation operation on the eroded binary image to obtain a dilated image, can remove noise from the binary image and preserve the shape of objects in the binary image to the greatest extent.

[0191] S2043. Determine the difference information between the binary image and the dilated image.

[0192] The binary image that has not undergone erosion before step S2041 is compared with the dilated image obtained after erosion and dilation operations are performed sequentially in steps S2041 and S2042 to obtain the difference between the two, that is, the difference information between the binary image and the dilated image.

[0193] It is understandable that performing an erosion operation on the binary image first to obtain an eroded binary image, and then performing a dilation operation on the eroded binary image to obtain a dilated image, can remove noise in the binary image and preserve the shape of objects in the binary image to the greatest extent. Therefore, the difference between the binary image and the dilated image is usually the detailed information missing from the image to be processed in the binary image.

[0194] S2044. Merge the difference information and skeleton information to obtain the updated skeleton information, and return to execute S2041 until the updated binary image meets the preset conditions.

[0195] Since the difference information between a binary image and a dilated image typically represents missing details from the image to be processed within the binary image, merging the difference information and the skeleton information is equivalent to supplementing the missing details in the original skeleton information. Electronic devices can supplement the missing details in the skeleton information through multiple erosion and dilation operations. After each erosion and dilation operation, the original binary image is updated using the eroded binary image until the updated binary image meets preset conditions, at which point the erosion and dilation operations cease.

[0196] The preset condition could be that the updated binary image has no non-zero pixels.

[0197] The image processing method provided in this application, after extracting the skeleton of a binary image to obtain the skeleton information of the image to be processed, can further perform erosion and dilation operations on the binary image to obtain a dilated image. Then, the dilated image and the binary image are compared to obtain difference information indicating that the binary image is missing detailed information in the image to be processed. The difference information and the skeleton information are then merged to obtain updated skeleton information. Since the difference information indicates that the binary image is missing detailed information in the image to be processed, merging the difference information and the skeleton information is equivalent to supplementing the skeleton information with the missing detailed information in the image to be processed. This allows the updated skeleton information to include more comprehensive skeleton information in the image to be processed, thereby making the ink image obtained based on the updated skeleton information more similar to the image to be processed, and improving the user experience.

[0198] The following is through Figures 10 to 13 The image processing method provided in this application embodiment will be described using the example of converting a captured blackboard image into an ink image in a note-taking application.

[0199] like Figure 10 As shown in (a), the display interface 10 of the electronic device displays an icon 101 for the "Notes" application. The icon 101 for the "Notes" application is used to open the note-taking application. In response to the user clicking the icon 101 for the "Notes" application, the display interface of the electronic device jumps to the main interface 20 of the note-taking application, as shown in (a). Figure 10 As shown in (b) of the diagram. The main interface 20 includes a "New" option 201. The "New" option 201 can be used to create a new note. For example... Figure 10 As shown in (c), the user clicks the "New" option 201 to create a new note, and the electronic device's display interface jumps to display interface 30. Display interface 30 has a "Take a Photo" option 301 and a "Select from Gallery" option 302.

[0200] If the user clicks the "Take Photo" option (301), such as Figure 11 As shown in (a) above. In response to the user clicking the "Take Photo" option 301, the display interface of the electronic device jumps to the photo-taking display interface, for example, as shown in (a). Figure 11 The display interface 40 is shown in (b) above. The display interface 40 includes a "Shoot" option 401. If the user clicks the "Shoot" option 401, in response to the click operation, an image within the current viewfinder is captured, and the captured image is displayed on the display interface 50, such as... Figure 11 As shown in (c) above. The display interface 50 also includes an ink image control 501, which is used to convert the captured blackboard image into an ink image. If the user clicks the ink image control 501, as shown... Figure 11As shown in (c), in response to a user's click, the electronic device converts the whiteboard image into an ink image. It is understood that converting a whiteboard image into an ink image involves significant rendering engine calls and takes considerable time to complete. Therefore, to enhance the user experience during the conversion process, the electronic device can display dynamic effects. For example, during the conversion process, the electronic device can display effects such as... Figure 11 The display interface 60 is shown in (d) above. On the display interface 60, dynamically displayed elements are overlaid on the whiteboard image. After the whiteboard image completes the conversion to an ink image, the electronic device can display, as shown below... Figure 11 The display interface 70 shown in (e) displays the converted ink image in the area where the original blackboard image was displayed.

[0201] If the user clicks option 302, such as "Choose from gallery", Figure 12 As shown in (a) above. In response to the user clicking the "Select from Gallery" option 302, the electronic device's display interface jumps to the gallery display interface, for example, as shown in (a). Figure 11 The display interface 80 is shown in (b) above. The display interface 80 includes thumbnails of multiple images. Users can select an image by clicking on its thumbnail. For example, as shown below... Figure 12 As shown in (b), if the user clicks on the image thumbnail 801, the user selects the image. In response to the user's click on the image thumbnail 801, the electronic device's display interface jumps to the display interface 90. The image 901 corresponding to the image thumbnail 801 selected by the user is displayed on the display interface 90, as shown in (b). Figure 12 As shown in (c) above. The display interface 90 also includes an ink image control 501, which is used to convert image 901 into an ink image. If the user clicks the ink image control 501, as shown below... Figure 12 As shown in (c), in response to a user's click, the electronic device converts image 901 into an ink image. It is understood that converting image 901 into an ink image involves significant rendering engine calls and takes considerable time to complete. Therefore, to enhance the user experience during the conversion process, the electronic device can display dynamic effects. For example, during the conversion of the whiteboard image into an ink image, the electronic device can display effects such as... Figure 12 The display interface 100 is shown in (d) above. On the display interface 100, dynamically displayed elements are overlaid on the image 901. After the image 901 completes the ink image conversion, the electronic device can display, as shown below. Figure 12 The display interface 110 shown in (e) displays the converted ink image in the area where the original image 901 was displayed.

[0202] The following is through Figure 13 How do the various modules in an electronic device interact to achieve the above? Figures 10 to 12 The detailed process of the function shown is explained. For example... Figure 13 As shown, the method includes:

[0203] S301. In response to the opening operation of the note-taking application, perform initialization operations on the note-taking application.

[0204] The opening of the note-taking application can refer to the user clicking the note-taking application icon, or it can be opened through other means; this application embodiment does not limit this. For example, the user can also open the note-taking application through a smart assistant, or the user can jump to the note-taking application from other applications by clicking a jump control in other applications.

[0205] For example, the process of a user clicking the note-taking app icon to open the note-taking app can be as follows: Figure 10 (a) to Figure 10 As shown in (c) in the figure.

[0206] S302, The note-taking application sends the first initialization command to the stroke control module.

[0207] The first initialization instruction is used to instruct the stroke control module to perform initialization. The first initialization instruction can be carried in the HnNoteEnginermpt function. The note application sends the HnNoteEnginermpt function to the stroke control module to instruct the stroke control module to perform initialization.

[0208] S303, The stroke control module responds to the first initialization instruction and performs an initialization operation.

[0209] S304, The stroke control module sends a second initialization command to the rendering module.

[0210] S305, the stroke control module sends a third initialization command to the image processing module.

[0211] S306. The rendering module responds to the second initialization instruction and performs an initialization operation.

[0212] S307, The image processing module performs an initialization operation in response to the third initialization instruction.

[0213] S308. In response to the insert image command, the note application creates a note canvas and displays the image to be processed as indicated by the insert image command.

[0214] The "Insert Image" command allows users to specify which image to insert into the note-taking application. Users can insert images by taking a photo, using the captured image as the inserted image; alternatively, users can select an image from their electronic device's gallery to insert into the note-taking application.

[0215] For example, the process of inserting an image into a note-taking application by taking a photo can be as follows: Figure 11 (a) to Figure 11 As shown in (c) in the figure.

[0216] For example, the process of selecting an image from the gallery and inserting it into a note-taking application can be as follows: Figure 12 (a) to Figure 12 As shown in (c) in the figure.

[0217] S309. In response to the command to start converting the ink image, the stroke control module displays the conversion animation on the display screen.

[0218] S310, the stroke control module sends image processing tasks to the image processing module.

[0219] The command to activate the ink image conversion function instructs the user to convert an image inserted into the note-taking application into an ink image. The note-taking application's display interface includes an ink image control; users can click the ink image control to activate the ink image conversion function and convert the image into an ink image.

[0220] For example, such as Figure 11 The display interface 50 shown in (c) includes an ink image control 501, which allows users to click on the ink image control 501 to convert the whiteboard image into an ink image.

[0221] For example, such as Figure 12 The display interface 90 shown in (c) includes an ink image control 501, which allows users to click on the ink image control 501 to convert the whiteboard image into an ink image.

[0222] The command to initiate the conversion of the ink image can be carried in the onActionitemClicked() function. When the user clicks the ink image control 501, the command to initiate the conversion of the ink image is sent to the stroke control module.

[0223] Transition animations can refer to dynamically displaying a group of elements on top of the image being processed. For example, transition animations can be like... Figure 11 (d) or as in Figure 12 As shown in (d) in the figure, dots of dynamically changing size are displayed on the image to be processed. Understandably, the size and color of these dots can change over time.

[0224] S311, The image processing module responds to the image processing task by extracting skeleton information from the image to be processed.

[0225] The specific process by which the image processing module extracts skeleton information from the image to be processed can be found above. Figure 8 and Figure 9 The illustrated embodiment will not be described in detail here.

[0226] S312, The image processing module sends the skeleton information of the image to be processed to the stroke control module.

[0227] S313, The stroke control module sends the skeleton information of the image to be processed to the rendering module.

[0228] S314 The rendering module performs image drawing and rendering based on the skeleton information of the image to be processed, and obtains the ink image.

[0229] S315, The rendering module sends the ink image to the stroke control module.

[0230] S316, The stroke control module displays ink images on the display screen.

[0231] After the ink image is displayed on the screen, the user can perform secondary editing on the ink image. For example, the user can select a part of the ink image and move, delete, change the color, or scale it.

[0232] It should be understood that although the steps in the flowcharts of the above embodiments are shown sequentially according to the arrows, these steps are not necessarily executed in the order indicated by the arrows. Unless explicitly stated herein, there is no strict order restriction on the execution of these steps, and they can be executed in other orders. Moreover, at least some steps in the flowchart may include multiple sub-steps or multiple stages. These sub-steps or stages are not necessarily completed at the same time, but can be executed at different times. The execution order of these sub-steps or stages is not necessarily sequential, but can be performed alternately or in turn with other steps or at least some of the sub-steps or stages of other steps.

[0233] It should be understood that the above examples are provided to help those skilled in the art understand the embodiments of this application, and are not intended to limit the embodiments of this application to the specific values ​​or scenarios illustrated. Those skilled in the art can obviously make various equivalent modifications or changes based on the above examples, and such modifications or changes also fall within the scope of the embodiments of this application.

[0234] The above text combined Figures 1 to 13 The image processing method of the embodiments of this application is described in detail below, and will be combined with Figure 14 and Figure 15This document describes in detail the apparatus embodiments of this application. It should be understood that the image processing apparatus in the embodiments of this application can execute the various image processing methods described in the foregoing embodiments of this application. That is, the specific working processes of the various products described below can be referred to the corresponding processes in the foregoing method embodiments.

[0235] Figure 14 This is a schematic diagram of the image processing apparatus provided in the embodiments of this application.

[0236] It should be understood that the image processing device 600 can perform... Figures 8 to 13 The image processing method shown; the image processing apparatus 600 includes: an acquisition unit 610 and a processing unit 620.

[0237] In one example, the acquisition unit 610 is used to acquire skeleton information of the image to be processed, which includes a set of contour points of each object in the image to be processed; based on the skeleton information, the ink image corresponding to the image to be processed is determined.

[0238] It should be noted that the image processing device 600 described above is embodied in the form of a functional unit. The term "unit" here can be implemented in software and / or hardware, and there is no specific limitation on this.

[0239] For example, a "unit" can be a software program, a hardware circuit, or a combination of both that implements the above functions. The hardware circuit may include an application-specific integrated circuit (ASIC), electronic circuitry, a processor (e.g., a shared processor, a proprietary processor, or a group processor) and memory for executing one or more software or firmware programs, integrated logic circuitry, and / or other suitable components that support the described functions.

[0240] Therefore, the units of the various examples described in the embodiments of this application can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0241] Figure 15 A schematic diagram of the structure of an electronic device provided in this application is shown. Figure 15 The dashed lines indicate that the unit or module is optional. The electronic device 700 can be used to implement the image processing method described in the above method embodiments.

[0242] The electronic device 700 includes one or more processors 701, which can support the implementation of the image processing method in the method embodiments of the electronic device 700. The processor 701 can be a general-purpose processor or a special-purpose processor. For example, the processor 701 can be a central processing unit (CPU), a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, such as discrete gate, transistor logic devices, or discrete hardware components.

[0243] The processor 701 can be used to control the electronic device 700, execute software programs, and process data from the software programs. The electronic device 700 may also include a communication unit 705 for inputting (receiving) and outputting (transmitting) signals.

[0244] For example, electronic device 700 may be a chip, communication unit 705 may be the input and / or output circuit of the chip, or communication unit 705 may be the communication interface of the chip, and the chip may be a component of terminal device or other electronic device.

[0245] For example, electronic device 700 can be a terminal device, communication unit 705 can be the transceiver of the terminal device, or communication unit 705 can be the transceiver circuit of the terminal device.

[0246] The electronic device 700 may include one or more memories 702, which store a program 704. The program 704 can be executed by the processor 701 to generate instructions 703, causing the processor 701 to execute the image processing method described in the above method embodiments according to the instructions 703.

[0247] Optionally, the memory 702 may also store data. Optionally, the processor 701 may also read the data stored in the memory 702, which may be stored at the same memory address as the program 704, or the data may be stored at a different memory address than the program 704.

[0248] The processor 701 and memory 702 can be configured separately or integrated together; for example, integrated on the system on chip (SOC) of the terminal device.

[0249] For example, the memory 702 can be used to store the related program 704 of the image processing method provided in the embodiments of this application, and the processor 701 can be used to call the related program 704 of the image processing method stored in the memory 702 when performing image restoration on the terminal device, and execute the image processing method of the embodiments of this application; for example, obtaining the skeleton information of the image to be processed, the skeleton information including the set of contour points of each object in the image to be processed; and determining the ink image corresponding to the image to be processed based on the skeleton information.

[0250] This application also provides a computer program product that, when executed by processor 701, implements the image processing method described in any of the method embodiments of this application.

[0251] The computer program product can be stored in memory 702, for example, program 704. Program 704 is finally converted into an executable object file that can be executed by processor 701 after processing such as preprocessing, compilation, assembly and linking.

[0252] This application also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a computer, implements the image processing method described in any of the method embodiments of this application. The computer program may be a high-level language program or an executable object program.

[0253] The computer-readable storage medium is, for example, memory 702. Memory 702 can be volatile memory or non-volatile memory, or memory 702 can include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM).

[0254] In this application, "at least one" means one or more, and "more than one" means two or more. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or multiple items. For example, at least one of a, b, or c can mean: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple.

[0255] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0256] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0257] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0258] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for example, the division of units is merely a logical functional division, and other division methods may exist in actual implementation; for example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces, and the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0259] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0260] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0261] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. An image processing method, characterized in that, include: Obtain the skeleton information of the image to be processed, wherein the skeleton information includes the set of contour points of each object in the image to be processed; Based on the skeleton information, the ink image corresponding to the image to be processed is determined.

2. The method according to claim 1, characterized in that, The process of obtaining skeleton information of the image to be processed includes a set of contour points for each object in the image to be processed, comprising: The image to be processed is converted to grayscale to obtain a grayscale image corresponding to the image to be processed. The grayscale image is binarized to obtain a binary image; Skeleton extraction is performed on the binary image to obtain the skeleton information of the image to be processed.

3. The method according to claim 2, characterized in that, The step of performing binarization transformation based on the grayscale image to obtain a binary image includes: The grayscale image is filtered to obtain a filtered grayscale image; The filtered grayscale image is binarized to obtain the binary image.

4. The method according to claim 2 or 3, characterized in that, The process of extracting the skeleton from the binary image to obtain the skeleton information of the image to be processed includes: The binary image is subjected to an erosion operation to obtain an eroded binary image, and the eroded binary image is used as the updated binary image. The eroded binary image is subjected to a dilation operation to obtain a dilated image; Determine the difference information between the binary image and the dilated image; The difference information and the skeleton information are merged to obtain updated skeleton information, and then the process of "performing an erosion operation on the binary image to obtain an eroded binary image" is returned until the updated binary image meets the preset conditions.

5. The method according to claim 4, characterized in that, The preset condition includes that the updated binary image contains no non-zero pixels.

6. The method according to any one of claims 2 to 5, characterized in that, The method further includes: Determine whether the average grayscale value of the grayscale image is less than a preset threshold; If the average grayscale value of the grayscale image is less than the preset threshold, the grayscale image is inverted to obtain an inverted grayscale image. The inverted grayscale image is binarized to obtain the binary image.

7. The method according to any one of claims 1 to 5, characterized in that, The step of determining the ink image corresponding to the image to be processed based on the skeleton information includes: The multiple contour points in the skeleton information are divided into N batches to obtain N batches of contour points; The outline points of the N batches are sequentially drawn and rendered to obtain the ink image corresponding to the image to be processed.

8. The method according to any one of claims 1 to 7, characterized in that, The method is applied to an electronic device, which includes an image processing module and a rendering module. The image processing module is used to obtain the skeleton information of the image to be processed; The rendering module is used to obtain the ink image corresponding to the image to be processed based on the skeleton information.

9. The method according to claim 8, characterized in that, The rendering module is further configured to divide the multiple contour points in the skeleton information into N batches to obtain N batches of contour points; and sequentially draw and render the N batches of contour points to obtain the ink image corresponding to the image to be processed.

10. An image processing apparatus, characterized in that, The image processing apparatus includes a processor and a memory, the memory being used to store a computer program, and the processor being used to call and run the computer program from the memory, causing the image processing apparatus to perform the image processing method according to any one of claims 1 to 9.

11. A chip, characterized in that, Includes a processor, which, when executing instructions, performs the image processing method as described in any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, causes the processor to perform the image processing method according to any one of claims 1 to 9.

13. A computer program product, the computer program product comprising: Computer program code, when executed by a processor, causes the processor to perform the image processing method according to any one of claims 1 to 9.

Images