A Real-Time Canvas Image Generation Method Based on Pen Stroke Trajectory and Text Conditions

By saving the canvas image before the contact begins and generating a local image based on the newly added brush stroke trajectory, the problem of processing the boundary between the new trajectory and the historical trajectory in real-time canvas image generation is solved. This achieves stable preservation of the local image and the existing canvas content, and reduces the drift of image position, outline and texture.

CN123118833APending Publication Date: 2026-11-17SHANGHAI MOPAN TECH CO LTD +1
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611618149.5
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-10-20
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing real-time canvas image generation methods have difficulty effectively distinguishing between new and historical trajectories when inputting continuous strokes, resulting in changes in the position, outline, and texture of unrelated areas. Furthermore, incomplete strokes can easily cause image position, outline, and texture drift during the update process.

Method used

By saving the canvas image before the contact begins and defining the local image area based on the newly added brush stroke trajectory, the local image is generated by reading the canvas image before the contact begins and generating the local image based on the brush stroke trajectory and text conditions. After the contact ends, the boundary is blended to ensure the stability of the existing canvas content.

Benefits of technology

It effectively separates the local changes of continuous brushstrokes from the existing canvas content, reduces the repeated generation of unrelated areas caused by new brushstrokes, maintains the continuity between the locally generated image and the adjacent content of the original canvas, and reduces image drift of incomplete brushstrokes during the update process.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123118833A_ABST
    Figure CN123118833A_ABST
Patent Text Reader

Abstract

This invention discloses a real-time canvas image generation method based on brush stroke trajectory and text conditions, belonging to the field of computer image processing technology. At the start of contact, the current canvas image is saved and the brush stroke sampling position is obtained. The sampling position is converted to canvas coordinates, and the local image region and trajectory overlay mask are determined based on the newly added brush stroke trajectory and brush width. Using the canvas image saved before contact begins as image input, the current brush stroke trajectory is used to form trajectory conditions, and the text conditions are encoded into text semantic conditions. A local image is generated and temporarily stored through a local conditional image generation model. When brush strokes are added, a local image is regenerated from the canvas before contact begins. After contact ends, a local image is regenerated based on the complete brush stroke trajectory, and after boundary fusion and edge checking through a writing mask, it is written into the current canvas image.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This invention relates to the field of computer image processing technology, specifically to a real-time canvas image generation method based on pen stroke trajectory and text conditions. Background Technology

[0002] Existing interactive image generation methods typically use user-drawn sketches, doodles, or partial masks as spatial control conditions, combined with text prompts to generate or edit the canvas image. In real-time drawing scenarios, the pen stroke trajectory continuously increases as the user moves, thus the current input conditions are constantly changing. Current processing methods mostly reconstruct the current sketch conditions after a pen stroke update and perform subsequent image generation accordingly, rarely distinguishing between newly added trajectories and previously stable historical trajectories.

[0003] When new brushstrokes only apply to the partial area of ​​the canvas, re-generating based on the current complete sketch can easily cause existing canvas content that hasn't changed in the previous iteration to participate in the processing again, resulting in changes to the position, outline, and texture of the main elements in unrelated areas. For continuous brushstrokes that are not yet complete, the local geometric shapes formed at different drawing stages may still be interpreted as complete structures, causing abrupt changes in the local structure of adjacent generated results. Therefore, real-time canvas image generation still needs to address the processing boundary issue between incremental changes in continuous brushstrokes and the stable preservation of existing canvas content. Summary of the Invention

[0004] To address the shortcomings of existing technologies, this invention provides a real-time canvas image generation method based on pen stroke trajectory and text conditions, in order to solve the problems mentioned in the background art.

[0005] To achieve the above objectives, the present invention provides the following technical solution: a real-time canvas image generation method based on pen stroke trajectory and text conditions, comprising the following steps: S1. Read the current canvas image, canvas coordinate relationship and text conditions, save the canvas image before the start of contact when the input device generates a contact start signal, and continuously obtain the pen stroke sampling position during the contact period. S2. Convert the brush stroke sampling position to canvas coordinates, determine the corresponding local image region based on the position traversed by the new brush stroke trajectory and the brush width, and read the local image region and its surrounding image from the canvas image saved before the contact begins; and form a trajectory overlay mask based on the actual position traversed by the new brush stroke trajectory and the brush width. The local image region and its surrounding image together constitute the image input range for local image generation. The trajectory overlay mask is used to limit the positions within the local image region where generation changes are allowed. S3. During the contact holding period, the corresponding local image area and the surrounding image in the canvas image saved before the contact started are used as image input. The current pen stroke trajectory is converted into trajectory conditions, and the text conditions are encoded into text semantic conditions. A local image is generated based on the image input, trajectory conditions and text semantic conditions through a preset local condition image generation model. The local image is temporarily stored and displayed in the corresponding local image area without replacing the canvas image saved before the contact started. S4. When the brush stroke trajectory continues to increase, the local image area is adjusted according to the newly added brush stroke trajectory, and the corresponding image is read again from the canvas image saved before the contact begins. The local image is generated again according to the added brush stroke trajectory and text conditions. The previously generated and temporarily stored local image is not used as the image basis for the next local image generation. S5. After the input device generates a contact end signal, the corresponding local image area is determined based on the complete stroke trajectory formed by this contact. The corresponding image is read from the canvas image saved before the contact started, and the local image is regenerated by combining the complete stroke trajectory and text conditions. A writing mask is formed based on the complete stroke trajectory. Within the boundary fusion zone corresponding to the boundary of the writing mask, the regenerated local image is weighted and fused with the canvas image saved before the contact started. After passing the edge check, the fused local image is written to the current canvas image.

[0006] Furthermore, S1 includes: The text condition uses text data that has been submitted when the input device generates a contact start signal; New text submitted after contact has begun is used when the next contact start signal arrives. Upon receiving the touch start signal, first save the current canvas image that has not been changed by this touch, and then obtain the touch sampling position of this touch. The sampling position that is valid in terms of canvas coordinates, is within the valid pixel range of the current canvas image, and has a valid contact mark is determined as the valid brush stroke sampling position.

[0007] Furthermore, S2 includes: New stroke trajectories are formed for adjacent valid stroke sampling positions that are not determined to have sampling gaps; The trajectory coverage area is formed based on the actual canvas position traversed by the newly added brush stroke trajectory and the brush width determined at the start of contact; The horizontal and vertical coordinates of the canvas pixel positions within the trajectory coverage area are read respectively. The left, right, upper, and lower boundaries of the local rectangular image range are determined by the minimum, maximum, minimum, and maximum values ​​of the horizontal and vertical coordinates of the canvas pixel positions respectively. Based on the left, right, upper, and lower boundaries, a minimum bounding rectangle corresponding to the trajectory coverage area is formed, and the minimum bounding rectangle is determined as a local image region; When any of the left, right, top, and bottom boundaries exceeds the effective pixel range of the current canvas image, the boundary is limited to the corresponding effective pixel boundary of the canvas.

[0008] Furthermore, S2 also includes: When the newly formed local rectangular image region intersects with an existing local image region and their boundaries are directly connected, the two are regarded as the same local image region. When there is an interval between the newly formed local rectangular image range and the existing local image region that has not been traversed by the brush stroke trajectory, the corresponding local image regions are retained respectively; Read each local image region and its surrounding image from the canvas image saved before the contact begins.

[0009] Furthermore, S3 includes: The current stroke trajectory is formed by all valid trajectories between the first valid stroke sampling position after the start of this contact and the latest valid stroke sampling position. Each local image generation reads the corresponding local image region and surrounding image from the canvas image saved before the contact begins, and uses the local image region and surrounding image as image input; the trajectory position within the image input range of the current stroke trajectory forms trajectory condition data, and the text conditions obtained at the start of this contact form text condition data; the preset local condition image generation model includes an image encoding unit, a trajectory condition encoding unit, a text encoding unit, a conditional denoising generation unit, and an image decoding unit; the image encoding unit of the preset local condition image generation model converts the image input into latent space image features, the trajectory condition encoding unit converts the trajectory condition data into trajectory condition features with corresponding spatial positional relationships to the latent space image features, and the text encoding unit converts the text conditions into text semantic features; the conditional denoising generation unit forms updated image features based on the latent space image features, trajectory condition features, and text semantic features, and the image decoding unit converts the updated image features into a local image.

[0010] Furthermore, S4 includes: When a new stroke trajectory is located inside an existing local image region, the boundary of the local image region remains unchanged, and the new stroke trajectory is merged into the current stroke trajectory. When a new stroke trajectory extends beyond an existing local image area, the actual image range traversed by the new stroke trajectory is incorporated into the corresponding local image area. After adjusting the local image area, the adjusted local image area is read again from the canvas image saved before the touch begins, and the local image is generated again based on the increased brush stroke trajectory and text conditions.

[0011] Furthermore, S4 also includes: When a local image region exceeds the single input size allowed by the current image project, the trajectory segments between adjacent valid stroke sampling positions are added to the current local image range in the order of sampling the valid stroke sampling positions. After adding the next trajectory segment, the image width of the current local image range is compared with the maximum input image width of the current image project, and the image height is compared with the maximum input image height of the current image project. If there is a comparison result that exceeds the corresponding maximum input value, the valid stroke sampling position at the end of the current local image range is used as the starting position of the next local image range, and the next trajectory segment is added to the next local image range. When a new stroke trajectory enters a local image range, a new local image is generated for that local image range and the adjacent local image ranges that are directly connected to its boundary. Local image areas that were not traversed by the newly added brush strokes and have no boundary connections will retain their existing temporary display content.

[0012] Furthermore, S5 includes: After the contact ends, the first valid pen stroke sampling position after the start of this contact is taken as the starting point of the complete pen stroke trajectory, and the last valid pen stroke sampling position before the end of the contact is taken as the ending point of the complete pen stroke trajectory. The local image area is redefined based on the actual location traversed by the complete brush stroke trajectory and the brush width used in this contact. The corresponding image is reread from the canvas image saved before the contact begins, and the local image is regenerated based on the complete brush stroke trajectory and text conditions.

[0013] Furthermore, S5 also includes: A writing mask is formed based on the actual location traversed by the complete brush stroke trajectory and the brush width used in this contact, and a boundary blending band is formed along the inner side of the writing mask boundary. For the pixel positions within the boundary blending band, the generated image weight is determined based on the distance from the pixel position to the writing mask boundary and the width of the boundary blending band. According to the generated image weight and the original canvas image weight, which is equal to the sum of the generated image weight, the pixels of the newly generated local image are weighted and blended with the pixels at the same canvas coordinate positions in the canvas image saved before the contact begins, forming the local image to be written. Within the edge inspection band inside the writing mask boundary, inspection positions are determined according to a predetermined inspection interval, and the local image to be written at each inspection position is obtained. The absolute difference between pixels at the same canvas coordinate position in each color channel and the canvas image saved before the contact begins is calculated, and the maximum value among the absolute differences of each color channel is determined as the color channel difference of the corresponding inspection position. The number of inspection positions where the color channel difference exceeds the allowable difference predetermined by the current image project is counted as the number of out-of-tolerance inspection positions. When the number of out-of-tolerance inspection positions is not greater than the allowable number predetermined by the current image project, the local image to be written is written to the corresponding local image region. When the number of out-of-tolerance inspection positions is greater than the allowable number, the reading range of the outer image is expanded, the canvas image saved before the contact begins is reread and the local image is regenerated, and boundary fusion and edge checking are re-executed.

[0014] Compared with existing technologies, this invention provides a real-time canvas image generation method based on pen stroke trajectory and text conditions, which has the following beneficial effects: 1. When the input device generates a contact start signal, the canvas image before the contact begins is saved. During continuous stroke input, the local image area is defined based on the actual position traversed by the new stroke trajectory. Each time a local image is generated, the canvas image saved before the contact begins is reread, and the local image temporarily stored in the previous round is not used as the basis for subsequent generation. After the contact ends, the local image is regenerated based on the complete stroke trajectory and written to the current canvas image. This allows the local changes of continuous strokes to be processed separately from the existing canvas content, reducing the repeated generation of unrelated areas caused by new strokes, and reducing image position, contour, and texture drift caused by incomplete strokes being successively solidified during continuous updates.

[0015] 2. Based on the brush stroke sampling position, canvas coordinates, and brush width, determine the local image area where the brush stroke actually acts, and adjust this area as new brush stroke trajectories are added. At the same time, read the surrounding image from the canvas image saved before the contact begins as the basis for the locally generated surrounding image, so that long-distance, curved, and rotating brush strokes can be locally processed according to their actual range, avoiding expanding the image update area by using the overall outer range of the entire brush stroke. After the contact ends, the writing range is redefined using the complete brush stroke trajectory, which helps to maintain the continuity between the locally generated image and the adjacent content of the original canvas, and reduces the repeated rewriting of canvas areas that have not undergone brush stroke changes.

[0016] 3. Under the same canvas size, this method does not use the entire canvas as the image input for each round of local generation. Instead, it reads the corresponding local image region and surrounding image based on the current brush stroke trajectory. Separate trajectory segments are used to form local image regions. When a local image region exceeds the allowed input size, it is divided along the trajectory acquisition sequence. This ensures that canvas areas far from the current brush stroke and not actually involved by the brush stroke are not included in the current round of image encoding, conditional generation, and image decoding. When the total number of pixels in the reading range corresponding to each local image region and surrounding image is less than the number of pixels in the entire canvas, the image space processed in this round of generation can be reduced.

[0017] 4. The brushstroke trajectory serves as the spatial condition for generating local conditional images, the textual conditions determined at the start of contact serve as the semantic conditions for the generated content, and the canvas image saved before the start of contact serves as the common image basis for each round of local generation. Local images formed during the contact period are only used for temporary display and are not used as the image basis for the next round of local image generation; after the contact ends, the final local image generation, boundary blending, and writing are performed based on the complete brushstroke trajectory, thereby enabling the brushstroke trajectory to define the position and structure of local image generation, reducing the direct rewriting of existing canvas content during continuous brushstroke input. Attached Figure Description

[0018] Figure 1 This is a schematic diagram of the process of a real-time canvas image generation method based on pen stroke trajectory and text conditions according to the present invention. Detailed Implementation

[0019] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0020] Example: Figure 1A real-time canvas image generation method based on pen stroke trajectory and text conditions is presented, including the following steps: S1. Read the current canvas image, canvas coordinate relationship and text conditions, save the canvas image before the start of contact when the input device generates a contact start signal, and continuously obtain the pen stroke sampling position during the contact period. S2. Convert the brush stroke sampling position to canvas coordinates, determine the corresponding local image region based on the position traversed by the new brush stroke trajectory and the brush width, and read the local image region and its surrounding image from the canvas image saved before the contact begins; and form a trajectory overlay mask based on the actual position traversed by the new brush stroke trajectory and the brush width. The local image region and its surrounding image together constitute the image input range for local image generation. The trajectory overlay mask is used to limit the positions within the local image region where generation changes are allowed. S3. During the contact holding period, the corresponding local image area and the surrounding image in the canvas image saved before the contact started are used as image input. The current pen stroke trajectory is converted into trajectory conditions, and the text conditions are encoded into text semantic conditions. A local image is generated based on the image input, trajectory conditions and text semantic conditions through a preset local condition image generation model. The local image is temporarily stored and displayed in the corresponding local image area without replacing the canvas image saved before the contact started. S4. When the brush stroke trajectory continues to increase, the local image area is adjusted according to the newly added brush stroke trajectory, and the corresponding image is read again from the canvas image saved before the contact begins. The local image is generated again according to the added brush stroke trajectory and text conditions. The previously generated and temporarily stored local image is not used as the image basis for the next local image generation. S5. After the input device generates a contact end signal, the corresponding local image area is determined based on the complete stroke trajectory formed by this contact. The corresponding image is read from the canvas image saved before the contact started, and the local image is regenerated by combining the complete stroke trajectory and text conditions. A writing mask is formed based on the complete stroke trajectory. Within the boundary fusion zone corresponding to the boundary of the writing mask, the regenerated local image is weighted and fused with the canvas image saved before the contact started. After passing the edge check, the fused local image is written to the current canvas image.

[0021] S1 specifically refers to: The current canvas image is read from the editable image buffer data of the current image project. When creating a new image project, the background image data after canvas initialization is read; when continuing to edit an existing image project, the canvas data after the previous stroke was successfully completed and the image was written is read. The width, height, and pixel format of the current canvas image are based on the actual image parameters used in the image project file. When the display window zooms in, zooms out, or pans the canvas, only the position of the image on the display device changes; the actual pixel size and pixel arrangement of the current canvas image remain unchanged.

[0022] The canvas coordinate relationship is read from the currently effective display parameters of the image display interface. The display parameters correspond to the starting position, display range and scaling relationship of the current canvas image in the drawing area, and are used to map the screen position obtained by the input device to the pixel position of the current canvas image.

[0023] Each stroke sampling position uses the canvas coordinate relationship that was in effect when the sampling event occurred. The obtained sampling position retains the corresponding canvas pixel position and will not be changed again due to subsequent canvas scaling and translation. If a valid canvas coordinate relationship cannot be obtained when the current sampling event occurs, the sampling event will not be added to the current stroke trajectory. After the image display interface restores a valid canvas coordinate relationship, the stroke sampling position will continue to be obtained from the next valid sampling event.

[0024] The text conditions are read from the text data that has already been submitted in the text input area. Text submission is based on the receipt of a text confirmation event by the image generation interface. If there is no valid text data that has been submitted at the start of the contact, the local image generation based on the text conditions will not be initiated, and the current canvas image will remain unchanged. When a new text confirmation event is generated after the start of the contact, the new text will not replace the text conditions already read in this contact, but will be used from the arrival of the next contact start signal.

[0025] The input device is a drawing input device capable of continuously outputting directional positions. Drawing input events are processed separately from interface operation events used for canvas zooming and panning. Only contact events received by the current drawing tool are included in the stroke sampling. When the input device supports multiple simultaneous contact positions, the contact identifier that generates the start signal of this contact and is received by the drawing tool is used as the data source for this stroke. The positions of other contact identifiers are not added to the current stroke trajectory until the contact ends.

[0026] Each valid sample acquires the screen's horizontal and vertical coordinates, sampling time, and contact flag. When the input interface simultaneously outputs an event sequence flag, the event sequence flag is retained along with the corresponding sampled data for subsequent determination of whether there is a data interruption between adjacent sampled events. The contact start signal is based on the current drawing input transitioning from never touching the canvas to effectively touching the canvas, and the contact end signal is based on the same drawing input transitioning from effectively touching the canvas to not touching the canvas.

[0027] Upon receiving the touch start signal, the system first reads and saves the current canvas image that has not yet been altered by the current brush stroke, and then receives the first valid sampling position of the current brush stroke. The canvas image saved before the touch starts retains its original pixel data until the current touch ends normally, and local images generated during the current touch are not written to this saved data.

[0028] When continuously acquiring brush stroke sampling positions, first check whether the sampling position falls within the valid pixel range of the current canvas image based on the canvas coordinate relationship. Sampling data that falls within the valid pixel range and has a valid contact mark is included in the current brush stroke; if the sampling position moves out of the valid canvas range, the sampling position outside the canvas is not used for subsequent local image region determination. When the input position re-enters the valid canvas range, data is acquired from the first valid sampling position after re-entry.

[0029] When the input interface resumes output after a data interruption, it retains the last valid sampling position before the interruption and the first valid sampling position after the resumption, and determines whether a sampling gap exists based on the sampling times of both. If the input interface also provides an event sequence marker, it combines the event sequence marker to determine whether a sampling gap exists. If a sampling gap is determined to exist, the actual unacquired trajectory is not filled between the two valid sampling positions.

[0030] When the input device is disconnected, the image drawing window loses input focus, or the current drawing input is canceled by the system, it is not considered a normal contact end signal. The final image writing process corresponding to the complete stroke is not initiated, and the local image data that has not yet been written to the current canvas image during this contact is cleared. The current canvas continues to use the canvas image saved before the contact started.

[0031] The sampling frequency of the input device is read from the current operating parameters of the device driver. The sampling frequency is only used as a device parameter to continuously obtain the pen stroke position and is not used as a basis for deleting sampling points or changing the spatial position of the trajectory. The valid sampling positions obtained during the contact are retained according to the actual acquisition order, and the corresponding trajectory data are read from the already obtained valid sampling positions for subsequent local image updates.

[0032] S2 specifically refers to: The pen stroke sampling positions entering this step use valid sampling data continuously acquired and retained by S1. Each pen stroke sampling position still retains the screen horizontal coordinate, screen vertical coordinate, and corresponding canvas coordinate relationship at the time the sampling event occurred. When processing the current pen stroke sampling position, the screen position is mapped to the pixel position in the current canvas image based on the starting display position of the canvas in the drawing area and the display scaling relationship that takes effect at the sampling time.

[0033] The canvas coordinates start from the top-left corner of the current canvas image, with the horizontal position corresponding to the image width and the vertical position corresponding to the image height. When the display window is scaled or translated in subsequent steps, the previously obtained brush stroke sampling positions are not reprocessed using the changed display relationships. Sampling events that S1 has already determined—such as invalid canvas coordinate relationships, sampling positions outside the valid canvas area, and interrupted input data—are not re-inputted in this step.

[0034] After the position transformation is completed, new stroke trajectories are formed for adjacent valid stroke sampling positions that are not determined to have sampling gaps; if a sampling gap is determined to exist between adjacent valid stroke sampling positions, no actual unobtained trajectory is added between them. The pen width reads the pen parameters that are already in effect on the drawing interface at the start of this contact.

[0035] When the drawing interface records the pen width in canvas pixels, this value is used directly; when the drawing interface records the pen width in display pixels, the value is mapped to the canvas pixel width according to the display scaling relationship that takes effect at the start of the contact, and the canvas pixel width remains unchanged during this contact.

[0036] When the brush width setting is changed during the contact hold period, the new setting is used from the next contact and does not change the range of the current brush stroke trajectory. For a newly added brush stroke trajectory, the actual canvas position traversed between two adjacent valid brush stroke sampling positions is used as the center range of the trajectory, and it is covered on both sides of the trajectory according to the fixed brush width, so that the coverage range is consistent with the range of canvas pixels that the drawing interface can reach with the same brush width.

[0037] When there are multiple pixel gaps between adjacent sampling positions, the intermediate passing position is obtained along the continuous direction between the two sampling positions, and the two sampling points are not treated as independent discrete positions. The trajectory coverage range formed in this way serves as the basic range of the local image region. For the currently added stroke trajectory, the local rectangular image range is formed by the smallest bounding rectangle of the trajectory coverage range in the canvas coordinate system; the horizontal and vertical coordinates of the canvas pixel positions in the trajectory coverage range are read respectively, and the left, right, upper, and lower boundaries of the local rectangular image range are determined by the minimum, maximum, minimum, and maximum values ​​of the horizontal and vertical coordinates of the canvas pixel positions, respectively; the smallest bounding rectangle corresponding to the trajectory coverage range is formed based on the left, right, upper, and lower boundaries, and the smallest bounding rectangle is determined as the local image region; when any of the left, right, upper, and lower boundaries exceeds the effective pixel range of the current canvas image, the boundary is limited to the corresponding effective pixel boundary of the canvas; when the current stroke continues to grow and generates the next newly added stroke trajectory, the local rectangular image range corresponding to the next newly added stroke trajectory is determined in the same way.

[0038] When the newly formed local rectangular image area intersects with an existing local image area or the boundary is directly connected, the two are read as the same continuous local image area; when there is an interval between the two that is not actually traversed by the brush stroke, they are retained as two local image areas, and the total bounding rectangle of the entire brush stroke is not used to include the blank canvas in the middle.

[0039] The subsequent image processing range for long-distance curved strokes, rotating strokes, and strokes with sampling gaps is determined according to the actual canvas position traversed by the effective stroke trajectory. When the newly added stroke trajectory is completely within an existing local image region, the local image region is not expanded; only the current stroke trajectory stored within that region is updated. When the newly added stroke trajectory extends beyond an existing local image region, only the image range actually involved by the new trajectory is incorporated into the corresponding local image region.

[0040] When the current brush stroke retraces a position that has already entered a local image region, that position retains its original region classification and does not expand again due to repeated traversal. The local rectangular image range is used to define the image input window used for this round of local image generation and is not directly used as a generation mask that allows changes to all pixels. A trajectory overlay mask is formed based on the actual position traversed by the new brush stroke trajectory and the current brush width. Pixels within the local rectangular image range that are outside the trajectory overlay mask retain the corresponding pixels from the canvas image saved before the start of the contact. The trajectory overlay mask is used to define the positions where image changes are allowed in this round. After the local image region is formed, the original pixels corresponding to that region are read from the canvas image saved before the start of the contact, while simultaneously reading a fixed-pixel-width outer image along the outer boundary of the local image region into the canvas.

[0041] The outer perimeter image is only used to provide adjacent existing image content for subsequent local image generation, and does not change the actual write boundary of the local image region. The width of the outer perimeter image is measured in canvas pixels and read from the generation configuration of the current image project. The generation configuration simultaneously saves the maximum input image width and maximum input image height currently in effect during the local image generation process; the initial outer perimeter image width is set so that the total read range formed by the local image region and the outer perimeter image does not exceed the maximum input image width and maximum input image height. The same initial outer perimeter image width is used during the same image project operation, and the initial outer perimeter image width is not temporarily changed based on the generation result corresponding to a single stroke.

[0042] When a local image region is close to the edge of the current canvas image, the outer image is read only from the effective pixel range of the canvas, without filling in non-existent pixels outside the canvas boundary. When the current image project's generation configuration requires a fixed input image width and a fixed input image height, the fixed input image width and fixed input image height are read from the current image project's generation configuration. When a local image region is restricted by the canvas boundary on one side and the corresponding outer image cannot be obtained, without changing the canvas coordinates of the local image region, the outer image reading range is increased to the side away from the canvas boundary, so that the total reading range formed by the local image region and the outer image reaches the fixed input image width and fixed input image height.

[0043] When two or more non-intersecting local image regions are formed during the contact, each local image region reads the corresponding original pixels and surrounding images from the canvas image saved before the contact begins, and then enters the local image generation process of S3 respectively. The canvas content that has not been touched by the actual brush stroke is not merged between multiple regions because they belong to the same contact.

[0044] After completing the reading of the local image area and surrounding image corresponding to the newly added pen stroke trajectory, the pen stroke trajectory data formed between the start of this contact and the latest valid pen stroke sampling position is retained, so that S3 can read the current complete pen stroke trajectory, text conditions and corresponding local image data on the canvas before the start of the same contact and continue processing.

[0045] S3 specifically refers to: Upon entering this step, the current stroke trajectory is formed by continuously acquiring stroke sampling positions in S1 and confirming their validity in S2, arranged in chronological order of acquisition time. The current stroke trajectory corresponds to all valid trajectory data acquired between the first valid stroke sampling position after the start of this contact and the latest valid stroke sampling position, not just the newly added stroke trajectory segment. Each position in the current stroke trajectory uses the canvas coordinates transformed by S2, and the trajectory width uses the fixed pen width acquired at the start of this contact, ensuring that different refresh times during the same contact process correspond to the same trajectory space range.

[0046] Before generating the local image, the local image region determined by S2 and the surrounding image are read from the canvas image saved before the touch begins. The read local image region corresponds to the image range actually traversed by the current pen stroke trajectory, and the surrounding image corresponds to the pre-determined adjacent image range outside the local image region.

[0047] When the current brush stroke trajectory is located in multiple discontinuous local image regions, the original image of each local image region is read separately at its corresponding position. Canvas content not traversed by the brush stroke between different regions is not included in the current local image input range. Subsequently, the trajectory positions located within the local image region and the surrounding image range are extracted from the current brush stroke trajectory to form trajectory input data corresponding to the current local image size. The positions in the trajectory input data are derived from the valid sampling positions after coordinate transformation by S2, without rereading the changed coordinate relationships of the display window.

[0048] The text conditions continue to use the submitted text data read and fixed by S1 at the start of contact. During this contact hold, if a new text confirmation event is generated in the text input area, the text conditions used for the current local image generation are not changed, and the new text content is only read when the input device generates a contact start signal next time.

[0049] When generating the current local image, the local image region and the surrounding image are read from the canvas image saved before the contact begins as the image input. The trajectory position in the current stroke trajectory within the corresponding local image region and the surrounding image range is converted into local coordinates relative to the reading range of the local image to form trajectory input data. The text conditions obtained at the beginning of this contact are used as text input.

[0050] The current image engineering preloads a trained local conditional image generation model. This model employs a latent space diffusion generation model and includes an image encoding unit, a text encoding unit, a trajectory conditional encoding unit, a conditional denoising generation unit, and an image decoding unit. The image encoding unit converts the local image region and its surrounding image into latent space image features. The text encoding unit converts the text conditions obtained at the start of the current contact into text semantic features. The trajectory conditional encoding unit reads the trajectory overlay mask formed by S2 and forms a trajectory condition map according to the corresponding position of the trajectory overlay mask within the current local image reading range. It then converts the trajectory condition map into trajectory condition features that have a corresponding spatial positional relationship with the latent space image features.

[0051] The conditional denoising generation unit uses latent space image features as the baseline image features, uses trajectory condition features to limit the spatial locations where changes are allowed in the current round, and uses text semantic features as the semantic conditions for the generated content. At each denoising processing location, latent space locations outside the trajectory condition coverage area continue to use the baseline image features formed from the canvas image saved before the start of the contact. Latent space locations within the trajectory condition coverage area form updated image features based on the current pen stroke trajectory and text semantic features. After completing the preset denoising processing, the image decoding unit converts the updated latent space image features into a locally generated image.

[0052] Textual conditions are used to define the semantic attributes of the generated content, trajectory conditions are used to define the spatial position of the semantic content in the current local image, and the canvas image saved before the contact begins is used to provide the original image content and surrounding image context corresponding to this local generation. During the current contact, these three elements maintain their respective semantic conditions, spatial conditions, and image-based functions.

[0053] The local image generation process reads the currently effective maximum input image width, maximum input image height, and pixel format from the current image project's generation configuration. The maximum input image width and maximum input image height together define the allowed single input size for the current image project. When processing the aforementioned image input, trajectory input data, and text input, the reading range formed by the local image region and its surrounding image does not exceed the maximum input image width and maximum input image height, resulting in generated image data corresponding to the current local image region. The generated image data maintains the same canvas position correspondence as the current canvas image for subsequent temporary storage, display, and verification of size, pixel format, and pen stroke end position.

[0054] The image basis in the current local image generation process always comes from the canvas image saved before the touch begins, and the previously generated and displayed local images are not read as the new image basis; when the current brush stroke continues to increase, the corresponding local image is reconstructed based on the complete brush stroke trajectory after the increase.

[0055] After a local image is generated, the image width, height, pixel format, and the current pen stroke end position corresponding to this generation are obtained. If the image width and height are inconsistent with the current local image region reading size, the generated image will not be displayed; if the pixel format of the generated image is inconsistent with the pixel format used by the current canvas image and the image project does not have a corresponding conversion method configured, the generated image will not be displayed; if the pen stroke end position corresponding to the generated image is earlier than the latest valid pen stroke sampling position already obtained, the generated image will not be used as the current display content.

[0056] The verified local image is written into temporary image data and used only to display the corresponding local image area. The coverage area of ​​the temporary image data is consistent with the local image area determined in S2, and the canvas image content saved before the start of the contact continues to be displayed outside the local image area.

[0057] The peripheral image is only used as a reference for local image generation and is not included in the direct writing range of the temporary image data. When multiple local images are generated during the contact holding period, each generated local image only replaces the previous round of temporary image in the corresponding local image area, without modifying the canvas image saved before the contact started, nor changing the base image read for subsequent local image generation.

[0058] When the input device sampling frequency is higher than the local image update frequency, continue to obtain valid stroke sampling positions according to S1, and form a new stroke trajectory according to S2. New stroke sampling positions obtained during the period before the local image generation request returns are not discarded. After the current local image processing is completed, the canvas image saved before the start of the touch is reread based on the latest valid stroke sampling position, and a new local image is generated based on the updated current stroke trajectory. After the previous local image generation returns, if the corresponding stroke end position is already earlier than the latest valid stroke sampling position, the previous return result will not cover the current temporary display area.

[0059] Local image display updates are performed according to the current operating parameters of the display device. The update frequency does not change the actual pen stroke sampling positions obtained by the input device, nor does it delete valid trajectory data that has already been obtained. When local image generation cannot be returned temporarily, the canvas image saved before the start of the current contact is displayed, and the valid pen stroke sampling positions generated by the input device continue to be received. After the image generation process resumes, the canvas image saved before the start of the contact is reread according to the currently formed complete pen stroke trajectory, and a local image is generated. Historical local images that have lost their current pen stroke correspondence are not sequentially added in.

[0060] The local image generated in this step is only used as display data in the current contact process and is not used as the base image for the next round of local image generation. When the brush stroke sampling position continues to increase, S4 still reads the corresponding range based on the canvas image saved before the start of the contact and regenerates the local image according to the increased brush stroke trajectory; after the input device generates a contact end signal, S5 then performs the final local image generation and writes the current canvas image according to the complete brush stroke trajectory formed in this contact.

[0061] S4 specifically refers to: Upon entering this step, the current stroke trajectory still uses all trajectory data continuously acquired in S1, formed into a valid trajectory in S2, and used for local image generation in S3. After the latest valid stroke sampling position changes relative to the end position of the stroke already involved in local image generation in the previous round, the newly added valid trajectory between the two is determined as the new stroke trajectory for this round. The new stroke trajectory maintains the same canvas coordinate system and brush width as the current stroke trajectory, and its range remains unchanged regardless of changes in the display window or user adjustments to brush parameters during the contact hold period.

[0062] When the newly added pen stroke trajectory is completely within the local image region of the previous round, the boundary of the local image region remains unchanged. The newly added pen stroke trajectory is simply incorporated into the current pen stroke trajectory according to the acquisition order, and the trajectory input data in the local image region is re-extracted according to the updated current pen stroke trajectory.

[0063] When a new brush stroke extends beyond the local image area of ​​the previous round, the new coverage area is first formed based on the actual canvas position traversed by the new brush stroke and the fixed brush width at the start of this contact, and then the new coverage area is incorporated into the corresponding local image area.

[0064] When a newly added coverage area intersects with or directly connects to an existing local image region, they are merged into the same local image region. When there is an interval between the newly added coverage area and the existing local image region that is not actually traversed by the current stroke, they are retained as separate local image regions, and the image processing range is not directly expanded by the total bounding rectangle of the two. After the local image region is adjusted, the original image within the adjusted local image region is read again from the canvas image saved before the start of the contact, and the corresponding outer image is reread according to the outer width determined in S2.

[0065] In the expanded local image region, both the original and newly added ranges use the canvas image saved before the start of the contact as the image basis for the local image generation in the current round. The temporary local image generated in the previous round does not enter the local image generation in the next round.

[0066] When the current stroke turns and passes through a previously visited position again, the revisited position remains in the current stroke trajectory. It is not deleted from the local image area due to repeated passage, nor is the image reading range of that position reduced because the temporary image of the corresponding position has been displayed in the previous round.

[0067] As the current brush stroke gradually approaches a closed trajectory from an open trajectory, before the closure is completed, each round of local image generation is based on the current brush stroke trajectory that has already been formed. After the current brush stroke trajectory is closed by adding a new brush stroke trajectory, the same local image area and surrounding image are read from the canvas image saved before the contact begins, and the local image is regenerated based on the current closed brush stroke trajectory. The temporary image content formed based on the open trajectory before the closure is not used.

[0068] When the current brush stroke continues to grow, causing the local image area to exceed the input size allowed by the current image project, the trajectory segment between two adjacent effective brush stroke sampling positions and the coverage area formed by the brush width are added to the current local image range in the order of sampling of effective brush stroke positions, starting from the initial effective brush stroke sampling position of the current local image range.

[0069] If, after adding the next trajectory segment, the width of the current local image range exceeds the maximum input image width of the current image project, or the image height exceeds the maximum input image height of the current image project, then the trajectory segment will not be incorporated into the current local image range. Instead, the effective stroke sampling position at the end of the current local image range will be used as the starting position of the next local image range, and the next local image range will continue to be formed from that trajectory segment. This process continues for subsequent trajectory segments until the currently acquired stroke trajectories are fully divided.

[0070] The width of each local image range does not exceed the maximum input image width of the current image project, and the image height does not exceed the maximum input image height of the current image project. Adjacent local image ranges continue to read the outer image of a fixed width. The far canvas area that has not been actually traversed by the current stroke will not be included in the local image update due to the overall expansion of the long stroke.

[0071] When a new stroke trajectory enters only one local image range, the local image is regenerated only for that local image range and the adjacent local image ranges that are directly connected to the boundary of that local image range; other local image ranges that have no boundary connection with the new stroke trajectory and have not been passed through by the new trajectory in this round will maintain the temporary display state of the previous round and will not trigger the generation of local images again.

[0072] When multiple separate local image regions exist simultaneously, each local image region reads the canvas image saved before the start of the contact, the corresponding surrounding image, and the trajectory input data according to the updated current stroke trajectory, and generates a local image separately. Canvas content between different local image regions that was not actually traversed by the current stroke is not included in the current round of image generation. After generating a new local image based on the updated current stroke trajectory, text conditions, adjusted local image regions, and surrounding images, the position of the stroke end corresponding to the returned image is first checked.

[0073] When the pen stroke end position corresponding to the returned image is consistent with the latest valid pen stroke sampling position that has entered the processing stage, the returned image is written to the temporary image data of the corresponding local image area and overwrites the previous round of temporary display content of that area; when the pen stroke end position corresponding to the returned image is earlier than the latest valid pen stroke sampling position that has entered the processing stage, the returned image is not used to update the current temporary display, but the canvas image saved before the start of the contact is reread according to the updated current pen stroke trajectory and the next round of local image generation is entered.

[0074] The display range of the generated local image is still limited to the corresponding local image area. The surrounding image is only used as a peripheral reference during the current generation and is not used as the direct display replacement range in the current round. During the contact holding period, regardless of whether the local image area is adjusted, expanded, or read in segments, the local image generated in this step is only written to the temporary image data. It does not modify the canvas image saved before the contact started in this step, nor is it used as the base image for subsequent rounds of rereading.

[0075] If the input device generates a cancellation event, disconnection event, the drawing window loses input focus, or the canvas coordinate relationship fails during this step, the generation of subsequent local images of the current stroke is stopped, the temporary image data that has not yet been written to the current canvas image during this contact is cleared, the current display is restored to the canvas image saved before the contact started, and the abnormally interrupted current stroke is not included as a complete stroke in the subsequent writing process.

[0076] This step is only repeated according to the newly added pen stroke trajectory when the input device is continuously in a valid contact state and continues to acquire adjacent valid pen stroke sampling positions that are not determined to have sampling gaps; after the input device generates a normal contact end signal, the local image update in this step is stopped, and the complete current pen stroke trajectory formed during this contact is handed over to S5 to perform the final local image generation and current canvas image writing.

[0077] S5 specifically refers to: After the input device generates a normal contact end signal, it first stops the local image update corresponding to S3 and S4, and only retains the valid pen stroke sampling positions obtained during this contact. Then, it reads the last valid pen stroke sampling position before the contact ends, and checks whether there are still valid sampling positions in the sampling data saved during this contact that have not yet been incorporated into the current pen stroke trajectory; if so, according to the sampling gap determination result determined in S1 and the new pen stroke trajectory formation rule in S2, the valid sampling positions that have not been processed and have not been determined to have sampling gaps are incorporated into the complete pen stroke trajectory.

[0078] When the end-of-contact event itself does not have valid canvas coordinates, the interface position where the event occurs is not used as the endpoint of the trajectory. Instead, the last valid stroke sampling position before the end-of-contact event is used as the endpoint of the complete stroke trajectory. The starting point of the complete stroke trajectory is the first valid stroke sampling position after the start of this contact, and the ending point is the last valid stroke sampling position before the normal end of the contact. Only trajectory segments obtained during this contact, confirmed as valid by S1, and not judged by S1 and S2 to have sampling gaps are retained. Trajectory segments that have been judged to have sampling gaps, positions outside the valid canvas area, and data generated during the abnormal cancellation phase are not included in the complete stroke trajectory.

[0079] After the complete stroke trajectory is formed, the local image area to be written is redefined based on the actual position of the complete stroke trajectory on the canvas and the fixed brush width at the beginning of this contact. The local image area is still determined according to the range formed in S2, and the range expanded for temporary display in stages S3 and S4 but ultimately not traversed by the effective trajectory is not directly used; when the complete stroke trajectory contains multiple trajectory segments separated by sampling gaps, corresponding local image areas are formed for each segment, and the blank canvas not actually traversed by the stroke is not directly covered by the total bounding rectangle of these consecutive segments.

[0080] After the local image region is determined, the corresponding outer image is reread according to the outer width that has been fixed in S2. The basic content of the local image region and the outer image is obtained again from the canvas image saved before the contact begins. The temporary local images formed in S3 and S4 are not read, nor are any pixels in the temporary local images used as the base image for the final generation.

[0081] The text conditions continue to use the submitted text content read and fixed at the start of the contact in S1, ensuring that the complete stroke trajectory corresponds to the same text conditions from start to finish, and does not change due to new text confirmation events occurring during the contact. When regenerating the local image, the trajectory position of the complete stroke trajectory in each local image region continues to use the canvas coordinates formed in S2 and the pen width fixed in this contact, thereby obtaining the complete trajectory input data corresponding to the current local image region.

[0082] If the local image region does not exceed the single input size allowed by the current image project, perform a final local image generation on the local image region and its surrounding image. If the local image region exceeds the single input size allowed by the current image project, divide the complete stroke trajectory into multiple local image ranges according to the local image range division rules determined by the effective stroke sampling position acquisition order in S4. The image width of each local image range does not exceed the maximum input image width of the current image project, and the image height does not exceed the maximum input image height of the current image project. Local image generation is then performed sequentially according to the order in which the complete stroke trajectory first enters each local image range.

[0083] The fixed outer image width of S2 is retained as a peripheral reference between adjacent local image ranges. Canvas areas far from the complete brush stroke trajectory and not actually traversed by the brush stroke are not included in the final local image generation even if the span of the complete brush stroke is large. After each local image region or range is generated, the image width, image height, pixel format, and corresponding local trajectory relationship of the returned image are obtained first.

[0084] If the width and height of the returned image are inconsistent with the current read range, the returned image will not be prepared for writing; if the pixel format of the returned image is inconsistent with the pixel format used by the current canvas image and the image project does not have a corresponding conversion method configured, the returned image will not be prepared for writing; if the returned image does not correspond to the complete pen stroke trajectory used by the current local image area, it will also not be prepared for writing.

[0085] The returned image that passes verification is initially reserved as the local image to be written in this step, and is not immediately written to the current canvas image. For cases where multiple local image ranges are processed sequentially, the next local image range is processed only after the returned image of the current local image range passes verification; if any local image range fails verification, the final writing process stops, and the canvas image saved before the start of the contact is retained as the current canvas image, without performing partial writing.

[0086] After the local image to be written is verified for size, pixel format, and trajectory relationships, a writing mask is formed based on the trajectory coverage area corresponding to the complete brush stroke trajectory. In the writing mask, the position actually covered by the complete brush stroke trajectory according to the current brush width is recorded as the generated writing area. A boundary blending band of a preset width is formed inwards from the boundary of the generated writing area. The positions outside the writing mask continue to use the canvas image pixels saved before the start of the contact. The width of the boundary blending band is read from the boundary blending configuration of the current image project and remains unchanged during the same image project execution.

[0087] For each pixel position within the boundary blending band, the distance from that pixel position to the boundary of the generated writing area is obtained, and the generated image weight is determined based on this distance and the width of the boundary blending band. When the pixel is located at the boundary of the generated writing area, the generated image weight is zero. As the pixel position moves inward into the generated writing area, the generated image weight gradually increases, and after reaching the inner boundary of the boundary blending band, the generated image weight becomes one. The generated image weight is determined by the ratio of the distance from the pixel position to the boundary of the generated writing area to the width of the boundary blending band, and this ratio is limited to between 0 and 1. The original canvas image weight corresponding to the generated image weight is determined by subtracting the generated image weight from one, so that the sum of the generated image weight and the original canvas image weight at the same pixel position is one.

[0088] The output pixels within the boundary fusion band are weighted and synthesized according to the weights of the generated image, using pixels at the same coordinates as the local image pixels to be written and pixels at the same coordinates as the canvas image saved before the contact begins. Pixels located inside the boundary fusion band in the generated writing region use the local image pixels to be written, while pixels outside the writing mask continue to use the canvas image pixels saved before the contact begins. After the above fusion is completed, edge checking is performed according to the edge checking criteria.

[0089] After the local image is written, an edge check is performed. The edge check uses the pre-saved check parameters of the current image project. The check parameters correspond one-to-one with the current pixel format and include the edge check band width, check interval, allowed difference, and allowed number. The edge check band width and check interval are both in canvas pixels, read from the edge check configuration of the current image project, and remain unchanged during the same image project using the same pixel format.

[0090] Taking RGB pixel format as an example, an edge inspection band is formed along the inner side of the boundary of the generated writing area corresponding to the writing mask. The inspection position is determined according to the inspection interval within the edge inspection band. For each inspection position, the generated pixel of the local image to be written at that inspection position is read, and the corresponding pixel of the canvas image saved before the start of contact is read according to the same canvas coordinates. The absolute difference of the R channel, the absolute difference of the G channel, and the absolute difference of the B channel of the two pixels are obtained respectively. The maximum value of the three color channel absolute differences is determined as the color channel difference of the inspection position.

[0091] After obtaining the color channel difference for each inspection location, locations where the color channel difference exceeds the allowable difference are marked as out-of-tolerance inspection locations, and the number of out-of-tolerance inspection locations is counted. The allowable difference and allowable number are read from the edge inspection configuration of the current image project, and the corresponding values ​​remain unchanged when the same pixel format is used in the same image project. If the number of out-of-tolerance inspection locations is not greater than the allowable number, the current edge inspection passes; if the number of out-of-tolerance inspection locations is greater than the allowable number, the current edge inspection fails.

[0092] When the number of out-of-tolerance check locations exceeds the allowable number, the local image to be written is not written to the current canvas image. Instead, the peripheral expansion step size saved in the current image project generation configuration is read, and the current peripheral image reading range is expanded according to the peripheral expansion step size.

[0093] The peripheral expansion step size is in units of canvas pixels. It is set during the image engineering deployment phase based on the maximum input image width, maximum input image height, and initial peripheral image width allowed in the local image generation process, and remains unchanged during the same image engineering operation.

[0094] After expanding the reading range of the outer image, the local image area and the expanded outer image are read again from the canvas image saved before the contact begins. The local image is then regenerated using the same complete brush stroke trajectory, the same text conditions, and the same local image area. Boundary fusion is re-executed based on the same write mask and boundary fusion band to form a new local image to be written, and edge checking is re-executed.

[0095] The maximum number of times the peripheral image reading range can be expanded is jointly limited by the maximum input image width, maximum input image height, initial peripheral image width, and peripheral expansion step size allowed by the current image project. After each expansion, the reading range formed by the local image region and the peripheral image must not exceed the maximum input image width and maximum input image height. If the number of out-of-tolerance check locations still exceeds the allowed number after reaching the maximum allowed number of expansions, all local images to be written corresponding to this contact are discarded, and the local images are not written to the current canvas image. The current canvas image continues to remain the canvas image saved before the contact began.

[0096] Once the edge check passes and all local image regions involved in this contact have passed verification, the local image to be written is written to the current canvas image according to the range of the local image region corresponding to the complete stroke trajectory; the canvas pixels outside the local image region continue to retain the canvas image content saved before the contact started, and the corresponding positions of the outer image are not included in the formal writing range.

[0097] When multiple separate local image regions exist simultaneously, each local image region completes edge checks and passes all checks before being written to the current canvas image. If any local image region fails the check, no local image region will be formally written in this contact.

[0098] After the writing is complete, the current canvas image becomes the new canvas basis for S1 to read at the start of the next contact. All temporary local image data formed in stages S3 and S4 is cleared, and the current contact ends. When the input device generates a normal contact start signal again, S1 will reread the current canvas image, text conditions, and canvas coordinate relationships, and save the new canvas image before the contact starts to enter the next round of processing.

[0099] The actual execution process of this method is illustrated using the example of continuously drawing the cup handle in an existing canvas image. This embodiment uses a 1024 pixel × 1024 pixel RGB canvas. The cup body and table background have already been formed on the canvas, and the current canvas image is in an editable state. The text input area has already provided the text condition that the red ceramic cup is placed on a wooden table, and the drawing tool uses a fixed brush width. The above canvas size and brush width are only used as image engineering configurations for this embodiment to illustrate the continuous brush input process and do not constitute a limitation on the scope of protection of the invention.

[0100] When the user prepares to draw the handle on the right side of the cup, the stylus generates a contact start signal upon first contact with the canvas. At this point, the current canvas image, current canvas coordinates, and submitted text conditions are read, and the canvas image before the contact start is saved before receiving the first valid sample position of this stroke. This saved image already contains the complete cup body and desktop background, but does not include the handle to be drawn. During this contact period, even if the displayed interface shows a partial image formed based on the stroke, the canvas image saved before the contact start retains the original pixel data from the moment the stroke began.

[0101] As the stylus moves from the upper right to the lower right of the cup, it continuously acquires stroke sampling positions and converts them into corresponding positions in a 1024 pixel × 1024 pixel canvas according to the canvas coordinates at the time of each sampling position. Adjacent valid sampling positions form new stroke trajectories in the input order. In the initial stage, when only a short outward-curving arc is formed, the local image area corresponding to the right side of the cup is determined based on the actual position traversed by the short arc and the fixed pen width. Then, the local image area and its surrounding image are read from the canvas image saved before the contact begins. The left side of the cup, the area on the table away from the cup, and other canvas areas not traversed by this stroke are not included in the scope of this local image processing.

[0102] The first local image is generated based on the currently formed short arc trajectory, the text conditions obtained at the start of contact, and the local canvas image saved before contact began. During the generation process, the local canvas image corresponding to the right side of the cup handle and the surrounding image are converted into image features, the trajectory covering mask corresponding to the current short arc is used to form trajectory condition features, and the text condition "the red ceramic cup is placed on the wooden table" is converted into text semantic features. The local condition image generation model then generates the first local image based on the above image features, trajectory condition features, and text semantic features.

[0103] The first partial image is only used as temporary display content during the current contact process, covering the corresponding position on the right side of the cup, and is not written to the canvas image saved before the contact begins. Since the current brushstroke has not yet formed a complete cup handle, the local shape displayed at this time only corresponds to the current short arc stage and is not used as the base image when subsequent brushstrokes are added.

[0104] The stylus continues downwards and rotates back towards the cup, with the newly added stroke trajectory gradually forming the outline of the outer and lower sides of the cup handle. When the newly added stroke trajectory is still within the existing local image area, the boundary of the local image area remains unchanged, and the newly acquired valid trajectory is incorporated into the current stroke trajectory; when the newly added stroke trajectory goes beyond the existing area, the local image area is expanded only according to the actual position traversed by the new trajectory, and the corresponding surrounding image is reread. After each update of the local image area, the base image corresponding to the right side of the cup is re-obtained from the canvas image saved before the contact began, without reading the temporary local image formed in the previous short arc phase.

[0105] As the stylus returns to the lower right side of the cup, the stroke gradually forms the complete outline of the cup handle from an open arc. At this point, local image generation is performed again based on all valid stroke trajectories formed from the start of contact to the current end. The temporary local image formed based on the open arc is not used in this round of base image reading; the area near the handle still uses the image of the cup and the table before the stroke as the image base, only changing the content of the stroke trajectory used for this round of processing.

[0106] After the user completes the outline of the cup handle and lifts the stylus, a normal contact end signal is generated. Local image updates cease during the contact holding period. The last valid stroke sample position before the end is read, and valid sample positions not yet processed in the previous round are added to the current stroke trajectory. Subsequently, the final local image area used is redefined based on the complete cup handle trajectory formed from the start to the end of the contact. This area is still formed according to the actual position traversed by the complete trajectory and a fixed pen width, excluding the entire outer range of the cup body and other areas that were read for display during the contact but were ultimately not traversed by the valid trajectory.

[0107] In the final generation stage, the local image corresponding to the handle and the surrounding image are read again from the canvas image saved before the contact begins. At the same time, the complete handle trajectory and the text conditions fixed at the beginning of this contact are read to regenerate the local image to be written. The short arcs, semicircles, and temporary images of each stage before closure that have already been displayed during the contact are not used as the basis for the final generated image.

[0108] After generation, the size, pixel format, and correspondence of the returned local image with the complete brush stroke trajectory are checked. Once verified, a write mask is formed based on the actual position traversed by the complete handle trajectory and the fixed brush width. A boundary blending zone is then formed along the inner edge of the write mask. The weights of the generated image are determined based on the distance from each pixel position within the boundary blending zone to the write mask boundary. The regenerated handle local image is then weighted and synthesized with the cup and table images saved before the start of the contact, according to the generated image weights and the corresponding original canvas image weights, to form the local image to be written. After boundary blending, the connection between the local image to be written and adjacent positions of the original cup and table images is checked according to the edge checking criteria determined in the image engineering. Once the edge check passes, only the local image to be written within the write mask range corresponding to the complete handle trajectory is written to the current canvas image. The image positions used as outer references are read and the original canvas pixels are maintained. The left side of the cup, the area inside the cup that is not included in the write mask, and the desktop background far from the cup are not written to.

[0109] After writing is complete, the current canvas image simultaneously contains the original cup body, the desktop background, and the cup handle created in this writing process. This image serves as the basis for the new canvas to be read at the start of the next stylus touch. Temporary local images generated during this touch are cleared. When the user subsequently adds new strokes near the cup handle, the next touch process is re-executed using the current canvas image after the cup handle has been written.

[0110] The above content is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A real-time canvas image generation method based on brush stroke trajectory and text conditions, characterized in that, Includes the following steps: S1. Read the current canvas image, canvas coordinate relationship and text conditions, save the canvas image before the start of contact when the input device generates a contact start signal, and continuously obtain the pen stroke sampling position during the contact period. S2. Convert the brush stroke sampling position to canvas coordinates, determine the corresponding local image region based on the position traversed by the new brush stroke trajectory and the brush width, and read the local image region and its surrounding image from the canvas image saved before the contact begins; and form a trajectory overlay mask based on the actual position traversed by the new brush stroke trajectory and the brush width. The local image region and its surrounding image together constitute the image input range for local image generation. The trajectory overlay mask is used to limit the positions within the local image region where generation changes are allowed. S3. During the contact holding period, the corresponding local image area and the surrounding image in the canvas image saved before the contact started are used as image input. The current pen stroke trajectory is converted into trajectory conditions, and the text conditions are encoded into text semantic conditions. A local image is generated based on the image input, trajectory conditions and text semantic conditions through a preset local condition image generation model. The local image is temporarily stored and displayed in the corresponding local image area without replacing the canvas image saved before the contact started. S4. When the brush stroke trajectory continues to increase, the local image area is adjusted according to the newly added brush stroke trajectory, and the corresponding image is read again from the canvas image saved before the contact begins. The local image is generated again according to the added brush stroke trajectory and text conditions. The previously generated and temporarily stored local image is not used as the image basis for the next local image generation. S5. After the input device generates a contact end signal, the corresponding local image area is determined based on the complete pen stroke trajectory formed by this contact. The corresponding image is read from the canvas image saved before the contact started, and the local image is regenerated by combining the complete pen stroke trajectory and text conditions. A writing mask is formed based on the complete brush stroke trajectory. Within the boundary fusion zone corresponding to the boundary of the writing mask, the newly generated local image is weighted and fused with the canvas image saved before the start of the contact. After passing the edge check, the fused local image is written into the current canvas image.

2. The real-time canvas image generation method based on pen stroke trajectory and text conditions according to claim 1, characterized in that, S1 includes: The text condition uses text data that has been submitted when the input device generates a contact start signal; New text submitted after contact has begun is used when the next contact start signal arrives. Upon receiving the touch start signal, first save the current canvas image that has not been changed by this touch, and then obtain the touch sampling position of this touch. The sampling position that is valid in terms of canvas coordinates, is within the valid pixel range of the current canvas image, and has a valid contact mark is determined as the valid brush stroke sampling position.

3. The real-time canvas image generation method based on pen stroke trajectory and text conditions according to claim 1, characterized in that, S2 include: New stroke trajectories are formed for adjacent valid stroke sampling positions that are not determined to have sampling gaps; The trajectory coverage area is formed based on the actual canvas position traversed by the newly added brush stroke trajectory and the brush width determined at the start of contact; The horizontal and vertical coordinates of the canvas pixel positions within the trajectory coverage area are read respectively. The left, right, upper, and lower boundaries of the local rectangular image range are determined by the minimum, maximum, minimum, and maximum values ​​of the horizontal and vertical coordinates of the canvas pixel positions respectively. Based on the left, right, upper, and lower boundaries, a minimum bounding rectangle corresponding to the trajectory coverage area is formed, and the minimum bounding rectangle is determined as a local image region; When any of the left, right, top, and bottom boundaries exceeds the effective pixel range of the current canvas image, the boundary is limited to the corresponding effective pixel boundary of the canvas.

4. The real-time canvas image generation method based on brush stroke trajectory and text conditions according to claim 3, characterized in that, S2 also includes: When the newly formed local rectangular image region intersects with an existing local image region and their boundaries are directly connected, the two are regarded as the same local image region. When there is an interval between the newly formed local rectangular image range and the existing local image region that has not been traversed by the brush stroke trajectory, the corresponding local image regions are retained respectively; Read each local image region and its surrounding image from the canvas image saved before the contact begins.

5. The real-time canvas image generation method based on pen stroke trajectory and text conditions according to claim 1, characterized in that, S3 include: The current stroke trajectory is formed by all valid trajectories between the first valid stroke sampling position after the start of this contact and the latest valid stroke sampling position. Each local image generation reads the corresponding local image region and surrounding image from the canvas image saved before the contact begins, and uses the local image region and surrounding image as image input; the trajectory position within the image input range of the current stroke trajectory forms trajectory condition data, and the text conditions obtained at the start of this contact form text condition data; the preset local condition image generation model includes an image encoding unit, a trajectory condition encoding unit, a text encoding unit, a conditional denoising generation unit, and an image decoding unit; the image encoding unit of the preset local condition image generation model converts the image input into latent space image features, the trajectory condition encoding unit converts the trajectory condition data into trajectory condition features with corresponding spatial positional relationships to the latent space image features, and the text encoding unit converts the text conditions into text semantic features; the conditional denoising generation unit forms updated image features based on the latent space image features, trajectory condition features, and text semantic features, and the image decoding unit converts the updated image features into a local image.

6. The real-time canvas image generation method based on pen stroke trajectory and text conditions according to claim 1, characterized in that, S4 include: When a new stroke trajectory is located inside an existing local image region, the boundary of the local image region remains unchanged, and the new stroke trajectory is merged into the current stroke trajectory. When a new stroke trajectory extends beyond an existing local image area, the actual image range traversed by the new stroke trajectory is incorporated into the corresponding local image area. After adjusting the local image area, the adjusted local image area is read again from the canvas image saved before the touch begins, and the local image is generated again based on the increased brush stroke trajectory and text conditions.

7. The real-time canvas image generation method based on brush stroke trajectory and text conditions according to claim 6, characterized in that, S4 also includes: When a local image region exceeds the single input size allowed by the current image project, the trajectory segments between adjacent valid stroke sampling positions are added to the current local image range in the order of sampling the valid stroke sampling positions. After adding the next trajectory segment, the image width of the current local image range is compared with the maximum input image width of the current image project, and the image height is compared with the maximum input image height of the current image project. If there is a comparison result that exceeds the corresponding maximum input value, the valid stroke sampling position at the end of the current local image range is used as the starting position of the next local image range, and the next trajectory segment is added to the next local image range. When a new stroke trajectory enters a local image range, a new local image is generated for that local image range and the adjacent local image ranges that are directly connected to its boundary. Local image areas that were not traversed by the newly added brush strokes and have no boundary connections will retain their existing temporary display content.

8. The real-time canvas image generation method based on pen stroke trajectory and text conditions according to claim 1, characterized in that, S5 include: After the contact ends, the first valid pen stroke sampling position after the start of this contact is taken as the starting point of the complete pen stroke trajectory, and the last valid pen stroke sampling position before the end of the contact is taken as the ending point of the complete pen stroke trajectory. The local image area is redefined based on the actual location traversed by the complete brush stroke trajectory and the brush width used in this contact. The corresponding image is reread from the canvas image saved before the contact begins, and the local image is regenerated based on the complete brush stroke trajectory and text conditions.

9. The real-time canvas image generation method based on brush stroke trajectory and text conditions according to claim 8, characterized in that, S5 also includes: A writing mask is formed based on the actual location traversed by the complete brush stroke trajectory and the brush width used in this contact, and a boundary blending band is formed along the inner side of the writing mask boundary. For the pixel positions within the boundary blending band, the generated image weight is determined based on the distance from the pixel position to the writing mask boundary and the width of the boundary blending band. According to the generated image weight and the original canvas image weight, which is equal to the sum of the generated image weight, the pixels of the newly generated local image are weighted and blended with the pixels at the same canvas coordinate positions in the canvas image saved before the contact begins, forming the local image to be written. Within the edge inspection band inside the writing mask boundary, inspection positions are determined according to a predetermined inspection interval, and the local image to be written at each inspection position is obtained. The absolute difference between pixels at the same canvas coordinate position in each color channel and the canvas image saved before the contact begins is calculated, and the maximum value among the absolute differences of each color channel is determined as the color channel difference of the corresponding inspection position. The number of inspection positions where the color channel difference exceeds the allowable difference predetermined by the current image project is counted as the number of out-of-tolerance inspection positions. When the number of out-of-tolerance inspection positions is not greater than the allowable number predetermined by the current image project, the local image to be written is written to the corresponding local image region. When the number of out-of-tolerance inspection positions is greater than the allowable number, the reading range of the outer image is expanded, the canvas image saved before the contact begins is reread and the local image is regenerated, and boundary fusion and edge checking are re-executed.