System and method for recording brush movement for traditional painting process

Abstract

The present invention relates to a system for recording brush movements in traditional painting work and a method thereof. A brush movement recording system for traditional painting work according to the present invention is characterized by comprising a preprocessing unit that receives brush tip data and a canvas image and extracts the actual brush tip position, a matching unit that corrects the initial brush tip data to the actual brush tip position, and a correction unit that rearranges the corrected brush tip data according to a time series order.

Description

Technology Field

[0001] The present invention relates to a system for recording brush movements in traditional painting work and a method thereof. Background Technology

[0003] Technology is being proposed to digitize traditional intangible culture and store and provide it as audiovisual materials.

[0004] However, according to conventional technology, since sensors must be attached directly to the brush or the worker's hand to record the movements of the brush work performed by an expert, limitations or distortions occur in the expert's work movements, and due to the bending characteristics resulting from the incomplete elasticity of the brush tool, there is a problem in that it is difficult to accurately capture the point where the tip of the brush meets the ground during the work process. The problem to be solved

[0006] The present invention is proposed to solve the aforementioned problems and aims to provide a system and method for recording brush movements in traditional painting work that enables the preservation of traditional intangible cultural heritage by precisely recording brush movements during the traditional painting process. means of solving the problem

[0008] The brush movement recording system for traditional painting work according to the present invention is characterized by comprising a preprocessing unit that receives brush tip data and a canvas image, unifies the coordinate system, and extracts the actual brush tip position; a matching unit that corrects the initial brush tip data to the actual brush tip position; and a correction unit that rearranges the corrected brush tip data according to a time series order.

[0009] The method for recording brush movements in traditional painting work according to the present invention is characterized by comprising: (a) a preprocessing step of converting brush tip data acquired using a multimodal sensor and a canvas image into an image to which a matching technique can be applied; (b) a matching step of correcting the initial brush tip data to the actual brush tip position; and (c) a correction step of rearranging the corrected brush tip data in a chronological order. Effects of the invention

[0011] According to an embodiment of the present invention, by precisely recording the rotation and position of the brush during the traditional painting process, it is possible to achieve the effect of detailed recording and preservation of the expert's work process.

[0012] According to conventional technology, only simple recordings such as video, music, and photographs were possible regarding the disappearing traditional painting process; however, according to an embodiment of the present invention, precise and detailed recording and preservation of the traditional painting process are possible, and it is possible to utilize it as useful material for the archetype and transmission of traditional painting techniques.

[0013] It helps to increase the general public's understanding of traditional painting processes, provides useful information for 3D reproduction of traditional painting work processes, and is also helpful for exhibitions of intangible cultural heritage.

[0014] The effects of the present invention are not limited to those mentioned above, and other unmentioned effects will be clearly understood by those skilled in the art from the description below. Brief explanation of the drawing

[0016] FIG. 1 illustrates a brush movement recording system for traditional painting work according to an embodiment of the present invention. FIG. 2 illustrates a design drawing of a brush marker according to an embodiment of the present invention and an example of mounting the marker. FIG. 3 illustrates a visualization image of brush movement data according to an embodiment of the present invention. FIGS. 4 and 5 illustrate a method for recording brush movements in traditional painting work according to an embodiment of the present invention. FIG. 6 illustrates a checkerboard shooting process for coordinate system transformation according to an embodiment of the present invention. Figure 7 is an enlarged image of the preprocessing result according to an embodiment of the present invention. FIG. 8 illustrates a result image of local matching before and after according to an embodiment of the present invention. FIG. 9 illustrates the before and after of the application of a correction step according to an embodiment of the present invention. Specific details for implementing the invention

[0017] The aforementioned objectives of the present invention, as well as other objectives, advantages, and features, and the methods for achieving them, will become clear from the embodiments described in detail below together with the accompanying drawings.

[0018] However, the present invention is not limited to the embodiments disclosed below but can be implemented in various different forms, and the following embodiments are provided merely to easily inform those skilled in the art of the purpose, structure, and effects of the invention, and the scope of the rights of the present invention is defined by the description in the claims.

[0019] Meanwhile, the terms used in this specification are for describing the embodiments and are not intended to limit the invention. In this specification, the singular form includes the plural form unless specifically stated otherwise in the text. As used in this specification, "comprises" and / or "comprising" do not exclude the presence or addition of one or more other components, steps, actions, and / or elements to the mentioned components, steps, actions, and / or elements.

[0021] In the following, to aid the understanding of those skilled in the art, the background of the proposed invention will be described first, followed by a description of embodiments of the invention.

[0022] Traditional intangible culture refers to intangible cultural heritage that lacks physical form, encompassing traditions or living expressions such as social customs or skills transmitted from generation to generation.

[0023] The subject of traditional intangible culture is people, and accordingly, traditional intangible culture is recreated in a constantly changing social environment and promotes the diversity and creativity of human culture.

[0024] However, as globalization intensifies cultural imbalances and reduces cultural diversity, traditional intangible culture is being replaced by international culture.

[0025] Furthermore, many traditional intangible cultural heritages are at risk of disappearing due to declining social demand and the disruption of transmission lines.

[0026] Therefore, it is necessary to preserve traditional intangible cultural heritage, which forms the foundation of cultural diversity and national identity.

[0027] Accordingly, many countries and organizations are carrying out numerous activities to preserve traditional intangible cultural heritage.

[0028] UNESCO began operating the 'Masterpieces of the Oral and Intangible Heritage of Humanity' program in 2001, and in 2003, it signed the Convention for the Preservation of Intangible Heritage with the participation of more than 30 countries, thereby encouraging many countries to take an interest in the preservation of traditional intangible cultural heritage.

[0029] In addition, traditional intangible culture, such as traditional music and traditional dance, is digitized and stored and provided as audiovisual materials on multimedia archive sites.

[0030] In addition, the European Union is carrying out the i-treasure project, which records traditional music and dance using ICT technology. Through detailed analysis of traditional intangible cultural heritage, it conducts precise documentation of the creative process and provides this data to enhance understanding of the archetypes and transmission of traditional skills.

[0031] In this way, process acquisition technologies, such as attaching sensors to the body or hands, are being developed for traditional music, traditional dance, and traditional pottery according to the characteristics of each discipline, and sensor utilization technologies tailored to the specific characteristics of the disciplines to be recorded are also being developed.

[0032] Examples include a voice recording sensor device mounted on the face and a wrist sensor for recording pottery.

[0033] However, there are significant difficulties in documenting the detailed production processes for traditional painting genres (Buddhist paintings, Dancheong).

[0034] To move beyond simple video recording and capture the detailed movements of a professional's brushwork, it is necessary to directly attach sensors; however, attaching sensors directly to the brush or hand results in significant discrepancies from the original process due to limitations or distortions in the professional's movements.

[0035] In addition, because the characteristics of the tool used in traditional painting (brush) are imperfect elastic bodies, unlike pencils or pens, it is difficult to capture the exact point where the brush tip, which deforms depending on the work, meets the paper surface.

[0036] The present invention is proposed to solve the aforementioned problems and proposes a system and method capable of recording the process of drawing by tracking the movement of a brush, which is a painting tool, through a multimodal sensor, and precisely recording the actual brush tip movement (data regarding position and rotation) by combining the drawn picture and the brush motion.

[0037] According to an embodiment of the present invention, it is possible to record precise brush movements by using a multimodal sensor to capture the brush movement and the process of drawing on a canvas, and by correcting the position of the brush tip based on the image drawn on the canvas.

[0039] FIG. 1 illustrates a brush movement recording system for traditional painting work according to an embodiment of the present invention.

[0040] The brush movement recording system for traditional painting work according to the present invention is characterized by comprising a preprocessing unit (151) that receives brush tip data and a canvas image, unifies the coordinate system, and extracts the actual brush tip position, a matching unit (152) that corrects the initial brush tip data to the actual brush tip position, and a correction unit (153) that rearranges the corrected brush tip data according to the time series order.

[0041] The preprocessing unit (151) converts the brush tip data and the canvas image obtained by the multimodal sensor into the same coordinate system and extracts the actual brush tip image from the canvas image.

[0042] The preprocessing unit (151) uses a checkerboard with a pattern to convert the brush tracking sensor coordinate system into the canvas shooting sensor coordinate system to generate an initial brush tip image.

[0043] The preprocessing unit (151) extracts a line segment using the distribution of brightness values ​​in the canvas image and extracts the center line of the extracted line segment.

[0044] The matching unit (152) matches the initial brush tip data in stroke units, and then matches it in pixel units using a cost function generated through a shape descriptor.

[0045] The correction unit (153) performs time series information rearrangement through a Hamilton path based on matching pairs.

[0046] FIGS. 4 and 5 illustrate a method for recording brush movements in traditional painting work according to an embodiment of the present invention.

[0047] A method for recording brush movements in traditional painting work according to an embodiment of the present invention is characterized by comprising a preprocessing step (S210) for converting brush tip data acquired using a multimodal sensor and a canvas image into an image to which a matching technique can be applied, a matching step (S220) for correcting initial brush tip data to the actual brush tip position, and a correction step (S230) for rearranging the corrected brush tip data in a chronological order.

[0048] Step S210 converts the brush tracking sensor coordinate system to the canvas shooting sensor coordinate system.

[0049] Step S210 takes multiple shots of the checkerboard and performs calibration using the relative positional relationships.

[0050] Step S220 includes a local matching step that matches an initial brush tip image on a stroke-by-stroke basis and a local matching step that matches the matched results on a pixel-by-pixel basis.

[0051] The local matching step generates a feature descriptor for the feature information at the pixel and generates a feature context for each pixel.

[0052] Step S220 calculates a cost function representing the difference between the shape descriptors of the locally matched brush tip image and the actual brush tip image, and extracts matching pairs using the calculated cost function.

[0053] Step S230 performs correction based on matching pairs, but rearranges extracted pixels with mismatched time-series information through a Hamiltonian path.

[0054] The recording unit (140) records brush movements a preset number of times (e.g., 120 times per second) through a multimodal sensor and films the process of drawing on the canvas.

[0055] The multimodal sensor consists of a brush tracking sensor and a canvas capturing sensor.

[0056] The brush tracking sensor consists of a marker-based motion tracking sensor that recognizes the marker attached to the brush to record brush rotation and calculates brush tip data, which is the brush position, based on the previously measured lengths of the bristles and handle.

[0057] The canvas shooting sensor consists of two 4K cameras, which are used to capture the process of painting on the canvas and the resulting images.

[0058] One camera films vertically above the canvas, and the other films from a diagonal position to capture the parts obscured by the operator from the camera vertically above the canvas.

[0059] FIG. 2 illustrates a design drawing of a brush marker according to an embodiment of the present invention and an example of mounting the marker.

[0060] Here, the weight of the marker mounting part is a preset weight (e.g., less than 10g) to minimize interference with the operator's work, and is attached to the back end of the brush.

[0061] FIG. 3 illustrates a visualization image of brush movement data according to an embodiment of the present invention.

[0062] Figure 3 visualizes the pivot point and brush rotation measured through the length of the bristles and handle.

[0063] handle_len is the handle length, hair_len is the bristle length, pivot point is the brush tip, and rotation is the brush rotation.

[0064] Since brush bristles are imperfect elastic bodies with bending properties, brush tip data measured through the length of the bristles and handle differs from the actual brush tip, and in video footage of the canvas, the actual brush tip and the color of the paint are identical, making differentiation impossible.

[0065] Accordingly, according to an embodiment of the present invention, the measured brush tip position is corrected based on the canvas image generated by the actual brush tip.

[0066] According to an embodiment of the present invention, a brush movement and a canvas image are captured, a preprocessing process is performed to convert the recorded brush movement and canvas image so that a matching technique can be applied, initial brush tip data is matched to actual brush tip data, and a correction is performed to rearrange the brush tip data into a time series.

[0067] Referring to FIG. 1, the brush tip correction unit (150) is composed of a preprocessing unit (151), a matching unit (152), and a correction unit (153).

[0068] The preprocessing unit (151) converts the brush tip data and canvas image received from the recording unit (140) into the same coordinate system through image analysis (S212), and extracts the initial brush tip position from the canvas image (S213).

[0069] The matching unit (152) corrects the initial brush tip data to the actual brush tip position extracted earlier.

[0070] The correction unit (153) generates correction data by rearranging the corrected brush tip data in chronological order.

[0071] The preprocessing unit (151) receives brush tip data and a canvas image from the recording unit (140) and converts them into an image to which a matching technique can be applied.

[0072] The preprocessing unit (151) converts the brush tip data into a canvas image coordinate system for image-based analysis (S212).

[0073] According to an embodiment of the present invention, a calibration process is performed to convert the brush tracking sensor coordinate system into the canvas shooting sensor coordinate system.

[0074] In the calibration according to an embodiment of the present invention, since the brush tracking sensor and the canvas shooting sensor are fixed, the checkerboard with a pattern is photographed multiple times and the calibration is performed through the relative positional relationship.

[0075] FIG. 6 illustrates a checkerboard shooting process for coordinate system transformation according to an embodiment of the present invention.

[0076] Take multiple shots of an infrared marker and a checkerboard with a circular pattern, changing the direction and position.

[0077] Next, a transformation matrix is ​​calculated using the relative positional relationships of the checkerboard captured multiple times, and this is used to transform the brush tracking sensor 3D coordinate system into the canvas capturing sensor 3D coordinate system.

[0078] Next, the canvas capturing sensor 3D coordinate system is converted to the canvas capturing sensor 2D coordinate system, and then the canvas image area is specified to generate an initial brush tip image (S213).

[0079] Figure 7 is an enlarged image of the preprocessing result according to an embodiment of the present invention.

[0080] Figure 7 (a) is the initial brush tip image, (b) is the canvas image, (c) is the extracted line segment image, and (d) is the thinning result image.

[0081] The preprocessing unit (151) performs the following process to extract actual brush tip data from the canvas image.

[0082] First, to extract line segments generated by the actual brush, line segments are extracted from the canvas image using the k-means method (S215).

[0083] The K-means method is a technique that groups images into k clusters based on the distribution of brightness values.

[0084] Through this, a line segment with a black brightness value is extracted. To obtain the brush tip position, the center line of the line segment is extracted from the extracted line segment through a thinning process (S216).

[0085] Here, thinning refers to the process of extracting a center line with a thickness of 1 pixel.

[0086] The matching section (152) is composed of a local matching section and a local matching section, and matches the initial brush tip image to the actual brush tip image.

[0087] The local matching section generally matches the initial brush tip image on a stroke-by-stroke basis, and the local matching section matches the matched results finely on a pixel-by-pixel basis.

[0088] The local matching section divides the initial brush tip image into stroke units and matches each stroke unit to the actual brush tip image without overlap.

[0089] Here, a stroke refers to a line drawn in one continuous motion without lifting the brush when drawing.

[0090] According to an embodiment of the present invention, the initial brush tip image is divided into stroke units using time information obtained from the recording unit (151) (S221).

[0091] Each divided stroke is moved along the x and y axes and matched without overlap to the location with the most points matching the actual brush tip image (S222).

[0092] FIG. 8 illustrates a result image of local matching before and after according to an embodiment of the present invention.

[0093] The white line in Fig. 8 is the actual brush tip image, and the red line is the brush tip motion data.

[0094] The local matching section matches the locally matched brush tip image to the actual brush tip image pixel by pixel through the steps of shape descriptor generation (S223), cost function calculation (S224), and matching pair extraction (S225).

[0095] A feature descriptor refers to representing feature information at each pixel, and a feature descriptor is generated at each pixel of the two input images.

[0096] According to an embodiment of the present invention, a shape context among shape descriptors is used, and after converting an image into a log-polar coordinate system, the shape context of each pixel is generated through a histogram distribution of distance and angle centered on each pixel.

[0097] Next, a cost function (C) representing the difference between the shape descriptors of the locally matched brush tip image and the actual brush tip image ij Calculate the cost function (C ij ) is equal to the following [Mathematical Formula 1].

[0098] [Mathematical Formula 1]

[0099]

[0100] Here, i refers to the pixel of the locally matched brush tip image, and j refers to the actual brush tip image pixel. h i represents the shape descriptor of an i-pixel, and h j represents the shape descriptor of j pixels.

[0101] Finally, C ijrepresents the feature descriptor cost function between pixel i and pixel j, and K represents the dimensionality of the feature descriptor.

[0102] The pixels of the locally matched brush tip image are matched to the actual brush tip image pixels that have the minimum cost function using the calculated cost function.

[0103] FIG. 9 illustrates the before and after of the application of a correction step according to an embodiment of the present invention.

[0104] The correction unit (153) performs a correction operation based on the matching pair data extracted from the matching unit (152).

[0105] The matching pairs extracted from the matching section (152) were matched without considering the time information of the traditional painting process, and thus contain a problem of time series order as shown in (a) of Fig. 9.

[0106] To resolve this, the correction unit (153) rearranges the extracted pixels that do not match the time series information through the Hamilton path as shown in (b) of FIG. 9.

[0107] A Hamiltonian path is a path that visits every vertex exactly once with minimum cost.

[0108] This aligns the flow of the strokes in a straight line and converts the matched pixels back into the brush tracking sensor coordinate system.

[0109] As a result, precise brush movement data, including the artisan's brush tip and brush rotation during the traditional painting process, can be recorded.

[0111] Meanwhile, the method for recording brush movements of traditional painting work according to an embodiment of the present invention may be implemented in a computer system or recorded on a recording medium. The computer system may include at least one processor, memory, a user input device, a data communication bus, a user output device, and a storage. Each of the aforementioned components communicates data through the data communication bus.

[0112] The computer system may further include network interfaces coupled to the network. The processor may be a central processing unit (CPU) or a semiconductor device that processes instructions stored in memory and / or storage.

[0113] Memory and storage may include various forms of volatile or non-volatile storage media. For example, memory may include ROM and RAM.

[0114] Accordingly, the method for recording brush movements of a traditional painting work according to an embodiment of the present invention can be implemented as a method executable on a computer. When the method for recording brush movements of a traditional painting work according to an embodiment of the present invention is performed on a computer device, computer-readable instructions can perform the recording method according to the present invention.

[0115] Meanwhile, the method for recording brush movements of traditional painting work according to the present invention described above can be implemented as computer-readable code on a computer-readable recording medium. Computer-readable recording media include all types of recording media in which data that can be decoded by a computer system is stored. For example, ROM (Read Only Memory), RAM (Random Access Memory), magnetic tape, magnetic disk, flash memory, optical data storage device, etc. In addition, the computer-readable recording medium can be distributed to computer systems connected via a computer network and stored and executed as code that can be read in a distributed manner.

[0117] The embodiments of the present invention have been described above. Those skilled in the art will understand that the present invention may be implemented in modified forms without departing from the essential characteristics of the invention. Therefore, the disclosed embodiments should be considered in an illustrative rather than a restrictive sense. The scope of the invention is defined by the claims, not by the foregoing description, and all variations within the scope of the claims should be interpreted as being included in the invention.

Claims

Claim 1 A preprocessing unit that receives brush tip data and a canvas image, unifies the coordinate system, and extracts the actual brush tip position; a matching unit that corrects the initial brush tip data to the actual brush tip position; and a correction unit that rearranges the corrected brush tip data according to a time series order, wherein the matching unit locally matches the initial brush tip image, which is the initial brush tip data, to the actual brush tip image on a stroke-by-stroke basis, and performs local matching on a pixel-by-pixel basis using the locally matched result, wherein the local matching is performed by matching to the actual brush tip image on a pixel-by-pixel basis through shape descriptor generation, cost function calculation, and matching pair extraction, and the matching unit uses the cost function of the following mathematical formula, [Mathematical Formula] - i is a pixel of the locally matched brush tip image, j is a pixel of the actual brush tip image, and h i is the shape descriptor of pixel i, and h j represents the shape descriptor of j pixels, and C ij is a feature descriptor cost function between pixel i and pixel j, and K is the dimensionality of the feature descriptor—a computer device that performs a brush movement recording of a traditional painting process, which involves matching pixels of a locally matched brush tip image to actual brush tip image pixels. Claim 2 A computer device for recording brush movements in traditional painting work, wherein, in claim 1, the preprocessing unit converts brush tip data acquired by a multimodal sensor and a canvas image into the same coordinate system and extracts an actual brush tip image from the canvas image. Claim 3 delete Claim 4 A computer device for recording brush movements in traditional painting work, wherein, in paragraph 2, the preprocessing unit converts the brush tracking sensor coordinate system into the canvas shooting sensor coordinate system using a patterned checkerboard to generate an initial brush tip image. Claim 5 A computer device for recording brush movements in traditional painting work, wherein the preprocessing unit extracts line segments using the distribution of brightness values ​​in the canvas image and extracts the center line of the extracted line segments. Claim 6 delete Claim 7 A computer device for recording brush movements of traditional painting work, wherein the correction unit in claim 1 performs time-series information rearrangement through a Hamiltonian path based on matching pairs. Claim 8 A method for recording brush movements in a traditional painting work, performed by a computer device that performs the recording of brush movements in a traditional painting work, comprising: (a) a preprocessing step of converting brush tip data acquired using a multimodal sensor and a canvas image into an image to which a matching technique can be applied; (b) a matching step of correcting initial brush tip data to the actual brush tip position; and (c) a correction step of rearranging the corrected brush tip data in a time-series order, wherein step (b) locally matches the initial brush tip image, which is the initial brush tip data, to the actual brush tip image on a stroke-by-stroke basis, and performs local matching on a pixel-by-pixel basis using the locally matched result, wherein the local matching is performed by matching to the actual brush tip image on a pixel-by-pixel basis through shape descriptor generation, cost function calculation, and matching pair extraction, and using the following [Mathematical Formula], [Mathematical Formula] - i is a pixel of the locally matched brush tip image, j is a pixel of the actual brush tip image, and h i is the shape descriptor of pixel i, and h j represents the shape descriptor of j pixels, and C ij is a feature descriptor cost function between i pixels and j pixels, and K is the dimensionality of the feature descriptor—a method for recording brush movements in traditional painting work that matches pixels of locally matched brush tip images to actual brush tip image pixels. Claim 9 In claim 8, the above step (a) is a method for recording brush movements in traditional painting work, wherein the brush tracking sensor coordinate system is converted to a canvas shooting sensor coordinate system. Claim 10 A method for recording brush movements in traditional painting work, wherein in claim 9, step (a) involves taking multiple photos of a checkerboard and performing calibration using relative positional relationships. Claim 11 delete Claim 12 delete Claim 13 delete Claim 14 A method for recording brush movements in traditional painting work, wherein in claim 8, the above step (c) performs correction based on the matching pairs, and rearranges information among the extracted pixels that does not match the time series information through a Hamilton path.

Images