RSTL display program and device

Abstract

To provide a program and a device which make it easy to visually check the states of RSTLs and wrinkle lines.SOLUTION: A program which displays reference line data of reference lines including RSTLs and / or wrinkle lines overlaid on a patient image 12 causes a processor of a computer to execute a process of reading the patient image 12, a process of inputting the patient image 12 into a learned model and obtaining a reference line image corresponding to the patient image, and a process of displaying the patient image 12 or an illustration of the patient image 12 with the obtained reference line image overlaid thereon.SELECTED DRAWING: Figure 4

Description

[Technical Field]

[0001] The present invention relates to a program and an apparatus for overlaying RSTL (Relaxed Skin Tension Line) on face data and displaying it. [Background technology]

[0002] Local flap surgery is a well-known plastic surgery technique. Local flap surgery is used to close skin defects, such as after tumor resection. The defect is repaired by making an incision in the skin adjacent to the defect and then pulling and shifting the skin to cover the defect. Local flaps are primarily used to repair relatively large defects after removal of facial moles or tumors.

[0003] When creating a local skin flap, it is important to design the incision line (skin flap design) to minimize deformation due to skin movement and suturing, that is, to make postoperative scars as inconspicuous as possible. Kawada Kyohei et al. have already put into practical use application software that can be downloaded and used on smartphones as a simple tool to support skin flap design (see, for example, Non-Patent Document 1 below). [Prior art documents] [Non-patent literature]

[0004] [Non-Patent Document 1] Kyohei Kawada et al. "Development of the Skin Flap App 'Muze'" Journal of the Japanese Society of Plastic and Reconstructive Surgery, Vol. 44, No. 2, 2024, p. 60-67 Summary of the Invention [Problem to be solved by the invention]

[0005] It is desirable to place the incision line along the wrinkle lines or the RSTL (Relaxed Skin Tension Line). The RSTL is a line that indicates the direction of maximum tension when the facial muscles are relaxed and static. Wrinkle lines generally coincide with the RSTL and are visible on the skin surface, but not all RSTLs necessarily appear visible on the skin surface. Furthermore, RSTLs do not necessarily coincide with wrinkle lines. In the past, in preoperative conferences, an attempt was made to consider the flap design by printing out an image of the patient's face and drawing the RSTL on it with a pen or other tool. However, there is considerable individual variation, especially in the face, making it difficult to determine the RSTL. [Means for solving the problem]

[0006] In order to achieve the above object, the program disclosed below is a program for overlaying and displaying reference line data of reference lines including at least one of RSTL and wrinkle lines on face data, A process of reading face data; A process of inputting face data into a trained model and obtaining reference line data corresponding to the face data; a process of overlaying the obtained reference line data on the face data and displaying the same; The program includes instructions for causing a computer processor to execute the program.

[0007] The device disclosed below is a device for overlaying and displaying reference line data of a reference line including RSTL on face data, A process of reading face data; A process of inputting face data into a trained model and obtaining reference line data corresponding to the face data; a process of overlaying the obtained reference line data on the face data and displaying the same; The device includes a processor that executes [Effects of the Invention]

[0008] According to the present invention, it is possible to provide a program and an apparatus for overlaying reference line data on face data and displaying the same. [Brief explanation of the drawings]

[0009] [Figure 1] FIG. 1 is a schematic diagram showing an example of an initial screen displayed on a display of a smartphone in one embodiment of the present invention. [Figure 2] FIG. 2 is a schematic diagram showing a screen for selecting a patient image. [Figure 3] FIG. 3 is a schematic diagram showing the state of the screen on which a patient image is loaded. [Figure 4] FIG. 4 is a schematic diagram showing a state in which reference lines are overlaid on a patient image. [Figure 5] FIG. 5 is a schematic diagram showing the selected skin flap pattern superimposed on the patient image. [Figure 6] FIG. 6 is a schematic diagram showing the state after the skin flap pattern has been inverted, reduced, and rotated from the screen of FIG. [Figure 7] FIG. 7 is a schematic diagram showing a state in which the screen of FIG. 6 has been switched to a screen displaying a predicted suture pattern "after surgery." [Figure 8] FIG. 8 is a block diagram showing the functional configuration of an application program according to one embodiment of the present invention. [Figure 9A] FIG. 9A is an example of the correspondence between a flap pattern and its corresponding predicted suture pattern. [Figure 9B] FIG. 9B is an example of the correspondence between a flap pattern and its corresponding predicted suture pattern. [Figure 10] FIG. 10 is a schematic diagram showing the process of creating a trained model. [Figure 11] FIG. 11 is a block diagram showing a functional configuration of an application program according to another embodiment of the present invention.

[0010] An embodiment of the present disclosure will be described below with reference to the drawings. The present disclosure is not limited to the following embodiment, and appropriate design modifications can be made within the scope of the configuration of the present disclosure. In the following description, the same reference numerals are used in common between different drawings for identical parts or parts having similar functions, and repeated description thereof will be omitted. The configurations described in the embodiment and modified examples may be combined or modified as appropriate. To facilitate understanding of the description, the drawings referred to below show simplified or schematic configurations, and some components may be omitted. Furthermore, in the drawings referred to below, actual images are represented by line drawings.

[0011] The program according to this embodiment is an application program (hereinafter abbreviated as "AP") that is downloaded and used on a smartphone, tablet, etc. First, the operation of this AP will be described.

[0012] [AP operation] FIG. 1 shows an example of an initial screen displayed on a smartphone display 10 by an AP downloaded to the smartphone. As shown in FIG. 1 , the right side of the screen displays operation buttons: an inversion button 11a labeled "Invert," a vertical zoom button 11b labeled "Zoom In / Zoom Out" with up and down arrows, a horizontal zoom button 11c labeled "Zoom In / Zoom Out" with left and right arrows, a reset button 11d labeled "Reset," a pre- and post-operative switch button 11e labeled "Pre-Operative" in FIG. 1 , and a patient image selection button 11f labeled "Patient." At the bottom of the screen, flap patterns 13a and 13b are displayed as examples of local flap patterns. In this example, the operation buttons are displayed on the right side and the flap patterns 13a and 13b on the bottom side. However, the operation buttons may be displayed on the bottom side and the flap patterns on the right side; their locations are not particularly limited.

[0013] Touching the patient image selection button 11f on this initial screen allows the user to select an image folder accessible from the smartphone, and the images stored in the selected image folder are displayed on the display 10 as a list (see, for example, FIG. 2). The image folder may be an image folder in the smartphone's built-in memory, or an external image folder accessible from the smartphone (e.g., on the cloud). FIG. 2 shows a state in which an image folder named "Recents," which stores images recently taken with the smartphone, is selected and the images stored therein are displayed as a list. Here, a user (e.g., a doctor) can touch any image to select it. For example, if the image in the lower right corner of the list display in FIG. 2 is selected, the selected image is displayed on the AP screen as a patient image 12, as shown in FIG. 3. Furthermore, when the patient image 12 is displayed on the AP screen, a reference line display button 11g labeled "reference line" appears below the patient image selection button 11f.

[0014] Here, when the reference line display button 11g is touched, reference lines including RSTL and wrinkle lines are overlaid on the patient image 12 as shown in FIG.

[0015] As described above, flap patterns 13a and 13b are displayed at the bottom of the screen. In FIG. 3, flap pattern 13a represents the incision line for Z formation, and flap pattern 13b represents the incision line for Dufourmental formation. The area displaying flap patterns 13a and 13b can be scrolled by dragging horizontally, allowing other flap patterns to be displayed. Note that the flap patterns available in the AP are not limited to the specific examples shown here; any other flap pattern can be pre-registered. Alternatively, the user may be allowed to register additional flap patterns. Note that in the following description, flap patterns 13a and 13b will be collectively referred to as "flap pattern 13."

[0016] The user touches and selects one of the flap patterns 13 displayed at the bottom of the screen, then drags it to overlay it on the patient image 12. Figure 5 shows the state in which the user has selected flap pattern 13b and overlaid it on the patient image 12. In this example, when flap pattern 13b is selected, the other flap patterns are hidden, but the other flap patterns may remain displayed. As shown in Figure 5, reference lines including RSTLs and wrinkle lines are overlaid on the patient image 12, so the user can adjust the position and angle of flap pattern 13b while looking at these reference lines.

[0017] As shown in FIG. 5, the portion of the flap pattern 13 other than the incision line is preferably a transparent image that allows the background (the patient image 12 with the flap pattern 13 superimposed thereon and the reference lines) to be visible. When the portion of the flap pattern 13 other than the incision line is a transparent image, the transparency is arbitrary. The user may also adjust the transparency. A lower transparency has the advantage of making the incision line more visible. Conversely, a higher transparency makes it somewhat more difficult to distinguish the incision line from the background, but has the advantage of making it easier to consider the location where the flap will be formed and the condition of the patient's skin and the reference lines around it. The thickness and transparency of the incision line itself may also be appropriately changed, and are not particularly limited as long as they achieve the effect of the flap pattern 13 described herein.

[0018] Here, the user can vertically enlarge or reduce the flap pattern 13b by touching the vertical enlargement / reduction button 11b and then pinching out or in. Furthermore, the user can horizontally enlarge or reduce the flap pattern 13b by touching the horizontal enlargement / reduction button 11c and then pinching out or in. Vertical and horizontal enlargement / reduction can be combined in any way. For example, by touching the vertical enlargement / reduction button 11b and then enlarging the flap pattern 13b vertically by x times, and then touching the horizontal enlargement / reduction button 11b, the flap pattern 13b that has been enlarged x times vertically can be enlarged or reduced horizontally by any desired factor. In this example, the flap pattern 13b can be enlarged or reduced vertically and horizontally separately, but it may also be enlarged or reduced vertically and horizontally simultaneously.

[0019] In addition, by touching and rotating the flap pattern 13b with two fingers, the flap pattern 13b can be rotated at any angle. Furthermore, by touching the inversion button 11a, the flap pattern 13b can be inverted inside out.

[0020] In this way, by combining the flip button 11a, vertical zoom button 11b, and horizontal zoom button 11c with touch screen operations, the flap pattern 13b can be placed at any position, angle, and zoom ratio on the patient image 12. For example, Fig. 6 shows the display state after the user has slightly reduced and rotated the flap pattern 13b from the state shown in Fig. 5 and placed it there.

[0021] During the above placement operations, the operation buttons displayed on the right and the flap patterns displayed below may be hidden, which improves the visibility of the patient image 12 and expands the placement area for the flap patterns 13, thereby improving convenience.

[0022] Here, when the user touches the pre-operative / post-operative switching button 11e, the wording on the pre-operative / post-operative switching button 11e changes to “Post-operative” as shown in FIG. 7. The AP screen display also switches to a screen showing the post-operative state, and as shown in FIG. 7, a predicted post-operative suture pattern 14b corresponding to the flap pattern 13b arranged on the patient image 12 is displayed. That is, the AP displays on the display what the post-operative suture pattern will look like depending on the position, size, and rotation angle of the flap pattern. When the user touches the pre-operative / post-operative switching button 11e again, the screen returns to the pre-operative state shown in FIG. 6. Therefore, the user can repeatedly adjust the position, zoom ratio, and rotation angle of the flap pattern 13b on the “pre-operative” screen and then check the predicted post-operative suture pattern 14b on the “post-operative” screen until the predicted post-operative suture pattern 14b is in the desired state.

[0023] As shown in FIG. 7 , the screen displaying the “postoperative” predicted suture line pattern 14b also displays a flip button 11a, a vertical zoom button 11b, a horizontal zoom button 11c, and the like. The user can use these buttons to flip and zoom in / out the predicted suture line pattern 14b. Furthermore, the user can rotate the predicted suture line pattern 14b by any angle by touching and rotating the predicted suture line pattern 14b with two fingers. After flipping, zooming in / out, or rotating the predicted suture line pattern 14b in this way, touching the preoperative / postoperative switch button 11e switches the AP screen to the “preoperative” state. At this time, the flap pattern 13b is displayed in a state that reflects the flipping, zooming in / out, and rotation performed on the predicted suture line pattern 14b. In other words, when the predicted suture line pattern 14b is flipped, the flap pattern 13b is also flipped. When the predicted suture line pattern 14b is enlarged or reduced in the vertical and / or horizontal directions, the flap pattern 13b is enlarged or reduced in the vertical and / or horizontal directions by the same magnification. Also, when the predicted suture line pattern 14b is rotated, the flap pattern 13b is also rotated by the same rotation angle.

[0024] Therefore, on the screen displaying the predicted suture line pattern 14b, by performing operations of inverting, enlarging / reducing, and rotating the predicted suture line pattern 14b so that it is in the ideal state, and then touching the pre-operative / post-operative switch button 11e, the ideal skin flap pattern 13b can be obtained. Furthermore, aligning the suture line along the RSTL and wrinkle lines makes it less likely that suture scars will remain after healing. As shown in FIG. 7, reference lines including the RSTL and wrinkle lines are overlaid on the facial image 12. Therefore, by performing operations of inverting, enlarging / reducing, and rotating the predicted suture line pattern 14b so that it is aligned as closely as possible with these reference lines, and then switching to the pre-operative state with the pre-operative / post-operative switch button 11e, the ideal skin flap pattern that matches the state of the patient's RSTL and wrinkle lines can be obtained.

[0025] When the reset button 11d is touched, the screen returns to the state shown in FIG.

[0026] While the method for operating the flap pattern 13 has been described above, similar operations may also be possible for the patient image 12. For example, in the state shown in FIG. 3, the operation buttons 11a to 11g described above are displayed for the flap pattern 13, but corresponding operation buttons may also be provided for the patient image. Also, even if operation buttons are not provided, operations such as pinching in and out and rotation may be possible when the flap pattern 13 is not displayed. When the flap pattern 13 is displayed, a button for switching between operating the flap pattern 13 and operating the patient image 12 may be provided so that both can be operated as appropriate.

[0027] [AP functional specifications] Next, the functional specifications of the AP for realizing the above-described operations are described. FIG. 8 is a block diagram schematically illustrating a configuration realized by the AP. Each functional unit illustrated in the block diagram of FIG. 8 is realized by a processor of a device (computer) such as a smartphone or tablet executing instructions from the AP. As shown in FIG. 8, the functional units realized by the AP include an input processing unit 101 that accepts input from a user via a touch panel, an image acquisition unit 102 that accesses an image folder, a pattern storage unit 103 that stores images of flap patterns and predicted suture line patterns, a reference line generation unit 104 that generates reference line data including RSTL and wrinkle lines using a trained model based on a patient image 12, and an image processing unit 105 that performs various image processing to generate a display image. The trained model will be described in detail later. Here, the pattern storage unit 103 is assumed to exist within a computer such as a smartphone; however, the pattern storage unit 103 may be provided outside the computer as long as it can be referenced by the AP.

[0028] The input processing unit 101 detects user operations on the touch panel. That is, it detects the user's touch positions and number of touches and actions (tap, drag, pinch out / in, rotation, etc.) on GUI objects (buttons, images, etc.) displayed on the display, and determines what instructions are given to which GUI objects.

[0029] As described above, when the patient image selection button 11f is touched, the image acquisition unit 102 accesses an image file outside the AP and performs processing to acquire a patient image.

[0030] The pattern storage unit 103 stores images of flap patterns and predicted suture line patterns in association with each other. Figures 9A and 9B show the association between flap patterns 13a and 13b and predicted suture line patterns 14a and 14b. In local flap surgery, the postoperative suture line pattern is generally determined according to the flap pattern. Such combinations of flap patterns and predicted suture line patterns are created as images by, for example, an experienced physician, and are registered in advance in the pattern storage unit 103 as pairs of flap patterns and predicted suture line patterns.

[0031] In the following description, the predicted suture line patterns 14a and 14b will be collectively referred to as the "predicted suture line pattern 14." As described above, the flap patterns used in the AP are not limited to the combination of flap patterns 13a and 13b. The number of flap patterns may be greater or less than this. When the AP is updated, an image of a newly designed flap pattern and an image of its corresponding predicted suture line pattern may be additionally registered in the pattern storage unit 103. Alternatively, a user may be able to register an image of a flap pattern that he or she has designed himself or herself and an image of its corresponding predicted suture line pattern in the pattern storage unit 103.

[0032] As described above, when the user performs an operation of inverting, enlarging / reducing, or rotating the flap pattern 13 on the patient image 12, the input processing unit 101 acquires the information and sends it to the image processing unit 105. In response to the user's operation on the flap pattern 13, the image processing unit 105 inverts, enlarges / reduces, or rotates the predicted suture line pattern 14 corresponding to the flap pattern 13 selected as the operation target, and displays it superimposed on the patient image 12 as the predicted post-operative suture line pattern 14.

[0033] For example, when an instruction to invert the flap pattern 13 is input by touching the invert button 11a, the image processing unit 105 receives the instruction from the input processing unit 101 and inverts the corresponding predicted suture line pattern 14. If the flap pattern 13 is enlarged or reduced vertically and / or horizontally by a pinch-out or pinch-in operation after touching the vertical enlargement / reduction button 11b and / or horizontal enlargement / reduction button 11c, the input processing unit 101 acquires the vertical enlargement / reduction ratio and / or the horizontal enlargement / reduction ratio and sends it to the image processing unit 105. The image processing unit 105 enlarges or reduces the image of the corresponding predicted suture line pattern 14 vertically and / or horizontally by the same magnification as the vertical enlargement / reduction ratio and / or the horizontal enlargement / reduction ratio sent from the input processing unit 101. Furthermore, when the flap pattern 13 is rotated by a user operation, the image processing unit 105 rotates the corresponding predicted suture line pattern 14 by the same angle as the rotation angle sent from the input processing unit.

[0034] Furthermore, when an instruction to invert the predicted suture line pattern 14 is input by touching the invert button 11a, the image processing unit 105 receives the instruction from the input processing unit 101 and inverts the corresponding flap pattern 13 inside out. If the predicted suture line pattern 14 is enlarged or reduced vertically and / or horizontally by a pinch-out or pinch-in operation after touching the vertical enlargement / reduction button 11b and / or horizontal enlargement / reduction button 11c, the input processing unit 101 acquires the vertical enlargement / reduction ratio and / or the horizontal enlargement / reduction ratio and sends it to the image processing unit 105. The image processing unit 105 enlarges or reduces the image of the corresponding flap pattern 13 vertically and / or horizontally by the same magnification as the vertical enlargement / reduction ratio and / or the horizontal enlargement / reduction ratio sent from the input processing unit 101. Furthermore, when the predicted suture line pattern 14 is rotated by a user operation, the image processing unit 105 rotates the corresponding flap pattern 13 by the same angle as the rotation angle sent from the input processing unit 101 .

[0035] Through the above-described processing by the input processing unit 101 and the image processing unit 105, the user can change the corresponding predicted suture line pattern 14 on the patient image 12 as the user inverts, enlarges, reduces, or rotates the skin flap pattern 13. Furthermore, the user can change the corresponding skin flap pattern 13 on the patient image 12 as the user inverts, enlarges, reduces, or rotates the predicted suture line pattern 14. This allows the user to easily design the skin flap pattern 13 appropriately so as to obtain an ideal predicted suture line pattern 14. The AP of this embodiment is also useful for inexperienced physicians to simulate and learn the correlation between the skin flap pattern 13 and the predicted suture line pattern 14. Furthermore, the AP of this embodiment is very useful when a physician explains the surgery to a patient undergoing surgery or to physicians from other departments, as it allows the physician to visually show the postoperative state.

[0036] In the above embodiment, the flap pattern 13 and the predicted suture line pattern 14 are treated as two-dimensional images, and two-dimensional image processing is performed to invert, enlarge, reduce, or rotate the predicted suture line pattern 14 / flap pattern 13 in response to a user operation to invert, enlarge, reduce, or rotate the flap pattern 13 / predicted suture line pattern 14. However, for more accurate flap design, it is also desirable to perform three-dimensional image processing on the predicted suture line pattern 14 / flap pattern 13 in accordance with the three-dimensional shape of the area where the flap pattern 13 / predicted suture line pattern 14 is located on the patient image 12. By performing three-dimensional image processing, it becomes possible to determine, for example, where a "dog ear" (a protuberance shaped like a dog ear) will be formed. In this case, a patient image 12 captured as a three-dimensional image is used, or a patient image 12 captured as a two-dimensional image is converted into a three-dimensional image and used. Note that such three-dimensional image processing can be achieved using known polygon technology or computer simulation technology, and the specific method is not limited.

[0037] [Creating a trained model] Here, creation of a trained model used in the reference line generation unit 104 will be described with reference to Fig. 10. As shown in Fig. 10, the trained model 400 used in the reference line generation unit 104 is created by having the learner 300 learn learning images.

[0038] The trained model 400 is obtained by preparing a large number of training images, inputting them into the learning device 300, and training the learning device 300 using machine learning or deep learning using a neural network. Training images can be full-face images (images of the entire face as seen from the front) or partial images. Partial images can be images of the face in profile, images from an oblique front view, or images of facial features (e.g., only the forehead, only the cheeks, etc.). To be used as training images, the original images may be subjected to various processes such as resolution adjustment, contrast adjustment, and noise removal. The learning algorithm used by the learning device 300 is not particularly limited. Training may be performed using training data or without training data. When training data is used, for example, RSTL or wrinkle line images handwritten by a specialist on appropriately cleansed training images can be used as training data. When a patient image 12 is input, the trained model 400 generated by such a training process outputs an image of reference lines including RSTL and wrinkle lines according to the input patient image 12.

[0039] When creating training data, for example, a training image is displayed on a tablet or computer screen, and a specialist handwrites RSTLs and wrinkle lines on the training image using an input device such as a touch pen or mouse. At this time, the specialist selects the appropriate wrinkle lines required for the flap design from the wrinkle lines visible on the image and traces them with a touch pen or other device. To facilitate tracing the wrinkle lines, image processing may be performed on the training image in advance to enhance the wrinkle lines. Since RSTLs are not necessarily present on the training image, the specialist writes the RSTLs required for the flap design with a touch pen or other device based on their experience and knowledge. Here, when handwriting RSTLs and wrinkle lines, it is not necessary to distinguish between RSTLs and wrinkle lines. The image or coordinate information of the handwritten pattern is stored as training data and used for training.

[0040] In addition, as long as the learning device 300 can identify an image or coordinate information of a handwritten pattern of RSTL or wrinkle lines, the image that can serve as training data may be in a digital format in which reference lines are added to the training image data using an input device, or in an analog format in which the training image data is printed, reference lines are added with a brush or pen, etc., and then scanned with an image input device such as a scanner or camera, and is not particularly limited.

[0041] The trained model 400 may be included in the reference line generating unit 104. Alternatively, the trained model 400 may be stored in an external server or the like. In the latter case, the reference line generating unit 104 accesses the trained model 400 stored in the external server or the like via a network, transmits the input patient image 12 to the trained model 400, and acquires the output (reference line) from the trained model 400. Note that even in the reference line image, it is not necessary to display the RSTL and wrinkle lines in a manner that allows the types to be distinguished.

[0042] In the above description, the reference lines generated by the reference line generation unit 104 are output as an image, and the image processing unit 105 overlays the reference line image on the patient image 12 and outputs it to the monitor. In other words, the patient image 12 output to the monitor is in the form of an original image (photograph). However, the output to the monitor may be an overlay image of a line drawing (illustration-like) obtained from the original image of the patient image 12 and the reference line image. In this case, the patient image may be lined by the display processing unit 105, or the trained model 400 may be created so that the reference line generation unit 104 also line-draws the patient image 12, i.e., the output from the trained model 400 is an illustrated patient image 12 to which a reference line image has been added.

[0043] In the above embodiment, the present invention has been described assuming that both the wrinkle lines and the RSTL are used as reference lines, but it is also possible to use only the wrinkle lines or only the RSTL as reference lines.

[0044] In the above embodiment, the present invention has been described as an application program that is downloaded to a smartphone or the like and used, but the present invention can also be embodied as a storage medium that stores the application program. The present invention can also be embodied as a device (including a smartphone or tablet) that includes a processor and various memories and executes the above-described application program. Furthermore, the present invention can also be embodied as a SaaS-type system in which the application program is executed on a server and accessed from a user terminal.

[0045] Furthermore, in the above embodiment, an example has been presented in which local skin flap surgery is performed on the patient's face, but the target body part is not limited to the face, and may include the limbs, trunk, and neck.

[0046] Furthermore, in the above embodiment, an example was described in which an application supporting the design of a flap pattern for local flap surgery included a function for displaying reference lines to facilitate flap design. However, the situation in which reference lines are displayed is not limited to the flap pattern design. There are various situations in which displaying reference lines is effective in surgical procedures other than local flap surgery in plastic surgery, or even in fields other than plastic surgery. Therefore, the uses and fields of use of an application program having a function for displaying reference lines such as RSTL and wrinkle lines are not limited to the above example. Furthermore, the reference lines are not limited to RSTL and wrinkle lines, and any lines can be used as long as they are useful when designing an incision pattern for tissues such as skin and muscle. Furthermore, embodiments of the present invention also include the display of any geometric shape, not just "lines."

[0047] In the above embodiment, a two-dimensional image was read and processed as the patient's face data, but the reference lines may be overlaid on a three-dimensional polygon image instead of a two-dimensional one. In this case, it is possible to draw three-dimensional reference lines for a multi-face three-dimensional face instead of a flat one.

[0048] In the above embodiment, after the patient image 12 is input, the reference line generation unit 104 adds reference lines to the patient image 12 using the trained model 400. However, as another embodiment, as shown in FIG. 11 , the configuration may further include an image preprocessing unit 106 that processes the input patient image 12 after the patient image 12 is input. The image preprocessing unit 106 performs so-called preprocessing on the patient image 12 acquired by the image acquisition unit 102, removing parts that protrude from the facial surface, such as moles and tumors, to smooth out the facial surface. In this preprocessing, the parts to be removed may be detected by automatic image recognition based on predetermined rules, or the user may specify the parts to be removed on the patient image 12. The preprocessed patient image 12 is sent to the reference line generation unit 104, where a reference line image is generated by the trained model 400. The generated reference line image is overlaid on the patient image 12 and displayed in the image processing unit 105. As a result, for example, for patient images 12 having protruding features from the facial surface, such as moles or tumors that are the subject of plastic surgery, the surface can be smoothed by preprocessing, improving the accuracy of adding reference lines in the reference line generation unit 104. In addition, by performing preprocessing in the image preprocessing unit 106 on areas that are discolored differently from other skin parts, such as burn scars or blemishes, the recognition accuracy of the patient image 12 is improved, and the accuracy of adding RSTL in the reference line generation unit 104 is also improved.

[0049] The method of preprocessing in the image preprocessing unit 106 is not particularly limited, and preprocessing of the input patient image 12 may be performed using a trained model other than the trained model 400 described above.

[0050] An embodiment of the present invention can also be described as follows.

[0051] (First Program) A program for overlaying and displaying reference line data including at least one of RSTL and wrinkle lines on face data, A process of reading face data; A process of inputting face data into a trained model and obtaining reference line data corresponding to the face data; a process of displaying the face data or an illustration of the face data overlaid with the obtained reference line data; A program containing instructions that cause a computer processor to execute the program.

[0052] According to the first program, when facial data is read, reference line data including at least one of RSTL and wrinkle lines corresponding to the facial data is obtained based on the trained model, and is displayed overlaid on the facial data or an illustration thereof. This allows RSTL and wrinkle lines to be easily visually confirmed on the screen when considering incision and suture patterns, for example, in a preoperative conference for plastic surgery. Another advantage is that it makes it easier for inexperienced doctors to learn about the state of RSTL and wrinkle lines.

[0053] (Second Program) A program for overlaying and displaying reference line data including at least one of RSTL and wrinkle lines on face data, A process of reading face data; pre-processing the facial data to generate processed facial data; inputting the processed face data into a trained model and obtaining reference line data corresponding to the processed face data; a process of displaying the face data or an illustration of the face data overlaid with the obtained reference line data; A program containing instructions that cause a computer processor to execute the program.

[0054] According to the second program, when face data is read, preprocessing is performed on the read face data. Preprocessing is an optional process performed on the face data to remove parts from the face data that hinder the generation of reference lines or to facilitate the generation of reference lines. Then, by inputting the processed face data into a trained model, reference line data corresponding to the processed face data is obtained and displayed overlaid on the face data or an illustration thereof. This not only has the effect of making it easy to visually check reference lines, including RSTLs and wrinkle lines, on the screen, but also has the advantage of obtaining more accurate reference lines by performing preprocessing before the reference lines are obtained.

[0055] (First device) An apparatus for displaying reference line data including at least one of RSTL and wrinkle lines overlaid on face data, A process of reading face data; A process of inputting face data into a trained model and obtaining reference line data corresponding to the face data; a process of overlaying the obtained reference line data on the face data and displaying the same; 1. An apparatus comprising: a processor that executes

[0056] According to the first device, when facial data is read, reference line data corresponding to the facial data is obtained based on a trained model and displayed as an overlay on the facial data or an illustration thereof. This allows the reference lines to be easily visually confirmed on the screen when considering incision and suture patterns, for example, in a preoperative conference for plastic surgery. Another advantage is that it makes it easier for inexperienced doctors to learn the state of the reference lines.

[0057] (Second device) An apparatus for displaying reference line data including at least one of RSTL and wrinkle lines overlaid on face data, A process of reading face data; pre-processing the facial data to generate processed facial data; inputting the processed face data into a trained model and obtaining reference line data corresponding to the processed face data; a process of displaying the face data or an illustration of the face data overlaid with the obtained reference line data; 1. An apparatus comprising: a processor that executes

[0058] According to the second device, when face data is read, the read face data is preprocessed. Preprocessing is an optional process performed on the face data to remove portions of the face data that may hinder the generation of reference lines or to facilitate the generation of reference lines. Then, by inputting the processed face data into a trained model, reference line data corresponding to the processed face data is obtained and displayed overlaid on the face data or an illustration thereof. This not only has the effect of making it easy to visually check the reference lines on the screen, but also has the advantage of obtaining more accurate reference lines by performing preprocessing before the reference lines are obtained. [Explanation of symbols]

[0059] 10: Display, 101: Input processing unit, 102: Image acquisition unit, 103: Pattern storage unit, 104: Reference line generation unit, 105: Image processing unit, 106: Image pre-processing unit, 300: Learning device, 400: Trained model

Claims

1. A program for displaying reference line data including at least one of RSTL and wrinkle lines overlaid on face data, A process of reading face data; A process of inputting face data into a trained model and obtaining reference line data corresponding to the face data; a process of displaying the face data or an illustration of the face data overlaid with the obtained reference line data; A program containing instructions that cause a computer processor to execute the program.

2. A program for displaying reference line data including at least one of RSTL and wrinkle lines overlaid on face data, A process of reading face data; pre-processing the facial data to generate processed facial data; inputting the processed face data into a trained model and obtaining reference line data corresponding to the processed face data; a process of displaying the face data or an illustration of the face data overlaid with the obtained reference line data; A program containing instructions that cause a computer processor to execute the program.

3. An apparatus for displaying reference line data including at least one of RSTL and wrinkle lines overlaid on face data, A process of reading face data; A process of inputting face data into a trained model and obtaining reference line data corresponding to the face data; a process of overlaying the obtained reference line data on the face data and displaying the same; 1. An apparatus comprising: a processor that executes

4. An apparatus for displaying reference line data including at least one of RSTL and wrinkle lines overlaid on face data, A process of reading face data; pre-processing the facial data to generate processed facial data; inputting the processed face data into a trained model and obtaining reference line data corresponding to the processed face data; a process of displaying the face data or an illustration of the face data overlaid with the obtained reference line data; 1. An apparatus comprising: a processor that executes

Images