Pen having imaging function and image evaluation system

Abstract

The invention provides a pen with a shooting function and an image evaluation system.The pen comprises a pen holder 11, a camera 12 installed on the pen holder 11 and a sensor assembly and further comprises a control unit, and the control unit is used for receiving detection data uploaded by the sensor assembly; and under the condition that the inclination angle of the pen is determined to be greater than the usable angle of the pen through the detection data, the camera is controlled to shoot, so that a plurality of images containing the created content are obtained.

Description

Technical Field

[0001] The present disclosure relates to the technical fields of smart pens, etc., and specifically relates to a pen with a shooting function and an image evaluation system. Background Art

[0002] A smart pen is a newly emerging smart product. During use, a smart pen can provide other functions besides writing, such as functions like voice recording, haptic feedback, and image shooting. These functions are turned on and off by setting switches and manually operated by a person. However, when operating the start and stop of the above functions during the process of writing and drawing with the pen, there are defects such as difficult operation and affecting the user's pen movement. Summary of the Invention

[0003] The present disclosure provides a pen with a shooting function and an image evaluation system.

[0004] In a first aspect of the present disclosure, a pen with a shooting function is proposed, including: a pen barrel; a camera and a sensor assembly installed on the pen barrel; and a control unit configured to receive detection data uploaded by the sensor assembly and, when it is determined through the detection data that the inclination angle of the pen is greater than the usable angle of the pen, control the camera to take pictures, obtaining multiple images containing creative content.

[0005] According to some embodiments of the present disclosure, the sensor assembly includes an attitude sensor for detecting the acceleration and angular velocity of the pen barrel when the camera takes pictures.

[0006] According to some embodiments of the present disclosure, the control unit is further configured to determine the moving speed of the pen through the acceleration and the angular velocity, and adjust the shooting frequency of the camera according to the change of the moving speed.

[0007] Controlling the camera to take pictures specifically means: controlling the camera to take pictures at regular intervals according to the shooting frequency until it is determined that the inclination angle is not greater than the usable angle and then stopping the shooting.

[0008] According to some embodiments of the present disclosure, the pen further includes: a wireless transmission unit installed on the pen barrel, configured to send the data collected by the camera and at least part of the sensor assembly to a server.

[0009] According to some embodiments of the present disclosure, the pen is a writing brush or a fountain pen or a pen with a replaceable refill.

[0010] According to some embodiments of the present disclosure, the sensor assembly includes a gravity sensor for detecting the inclination angle.

[0011] According to some embodiments of the present disclosure, the available angle is one of the angles in the angle range of [5, 70].

[0012] According to some embodiments of the present disclosure, the camera is installed at a position on the outer wall of the pen shaft close to the pen tip and faces the pen tip.

[0013] A second aspect of the present disclosure provides an image evaluation system, including: a pen with a shooting function according to any of the above embodiments; and an evaluation module configured on the server side, which is used to receive the acceleration data, angular velocity data, and multiple images uploaded by the pen, perform image correction on the multiple images through the acceleration data and the angular velocity data, splice the corrected images to obtain a panoramic view of the creation content, and output an aesthetic evaluation result of the creation content through an evaluation model.

[0014] According to some embodiments of the present disclosure, the method of performing image correction on multiple images through the acceleration data and the angular velocity data is specifically: calculating the pose matrix of the pen at that moment through the acceleration data and the angular velocity data at the same moment; and performing image correction on the image taken at that moment through the inverse matrix of the pose matrix, and the viewing angle inclination of the corrected image is less than or equal to the viewing angle inclination of the captured image.

[0015] According to some embodiments of the present disclosure, the steps of splicing the corrected images to obtain a panoramic view of the creation content include: extracting feature points from each frame of the corrected image; performing image registration based on the correspondence of the feature points between adjacent frames; and splicing the frames of images located in the same coordinate system obtained by registration to obtain a panoramic view of the creation content.

[0016] According to some embodiments of the present disclosure, the steps of performing image registration based on the correspondence of the feature points between adjacent frames include: determining the correspondence of the feature points between adjacent frames; determining the transformation matrix between the adjacent frames based on the correspondence; and performing image registration on each frame of image based on the transformation matrix.

[0017] According to some embodiments of the present disclosure, the method of splicing the frames of images located in the same coordinate system obtained by registration is specifically: splicing the frames of images located in the same coordinate system obtained by registration based on a temporal consistency algorithm.

[0018] According to some embodiments of the present disclosure, the steps of outputting an aesthetic evaluation result of the creation content through an evaluation model include: extracting the area of the creation content from the panoramic view; and inputting the area of the creation content into the evaluation model, and obtaining an aesthetic evaluation result of the creation content based on the model output.

[0019] According to some embodiments of the present disclosure, the model output includes scores of multiple scoring items in stroke structure, color depth, and content layout; the step of obtaining the aesthetic evaluation result of the created content based on the model output includes: inputting the scores and descriptions of the corresponding scoring items into a language model to obtain the aesthetic evaluation result of the created content.

[0020] According to some embodiments of the present disclosure, the aesthetic evaluation result includes a style evaluation result and a writing stroke suggestion. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The drawings illustrate exemplary embodiments of the present disclosure and, together with the description thereof, are used to explain the principles of the present disclosure. These drawings are included to provide a further understanding of the present disclosure and are included in this specification and form a part of this specification.

[0022] Figure 1 The overall structural schematic diagram of a pen with a photographing function showing some embodiments of the present disclosure.

[0023] Figures 2 - 4 The overall flowchart showing the control unit controlling the camera to collect images in some embodiments of the present disclosure.

[0024] Figure 5 The application scenario schematic diagram of an image evaluation system showing some embodiments of the present disclosure.

[0025] Figures 6 - 9 The overall flowchart showing the image evaluation system evaluating a user's work in some embodiments of the present disclosure. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0026] The present disclosure will be further described in detail below in conjunction with the drawings and embodiments. It can be understood that the specific embodiments described herein are only used to explain the relevant content and do not limit the present disclosure. Additionally, it should be noted that for the sake of description, only parts related to the present disclosure are shown in the drawings.

[0027] It should be noted that, without conflict, the embodiments in the present disclosure and the features in the embodiments can be combined with each other. The technical solutions of the present disclosure will be described in detail below with reference to the drawings and embodiments.

[0028] Unless otherwise specified, the exemplary embodiments / examples shown will be understood to provide exemplary features of various details of some ways that can implement the technical concept of the present disclosure in practice. Therefore, unless otherwise specified, without departing from the technical concept of the present disclosure, the features of various embodiments / examples can be combined, separated, interchanged, and / or rearranged additionally.

[0029] The terms used herein are for the purpose of describing particular embodiments and are not limiting. As used herein, unless the context clearly indicates otherwise, the singular forms "a", "an" and "the" are also intended to include the plural forms. In addition, when the terms "comprising" and / or "including" and their variants are used in this specification, it is stated that there are the stated features, integers, steps, operations, components, assemblies, and / or groups thereof, but does not exclude the presence or addition of one or more other features, integers, steps, operations, components, assemblies, and / or groups thereof. It should also be noted that, as used herein, the terms "substantially", "about" and other similar terms are used as approximate terms and not as terms of degree, so they are used to explain the inherent deviations of measured values, calculated values and / or provided values that those of ordinary skill in the art will recognize.

[0030] The smart pen can provide various related functions other than writing and drawing, but these functions need to be manually triggered. For example, taking a photo each time a button is pressed, which is inconvenient to operate and has a low degree of intelligence. For this reason, the present disclosure proposes a pen with a shooting function.

[0031] Figure 1 The overall structural schematic diagram of the pen with a shooting function according to some embodiments of the present disclosure is shown. As Figure 1 shown, the pen with a shooting function includes a pen barrel 11, a camera 12 and a sensor assembly installed on the pen barrel 11, and also includes a control unit. The sensor assembly and the control unit are not shown in Figure 1 it.

[0032] The user can hold the pen barrel 11 and write and draw (i.e., create) on the paper 14. The pen in this embodiment can be Figure 1 the writing brush shown, or a fountain pen, or a ballpoint pen or other pen with a replaceable refill. Taking the writing brush as an example, the user can write Chinese calligraphy with the writing brush on the paper 14. A detachable pen tip 13 can be installed at the tip of the pen barrel 11, and the type and shape of the pen hair can be changed by replacing the pen tip 13.

[0033] Figure 2 The overall flow schematic diagram of the control unit controlling the camera to collect images according to some embodiments of the present disclosure is shown. As Figure 2 shown, the control unit is used to execute the following steps S110 and step S120.

[0034] S110, receiving the detection data uploaded by the sensor assembly.

[0035] S120, when it is determined through the detection data that the inclination angle of the pen is greater than the usable angle of the pen, controlling the camera to take pictures to obtain multiple images containing the creation content.

[0036] The control unit and the sensor assembly can be built inside the pen shaft 11, or integrated with the camera 12 and jointly installed at Figure 1 the position of the camera 12. The camera 12 and the sensor assembly are electrically connected to the control unit respectively. It can be understood that a power supply device such as a battery can be installed at the position of the camera 12 or inside the pen shaft 11 to supply power to electrical devices such as the control power supply, the camera 12 and the sensor assembly.

[0037] The sensor assembly can continuously detect acceleration or other types of data when powered, and continuously upload the detected data to the control unit for determining the inclination angle of the pen. The inclination angle refers to the angle between the pen (i.e., the pen shaft 11) and the horizontal plane. The inclination degree of the pen can be reflected by the inclination angle, so as to determine whether the pen is in an unused state.

[0038] Exemplarily, the sensor assembly may include a gravity sensor for detecting the inclination angle. The gravity sensor (G-sensor) can convert gravity into an electrical signal through a displacer made of an elastic sensitive element and a storage spring, and measure the acceleration caused by gravity, that is, the detected data is acceleration data. The control unit can calculate the inclination angle of the pen relative to the horizontal plane through the acceleration signal uploaded by the gravity sensor.

[0039] The control unit can also continuously judge whether the inclination angle is greater than the usable angle when powered. The usable angle refers to the angle at which the pen can be used for creation (writing and drawing). When the pen is placed flat on the desktop or the pen holder, the detected inclination angle of the pen will be less than or equal to the usable angle of the pen. At this time, the pen is not in a used state, so the control unit will not control the camera 12 to take pictures. When the user picks up the pen to write, due to the pen-holding posture, the pen is close to an upright posture, so the inclination angle of the pen will be greater than the usable angle. When the inclination angle changes from never being greater than the usable angle to being greater than the usable angle, the control unit will determine that the inclination angle is greater than the usable angle through the acceleration uploaded by the sensor assembly, and then start triggering the continuous shooting action of the camera until the user puts down the pen after finishing writing, then the control unit will determine that the inclination angle is not greater than the usable angle, and control the camera to stop shooting.

[0040] The above-mentioned available angle can be any angle within the range of [5, 70] degrees. Since the pen-holding postures of different pens are different, the range of inclination angle changes of the pen during the process of writing and painting (abbreviated as creation) is also different. For example, for a writing brush, since the writing brush is usually held in a nearly vertical posture when writing, the available angle of the writing brush can be larger than that of other pens. For example, the available angle can be set to 70 degrees, so that almost all of the captured images are the images of the user during the writing and painting process, and as few images as possible of the user from picking up the pen to starting to write and paint are included. For pens such as fountain pens, since the pen-holding posture is relatively inclined, the available angle can be set to 20 degrees or other angles.

[0041] It can be understood that for some writing brushes, instead of being placed on a pen rest, they may be hung on a pen rack or placed in a pen container. Therefore, the sensor assembly may further include a pressure sensor installed on the surface of the pen shaft 11. One or more pressure sensors may be provided, and these pressure sensors are all connected to the control unit and continuously powered by the power supply device for detecting the pressure received by the pen shaft 11. If the user hangs the pen on the pen rack, although the user is not writing or painting, the control unit will determine that the user is writing or painting and thus trigger the camera to take a picture.

[0042] Based on this, the pressure sensor can be combined with the gravity sensor to determine whether the user is writing or painting. Only when the pressure data feedback by the pressure sensor indicates the existence of pressure and the inclination angle is greater than the available angle, the control unit will determine that the user is writing or painting, and thus control the camera to start shooting. If the pressure data feedback by the pressure sensor indicates the absence of pressure, or the inclination angle is not greater than the available angle, the control unit will not determine that the user is writing or painting, and thus control the camera to stop shooting.

[0043] The camera can be installed at a position on the outer wall of the pen shaft near the pen tip and facing the pen tip, as Figure 1 shown. In this way, the camera is at an appropriate distance from the pen tip, and the captured images can include the content newly written or painted by the user. This can not only make the captured images clearer, but also not be too close to the pen tip, resulting in too small a field of view, so that less of the content newly written or painted by the user is included in the captured images. Specifically, the camera can be a pinhole camera with a resolution greater than or equal to 1080P and a frame rate of 30fps.

[0044] Suppose the user picks up the pen from the table at time T1. Then the inclination angle at T1 is greater than the usable angle. The control unit sends a start shooting signal to the camera, and the camera starts continuously shooting images. At time T2, the user starts writing Chinese calligraphy with a brush on the paper, and the posture of the pen always remains in a posture where the inclination angle is greater than the usable angle, and the camera also continuously shoots images. At time T3, the user has written a total of n characters and places the pen on the pen holder. At this time, the inclination angle is less than the usable angle. The control unit sends a stop shooting signal to the camera, and the camera stops shooting, obtaining a total of m images. Among them, each captured image contains a part of the user's writing and drawing content. For example, if the user shoots n images during the process of writing a character, if the character is relatively large, each image only contains a part of the strokes of this character; if the character is relatively small, each image may contain at least one complete character and at least one incomplete character among the n characters. These images can be used to be uploaded to the server and downloaded by the user for viewing later, and can also be used to be processed by the model and evaluate the writing content.

[0045] The pen with a shooting function proposed according to the embodiments of the present disclosure can determine the inclination angle of the pen in real time through the acceleration data detected by the sensor, so as to dynamically start and stop the shooting of the camera through the change of the inclination angle of the pen. The whole process is fully automated processing. The image transmission bandwidth occupancy is reduced to 200KB / minute, and there is no need to set a switch for starting and stopping shooting, nor does the user need to perform multiple manual controls to achieve multiple shootings, reducing user intervention and not interfering with and affecting the user's writing and drawing pen movement process, realizing non-intrusive image acquisition. At the same time, when the user is not writing, the camera shooting can be automatically stopped, reducing the power consumption of the pen and improving the battery life. The power consumption can be reduced by about 40%, and the battery life can reach about 8 hours.

[0046] In addition, compared with obtaining the trajectory image of writing and drawing by installing a pressure sensor on the back of a special writing medium and collecting the pressure data when writing and drawing on the writing medium with a pen, it is possible to obtain the data containing the writing trajectory by creating on ordinary paper without using a specific writing medium, eliminating the configuration of the writing medium, and not being limited by the size of the writing medium in terms of the writing size range, improving the convenience of trajectory data collection, and not requiring power supply for the writing medium, reducing the power consumption.

[0047] The sensor assembly may include an attitude sensor for detecting the acceleration and angular velocity of the pen barrel when the camera shoots. The attitude sensor may include sensors such as an accelerometer, a gyroscope, and a magnetometer. These sensors can detect angular velocity data and acceleration data in real time, and these data can be used to realize functions such as adjusting the shooting frequency and correcting the captured images.

[0048] Figure 3The figure shows a schematic diagram of the overall process of the control unit controlling the camera to capture images in some other embodiments of the present disclosure. Refer to Figure 3 When the detection data of the sensor component includes acceleration data and angular velocity data, the control unit can also be used to perform the following step S130. Step S130 and step S120 can be executed asynchronously.

[0049] S130, Determine the moving speed of the pen through acceleration and angular velocity, and adjust the shooting frequency of the camera according to the change of the moving speed.

[0050] The moving speed of the pen can be calculated through the acceleration data and the angular velocity data. For example, the moving speed can be calculated by means of integration and approximate calculation. If the moving speed slows down, the shooting frequency of the camera 12 can be lowered to reduce the number of images containing the same creative content in the captured images. If the moving speed increases, the shooting frequency of the camera 12 can be increased to avoid the incoherence of the image content due to the lack of repeated content between adjacent frame images. This can adapt to complex usage scenarios and improve robustness.

[0051] It can be understood that the period of the sensor component for detecting the detection data can be a multiple of the shooting frequency, so that regardless of how the shooting frequency changes, the acquisition frequency of the detection data is a multiple of the shooting frequency, which can ensure that there are corresponding acceleration data and angular velocity data each time an image is captured.

[0052] In addition, the change of the moving speed can be determined by a speed interval. Different speed intervals can correspond to different shooting frequencies. The control unit can determine the speed interval where the current moving speed is located in real time. When the moving speed changes to belong to another speed interval, the corresponding shooting frequency can be set according to the current speed interval.

[0053] Figure 4 The figure shows a schematic diagram of the overall process of the control unit controlling the camera to capture images in some other embodiments of the present disclosure. Refer to Figure 4 In step S120, the specific method of controlling the camera to take pictures can be: controlling the camera to take pictures at a shooting frequency until it is determined that the inclination angle is not greater than the available angle, and then stop taking pictures. The shooting frequency can be set with a default value, and the default value can be 2 times per second. The default value can correspond to one of the speed intervals q3. When it is detected that the moving speed of the pen changes from belonging to the q3 interval to belonging to a speed interval q4 with a greater speed, the shooting frequency will also change from the default value to a higher frequency (such as 4 times or other frequencies). When it is detected that the moving speed of the pen changes from belonging to the q3 interval to a speed interval q2 with a smaller speed, the shooting frequency will also change from the default value to a lower frequency (such as 1.5 times or other frequencies).

[0054] The pen may further include a wireless transmission unit installed on the pen shaft, configured to send data collected by the camera and at least some of the sensor components to the server. The wireless transmission unit may be integrated with the camera 12, the control unit and the sensor components and installed together at the position of the camera 12 in Figure 1 the pen. The wireless transmission unit is used to upload the images collected by the camera 12 and the acceleration data and angular velocity data collected by the attitude sensor in the sensor components to the server, so that the server can process these data. The data collected by the gravity sensor may not need to be uploaded and is only used for determining the start and stop of shooting at the pen tip locally. The wireless transmission unit may specifically include one or more of a wifi unit and a bluetooth unit, and wirelessly transmits data via wireless wifi or bluetooth.

[0055] The control unit may send the images in batches through the wireless transmission unit. For example, every time the camera 12 captures k images, the control unit sends a group of k images through the wireless transmission unit. k may be 60 or other values.

[0056] Correspondingly, the acceleration data and angular velocity data may also be sent in batches through the wireless transmission unit, and the acceleration data and angular velocity data may be sent as data in the same batch as the images. The overall acquisition period of multiple images included in the data of the same batch is the same as the overall acquisition period of the velocity data. For example, when the control unit receives 60 images sent by the camera, these 60 images are used as the images in the batch data D1, and the acceleration data and angular velocity data collected during the time of collecting these 60 images will be used as the velocity data in the batch data D1, and together with the 60 images, form the batch data D1, and the control unit sends D1 through the wireless transmission unit.

[0057] The integrated circuit including the camera, the sensor components, the control unit and the wireless transmission unit may implement its hardware structure using a bus architecture. The bus architecture may include any number of interconnecting buses and bridges, depending on the specific application of the hardware and the overall design constraints. The bus connects various circuits including one or more processors, memories and / or hardware modules together. The bus may also connect various other circuits such as peripheral devices, voltage regulators, power management circuits, external antennas, etc.

[0058] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, an Extended Industry Standard Architecture (EISA) bus, etc. The bus can be divided into an address bus, a data bus, a control bus, etc. For the sake of representation, only one connection line is used in this figure, but it does not mean that there is only one bus or one type of bus.

[0059] Among them, the processor can be a Central Processing Unit (CPU). The processor can also be other general-purpose processors, Digital Signal Processors (DSPs), Application Specific Integrated Circuits (ASICs), Field-Programmable Gate Arrays (FPGAs) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components and other chips, or a combination of the above types of chips.

[0060] The memory can be used as a non-transitory computer-readable storage medium and can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as program instructions of the computer program in the embodiments of the present disclosure. The processor runs the non-transitory software programs, instructions, and modules stored in the memory.

[0061] The memory can include a program storage area and a data storage area. Among them, the program storage area can store an operating system and application programs required for at least one function; the data storage area can store data created by the processor, etc. In addition, the memory can include high-speed random access memory and can also include non-transitory memory, such as at least one disk storage device, a flash memory device, or other non-transitory solid-state storage devices. In some embodiments, the memory 1300 may optionally include a memory remotely set relative to the processor 1200, and these remote memories can be connected to the processor 1200 through a network. Examples of the above networks include but are not limited to the Internet, an enterprise intranet, a local area network, a mobile communication network, and combinations thereof.

[0062] Figure 5 The application scenario schematic diagram of the image evaluation system according to some embodiments of the present disclosure is shown. Refer to Figure 5, the application scenario includes a pen 10, a server 20, and a client 30. The pen 10 can communicate with the server 20 and the client 30 respectively to perform data sending and receiving. In the present disclosure, the pen 10, the server 20, and the client 30 each include at least one processor and at least one memory.

[0063] Exemplarily, the client 30 can be a terminal device held by a user, and the server 20 can be a server of a service provider's business system. After the pen 10 is powered on, the client 30 can connect to the pen 10 via Bluetooth and configure the Wi-Fi information of the pen 10 via Bluetooth. After the configuration is completed, whenever the pen 10 is powered on, it can automatically connect to the network via Wi-Fi configuration and connect to the server 30 via Wi-Fi, so as to upload multiple collected images and detection data at multiple moments to the server of the server 30. The server can generate a panoramic view of the user's work based on the images and detection data, perform an aesthetic evaluation on the panoramic view, obtain an evaluation result and save it. The user can access the server 20 via the client 30 and download the panoramic view of their own work and its corresponding evaluation content from the server.

[0064] In Figure 5 the shapes and structures of the pen 10, the server 20, and the client 30 shown should not be construed as limiting the protection scope of the present disclosure. In the present disclosure, the "terminal device" can be different types of electronic devices. For example, the terminal device can be a mobile phone, a tablet computer, a laptop computer, or a desktop computer, etc. Additionally, the server can be a server with a physical form or a cloud server, and the present disclosure does not limit the type of the server.

[0065] The image evaluation system can include the pen 10 and an evaluation module configured in the server 20.

[0066] The pen 10 can be the pen with a shooting function described in any of the above embodiments, including a pen barrel, a camera and a sensor assembly installed on the pen barrel, and also includes a control unit. The control unit is used to receive the detection data uploaded by the sensor assembly, and when it is determined through the detection data that the inclination angle of the pen is greater than the usable angle of the pen, control the camera to take pictures to obtain multiple images containing the creation content.

[0067] Figure 6 shows a schematic diagram of the overall process of the image evaluation system of some embodiments of the present disclosure for evaluating a user's work. Refer to Figure 6 , the evaluation module is used to execute the following steps S210, step S220, step S230, and step S240.

[0068] S210, receive the acceleration data, angular velocity data, and multiple images uploaded by the pen.

[0069] The pen 10 communicates and connects with the server 20 (evaluation module) via Wi-Fi. If the pen 10 sends images, acceleration data, and angular velocity data in batches, the communication module will also receive the corresponding batch of data.

[0070] S220, perform image correction on multiple images using the acceleration data and angular velocity data.

[0071] During the process from when the user picks up the pen to when the user puts down the pen, the camera captures a total of m images, which are transmitted in multiple batches. When the server receives all batches of data, it obtains m images, as well as the acceleration data and angular velocity data detected during the process.

[0072] The acceleration data and angular velocity data at the same moment can be used to identify the posture of the pen 10 at that moment. Since the pen will be tilted to a certain extent during the user's writing and drawing creation process, the captured images are also images with a tilted perspective. The identified posture can be used to correct the images captured at the above-mentioned moment to obtain images with a frontal perspective. The frontal perspective refers to the perspective perpendicular to the paper surface.

[0073] S230, splice the corrected images to obtain a panoramic view of the creation content.

[0074] Each of the above m images only contains a part of the user's creation content. Therefore, the content of each image can be spliced through an image splicing algorithm to obtain an image containing all of the user's creation content, that is, a panoramic view. If the user writes a total of n characters, the panoramic view can contain all n characters.

[0075] S240, output the aesthetic evaluation result of the creation content through the evaluation model.

[0076] The evaluation model is a pre-trained model used to perform aesthetic evaluation on the input images, such as evaluating in terms of color, lines, structure, etc. The evaluation result can include only a score or a comment expressed in natural language. Inputting the panoramic view into the pre-trained evaluation model can obtain the evaluation result.

[0077] The image evaluation system proposed according to the embodiments of the present disclosure can perform real-time image feedback through the pen, perform image perspective correction on the images collected during the user's creation process using the collected acceleration and angular velocity, eliminate image distortion caused by image tilt and rotation, generate a panoramic view containing all of the user's current creation content with a frontal perspective, and the amount of invalid data transmission can be reduced by approximately 70%. And use the evaluation model to perform aesthetic analysis on the user's works, evaluate the user's creation content (written characters or paintings) from an aesthetic perspective, so as to inform the user of the advantages, disadvantages, and style of the user's works and help the user create better.

[0078] Figure 7 The figure shows a schematic diagram of the overall process of the image evaluation system according to other embodiments of the present disclosure for evaluating user works. Refer to Figure 7 , the method of image correction for multiple images through acceleration data and angular velocity data (step S220) may specifically include the following steps S221 and S222.

[0079] S221, calculate the pose matrix of the pen at this moment through the acceleration data and angular velocity data at the same moment.

[0080] The direction of the pen 10 relative to the paper surface can be obtained through acceleration and angular velocity, and the matrix of the pose of the pen 10 in three-dimensional space, that is, the pose matrix, can be calculated. The pose matrix describes the conversion relationship between the coordinate system of the pen and the world coordinate system. The world coordinate system can be the plane coordinate system where the paper is located.

[0081] S222, perform image correction on the image captured at this moment through the inverse matrix of the pose matrix. Among them, the viewing angle inclination of the corrected image is less than or equal to the viewing angle inclination of the captured image.

[0082] After obtaining the pose matrix, the inverse matrix of the pose matrix can be calculated, and the image coordinates of the obliquely captured image can be converted back to the correct position on the paper surface through the inverse matrix. For example, for each pixel point on the captured image, use the inverse pose matrix for coordinate transformation and remap them to the correct two-dimensional coordinates, thereby correcting the distortion of the captured image. The viewing angle corresponding to the corrected image has a lower inclination or even completely eliminates the inclination compared to the viewing angle corresponding to the original captured image, and an image under a viewing angle (frontal viewing angle) that is perpendicular to the paper surface is obtained. For each of the m captured images, correction is performed through the acceleration data and angular velocity data when the image is captured, and m corrected images are obtained. For calligraphy works, the problem of low character recognition rate caused by image distortion and background interference is solved, and the character recognition accuracy can reach about 98.5%, which is higher than the maximum accuracy of 85% of the traditional scheme.

[0083] Figure 8 The figure shows a schematic diagram of the overall process of the image evaluation system according to other embodiments of the present disclosure for evaluating user works. Refer to Figure 8 , the method of stitching the corrected images to obtain a panoramic view of the creation content (step S230) may specifically include the following steps S231, S232, and S233.

[0084] S231, extract feature points from each frame of the corrected image. After obtaining m corrected images, for each image, feature points such as corners and edges are extracted. The algorithm for extracting feature points can be the SIFT (Scale-invariant feature transform) algorithm, SURF (Speeded Up Robust Features), ORB (Oriented FAST and Rotated BRIEF) algorithm, or other algorithms.

[0085] S232, perform image registration based on the correspondence of feature points between adjacent frames.

[0086] For every two images adjacent in time sequence, match the feature points between the two images to obtain the correspondence between the feature points of the two images, and thus use the correspondence to achieve image registration. The registered images will be in the same coordinate system, that is, the origin positions of the registered images are the same.

[0087] Specifically, the method of performing image registration (step S232) based on the correspondence of feature points between adjacent frames may include: determining the correspondence of feature points between adjacent frames; determining the transformation matrix between adjacent frames based on the correspondence; and performing image registration on each frame image based on the transformation matrix.

[0088] After obtaining the correspondence of feature points, the geometric transformation relationship between adjacent images can be calculated therefrom. The geometric transformation relationship is represented by a transformation matrix. Specifically, the transformation matrix can be calculated by affine transformation, homography transformation, or other methods, and the transformation relationship such as translation, rotation, and scaling is described by the transformation matrix. Since the camera does not need to change the focal length during shooting and the camera will not move on the pen shaft, the transformation matrix is mainly a translation matrix and a rotation matrix. The rotation matrix is generated because the pen shaft may rotate during the user's writing process, causing the image to rotate horizontally, so image registration is performed through the rotation matrix.

[0089] S233, splice the registered frame images in the same coordinate system to obtain a panoramic view of the creative content.

[0090] After registration, all m images are in the same coordinate system, and then the complete appearance of the painting written by the user can be restored through image splicing to obtain a panoramic view containing the complete appearance of the work. When performing image fusion and splicing, the duplicate content in different images can be eliminated by the direct averaging method, the weight-based fusion method, or other methods.

[0091] Specifically, the method of stitching the registered images in the same coordinate system (step S233) may include: stitching the registered images in the same coordinate system based on the temporal consistency algorithm.

[0092] The temporal consistency algorithm is used to ensure the temporal consistency and coherence of the generated image sequence by utilizing temporal information during the stitching process. For example, the movement path of specific strokes or feature points can be traced throughout the video sequence to ensure the continuity and natural transition of the writing trajectory. For frames that may exhibit jitter or instability, smoothing techniques (such as the Kalman filter) can be employed to optimize the trajectory and ensure the stability of the stitching result. Subsequently, some post-processing operations can be performed on the stitched image, such as cropping unnecessary edge parts, adjusting brightness and contrast to enhance the visual effect, etc.

[0093] Figure 9 The schematic diagram shows the overall process of the image evaluation system in some other embodiments of the present disclosure for evaluating the user's work. Refer to Figure 9 , the method of outputting the aesthetic evaluation result of the creative content through the evaluation model (step S230) may specifically include the following steps S241 and S242.

[0094] S241, extract the region of the creative content from the panoramic image. For example, a U-Net segmentation network can be used to extract the part of the creative content from the background of the panoramic image, such as extracting the black text area from a white paper.

[0095] S242, input the region of the creative content into the evaluation model, and obtain the aesthetic evaluation result of the creative content based on the model output.

[0096] The evaluation model is equivalent to an aesthetic evaluation model, and specifically can be a convolutional neural network model (CNN). The evaluation model can evaluate the work from multiple perspectives. For example, the model output can include the scores of multiple scoring items such as stroke structure, color depth, and content layout, that is, evaluate from the above three scoring perspectives. The score range can be from 0 to 100 points.

[0097] Taking the evaluation of calligraphy works as an example, when the evaluation model is trained, a large dataset of calligraphy sample images with different styles, qualities, font styles, font sizes, and colors can be constructed first, and the images in it are manually given aesthetic scores. Then, data preprocessing is carried out to make the sizes, resolutions, etc. of the images in the dataset the same or approximately the same. Then the dataset can be divided into a training set (which can account for 85% of the images) and a test set (which can account for 15% of the images). The traditional calligraphy rules are encoded as a loss function, or the mean squared error (MSE) or other methods are used as the loss function. The images in the training set are input into the evaluation model and the parameters are adjusted using the loss function, thus completing the supervised training. After that, the images in the test set are input into the evaluation model, and the model output is compared with the corresponding manual scores to determine whether the accuracy rate of the evaluation model meets the requirements. The evaluation accuracy rate can be approximately increased to 92%, which is higher than the highest 75% accuracy rate of the traditional scheme.

[0098] In step S242, the method for obtaining the aesthetic evaluation result of the creative content based on the model output can specifically be: inputting the score and the description of the corresponding evaluation item into the language model to obtain the aesthetic evaluation result of the creative content.

[0099] In order to make the evaluation result easier to understand, the natural language introductions of each evaluation item and the corresponding scores of these evaluation items can be input into the large language model (LLM), and the large language model is used to give the natural language evaluation result.

[0100] The aesthetic evaluation result can include the style evaluation result and the writing stroke suggestions. For example, the aesthetic evaluation result is: "The writing stroke is smooth, but the composition is slightly loose."

[0101] The evaluation result obtained by the evaluation model does not need to rely on a fixed rule library, and can combine calligraphy aesthetic rules with AI generation capabilities to provide real-time creation optimization and expert-level evaluation, with personalized feedback and generation capabilities.

[0102] The evaluation module can also be used to send the panoramic image and the corresponding aesthetic evaluation result to the user terminal 30. Or, the user can operate the user terminal 30 to log in to the server 20 and then download the calligraphy works written by himself before and their evaluation contents.

[0103] In the description of this specification, the descriptions referring to terms such as "one embodiment / way", "some embodiments / ways", "example", "specific example", or "some examples", etc., mean that the specific features, structures, or characteristics described in connection with that embodiment / way or example are included in at least one embodiment / way or example of the present disclosure. In this specification, the schematic expressions of the above terms do not necessarily refer to the same embodiment / way or example. Moreover, the specific features, structures, or characteristics described can be combined in a suitable manner in any one or more embodiments / ways or examples. In addition, without contradiction, those skilled in the art can combine and combine the different embodiments / ways or examples described in this specification and the features of different embodiments / ways or examples.

[0104] In addition, the terms "first" and "second" are used only for descriptive purposes and cannot be construed as indicating or implying relative importance or implicitly specifying the quantity of the indicated technical features. Thus, the features defined with "first" and "second" may explicitly or implicitly include at least one of such features. In the description of the present disclosure, "a plurality" means at least two, such as two, three, etc., unless otherwise specifically defined.

[0105] Those skilled in the art should understand that the above embodiments are merely for clearly illustrating the present disclosure and are not intended to limit the scope of the present disclosure. For those skilled in the art, other changes or modifications can be made based on the above disclosure, and these changes or modifications are still within the scope of the present disclosure.

Claims

1. A pen with a shooting function, characterized in that, Comprising: A pen shaft; A camera and a sensor assembly mounted on the pen shaft; And A control unit, configured to receive detection data uploaded by the sensor assembly, and control the camera to take pictures when it is determined through the detection data that the inclination angle of the pen is greater than the available angle of the pen, so as to obtain multiple images containing the creation content.

2. The pen with a photographing function according to claim 1, wherein The sensor assembly includes an attitude sensor for detecting the acceleration and angular velocity of the pen shaft when the camera takes pictures.

3. The pen with a photographing function according to claim 2, wherein The control unit is further configured to determine the moving speed of the pen through the acceleration and the angular velocity, and adjust the shooting frequency of the camera according to the change of the moving speed.

4. The pen with a photographing function according to claim 1 or 2, characterized in that, Controlling the camera to take pictures specifically means: controlling the camera to take pictures at regular intervals according to the shooting frequency until it is determined that the inclination angle is not greater than the available angle and then stopping shooting.

5. An image evaluation system, characterized in that, Comprising: The pen according to any one of claims 1-4; And An evaluation module configured on the server side, for receiving the acceleration data, angular velocity data and multiple images uploaded by the pen, performing image correction on the multiple images through the acceleration data and the angular velocity data, splicing the corrected images to obtain a panoramic view of the creation content, and outputting an aesthetic evaluation result of the creation content through an evaluation model.

6. The image evaluation system according to claim 5, characterized in that, The method of performing image correction on the multiple images through the acceleration data and the angular velocity data specifically is: Calculating the attitude matrix of the pen at this moment through the acceleration data and the angular velocity data at the same moment; and Performing image correction on the image taken at this moment through the inverse matrix of the attitude matrix, and the viewing angle inclination of the corrected image is less than or equal to the viewing angle inclination of the taken image.

7. The image evaluation system according to claim 5, wherein The steps of splicing the corrected images to obtain a panoramic view of the creation content include: Extracting feature points from each frame of the corrected image; Performing image registration based on the corresponding relationship of the feature points between adjacent frames; and Splicing the frames of images obtained by registration and located in the same coordinate system to obtain a panoramic view of the creation content.

8. The image evaluation system according to claim 7, characterized in that, The steps of performing image registration based on the corresponding relationship of the feature points between adjacent frames include: Determining the corresponding relationship of the feature points between adjacent frames; Determining the transformation matrix between the adjacent frames based on the corresponding relationship; and Performing image registration on each frame of image based on the transformation matrix.

9. The image evaluation system according to claim 5, wherein The steps of outputting an aesthetic evaluation result of the creation content through an evaluation model include: Extracting the area of the creation content from the panoramic view; and Inputting the area of the creation content into the evaluation model, and obtaining an aesthetic evaluation result of the creation content based on the model output.

10. The image evaluation system according to claim 9, wherein, The model output includes scores of multiple scoring items such as stroke structure, color depth, and content layout; The steps of obtaining an aesthetic evaluation result of the creation content based on the model output include: Inputting the scores and the descriptions of the corresponding scoring items into a language model to obtain an aesthetic evaluation result of the creation content.

Images