Coaxial fiber printing control method and device based on reinforcement learning

Abstract

The application relates to the field of artificial intelligence, and provides a coaxial fiber printing control method and device based on reinforcement learning. The method comprises the following steps: receiving device parameters of a coaxial printing device and coaxial fiber images collected by a camera on the coaxial printing device; identifying the coaxial fiber images to obtain fiber features; determining a current state from the fiber features and the device parameters, inputting the current state into a pre-trained expected return prediction model to obtain expected return values of each action, the expected return prediction model being part of reinforcement learning and being used for predicting expected returns of each action in a given state; controlling the coaxial printing device to run by using the action with the maximum expected return value; and judging whether a printing head of the coaxial printing device has run to an end point, if not, returning to execute the steps of receiving the device parameters and the coaxial fiber images and the subsequent steps, and if yes, shutting down the coaxial printing device. The application can intelligently determine optimal printing parameters, and can improve printing efficiency and the printing device.

Description

TECHNICAL FIELD

[0001] The present application relates to the field of artificial intelligence, and in particular to a coaxial fiber printing control method and device based on reinforcement learning. BACKGROUND

[0002] Semi-solid extrusion printing technology is a technology that uses pressure to extrude semi-solid slurry through a needle to form a specific shape object layer by layer. This printing technology does not have specific requirements for the melting point of the material, and has the characteristics of simple process and low cost. In the prior art, according to the different properties of the slurry, the common solidification methods of semi-solid extrusion printing are divided into normal temperature natural solidification, heating solidification and ultraviolet light solidification. Among them, heating solidification is easy to cause harm to the biological body and can only be used for extracorporeal printing. Although ultraviolet light solidification is carried out at room temperature, it will cause potential harm to the biological body due to the ultraviolet light itself, and is also not suitable for intracorporeal printing. For the normal temperature natural solidification slurry, it will not cause potential harm to the biological body, and the semi-solid extrusion printing adopts a replaceable disposable needle cylinder, which can easily meet the medical device production standards. Therefore, the normal temperature natural solidification method makes the semi-solid extrusion printing technology a very potential biomedical device manufacturing technology. At present, some scientists have used semi-solid extrusion printing technology to flexibly construct various biomedical fiber devices, such as strain sensors, multi-channel neural probes, electrochemical biosensor arrays and biophysical sensor arrays.

[0003] Fiber devices are characterized by their extremely small diameter, which can achieve minimally invasive implantation in the biomedical field and probe the internal organs. However, single-component printing can only achieve the construction of simple fiber devices due to the single material, or requires pre-designed processes and different materials in different stages, which restricts the flexibility of device preparation. In contrast, a coaxial printing head can use different materials in its inner and outer channels, can release substances to specific receptors over time, improve mechanical properties, improve stability and biocompatibility, and greatly expand the combination and possibility of extrusion printing fibers, while simplifying the cumbersome workflow. Combined with the use of room temperature solidifiable slurry, semi-solid extrusion technology is a very promising technology for printing coaxial fibers.

[0004] Currently, the coaxial fiber is printed by semi-solid extrusion printing, and repeated experiments are needed before the experiment to find the most suitable combination of printing parameters according to the material properties. Coaxial printing needs two channels to print synchronously. Since the properties of the slurry used in the inner and outer channels are different, it is necessary to repeatedly adjust and try to optimize the combination of printing parameters to achieve the best printing effect. In summary, the way of determining the printing parameters by repeated experiments has the problems of complicated experiment, large workload and easy to introduce human error, and limits the printing efficiency and repeatability. In addition, for some slurry that solidifies at room temperature, the viscosity changes with time, but the existing fiber printing method ignores the change of the viscosity of the slurry that solidifies at room temperature with time, so the printing parameters do not change with the change of the viscosity of the slurry, which leads to the problems of smaller width and discontinuity of the fiber printing under the same extrusion force or air pressure. SUMMARY

[0005] The present application is used to solve the problems of complicated and large workload in determining the fiber printing parameters in the prior art, and the viscosity of the slurry of the material that solidifies at room temperature changes with time, which leads to the problems of width and discontinuity of the fiber printing.

[0006] To solve the above technical problems, the first aspect of the present application provides a coaxial fiber printing control method based on reinforcement learning, comprising:

[0007] receiving device parameters of a coaxial printing device and coaxial fiber images collected by a camera on the coaxial printing device;

[0008] identifying fiber features from the coaxial fiber images, wherein the fiber features include inner and outer fiber morphology features and fiber size features;

[0009] determining a current state from the fiber features and the device parameters, inputting the current state into a pre-trained expected return prediction model to obtain expected return values of each action, the expected return prediction model is part of reinforcement learning and is trained by an experience pool, and is used to predict the expected return of taking each action in a given state, each training sample in the experience pool includes: state s, action a, reward r and new state s', the action is a printing parameter, the reward is calculated by a reward function set according to the coaxial fiber printing requirements, and the expected return for training is calculated by using the reward in the training sample;

[0010] controlling the coaxial printing device to run by using the action with the maximum expected return value;

[0011] judging whether the print head of the coaxial printing device has run to the end point, if not, returning to execute the steps of receiving the device parameters and the coaxial fiber images and the subsequent steps, and if yes, closing the coaxial printing device.

[0012] As a further embodiment of the present application, the action control using the maximum expected return value further comprises:

[0013] Obtaining device parameters of the coaxial printing device and a coaxial fiber image collected by a camera on the coaxial printing device;

[0014] Identifying fiber features from the coaxial fiber image;

[0015] Determining the latest current state according to the identified fiber features and the device parameters;

[0016] Calculating a feedback reward value according to the latest current state and the return function;

[0017] Updating the expected return prediction model according to the feedback reward value.

[0018] As a further embodiment of the present application, identifying fiber features from the coaxial fiber image comprises:

[0019] Preprocessing the coaxial fiber image to obtain a binary image;

[0020] Extracting contour information from the binary image using a feature detection operator;

[0021] Extracting region boundaries from the contour information using a boundary tracking algorithm;

[0022] Determining fiber types of the region boundaries according to the inclusion relationship between the region boundaries, wherein the fiber types include inner layer fibers and outer layer fibers;

[0023] Obtaining inner layer fiber images and outer layer fiber images according to the region boundaries and their fiber types;

[0024] Analyzing the inner layer fiber images and the outer layer fiber images using Hough transform to obtain inner and outer layer fiber morphological features;

[0025] Fitting to obtain circular contours of the inner and outer layer fibers according to the inner and outer layer fiber morphological features;

[0026] Calculating fiber size features according to the circular contours of the inner and outer layer fibers.

[0027] As a further embodiment of the present application, preprocessing the coaxial fiber image to obtain a binary image comprises:

[0028] Performing adaptive threshold processing on the coaxial fiber image to obtain a first binary image;

[0029] Performing denoising processing on the first binary image to obtain a second binary image;

[0030] The second binary image is subjected to morphological opening and closing operations to obtain a binary image.

[0031] In further embodiments of the present application, the printing parameters include: a coaxial printing head moving speed, a coaxial printing head height, and an extrusion speed of each channel of the coaxial printing device to be executed in the next step.

[0032] The fiber size features include: an inner layer fiber diameter, an outer layer fiber diameter, and a fiber width.

[0033] The device parameters include: a coaxial printing head moving speed, a coaxial printing head height, and an extrusion speed of each channel of the coaxial printing device.

[0034] In further embodiments of the present application, after the coaxial printing device is controlled to act according to the printing parameters, the method further includes:

[0035] It is judged whether the sliding table moves to the limit switch of the end socket, and if so, the coaxial printing device is stopped.

[0036] In further embodiments of the present application, the reward function is set according to the coaxial fiber printing requirements, and the reward function includes:

[0037] According to the inner and outer layer fiber morphology features, a first reward term representing an inner layer fiber printing morphology reward and a second reward term representing an outer layer fiber printing morphology reward are set.

[0038] According to the change difference of the inner and outer channel printing fiber sizes, a penalty term is set.

[0039] The first reward term, the second reward term, and the penalty term are added to obtain the reward function.

[0040] In further embodiments of the present application, the first reward term is represented by the following formula:

[0041] R1(s, a) = α1·C1;

[0042] The second reward term is represented by the following formula:

[0043] R2(s, a) = α2·C2;

[0044] The penalty term is represented by the following formula:

[0045] P1(s, a) = -α3·abs(Δd 内 ) - α4·abs(Δd 外 );

[0046] Wherein, R1(s, a) is the first reward value; R2(s, a) is the second reward value; P1(s, a) is the penalty term; C1 is 1 when the inner layer fiber morphology is a continuous straight line, and -1 when the inner layer fiber morphology is a discontinuous straight line; C2 is 1 when the outer layer fiber morphology is a continuous straight line, and -1 when the outer layer fiber morphology is a discontinuous straight line; Δd 内 is the inner channel fiber diameter difference; Δd 外 is the outer channel fiber diameter difference; α1, α2, α3, α4 are hyperparameters.

[0047] As a further embodiment of the present application, the expected return prediction model is trained using a DQN network, the DQN network comprising a Q network and a target network with the same structure, the Q network and the target network comprising an input layer, a hidden layer and an output layer.

[0048] The input of the input layer is the state of a certain period, the activation function of the hidden layer is a ReLU function, the activation function of the output layer is a softmax function, and the output of the output layer is the expected return of each action.

[0049] As a further embodiment of the present application, the process of training the expected return prediction model using the DQN network comprises:

[0050] S1, initializing the Q network, the target network and the initial state, and setting the printing path;

[0051] S2, generating an experience pool:

[0052] Based on the current state s, an action a is selected using a preset strategy, and the coaxial printing device is controlled to run using the action a;

[0053] A new state s' is obtained by recognizing the fiber image collected by the camera on the coaxial printing device, the reward r after the action is calculated according to the reward function, the predicted Q value of each action is calculated using the Q network, and the action with the maximum Q value is selected as the new action a';

[0054] The state s, the action a, the reward r and the new state s' are stored as sample data in the experience pool;

[0055] It is judged whether the printing head on the coaxial printing device has run to the end of the path, if yes, the initial state is reset, and the S2 is repeatedly executed until a first preset number of times is reached, if not, the new state s' is updated as the state s, the new action a' is updated as the action a, the coaxial printing device is controlled to run using the action a, and the steps of recognizing the fiber image and the like are repeated;

[0056] S3, training network parameters:

[0057] Sample data is obtained from the experience pool, and the sample data is input into the Q network and the target network to obtain a predicted Q value and a target Q value represented as follows:

[0058] Q(s,a; θ Q ),

[0059]

[0060] wherein Q(s,a; θ Q ) is a predicted Q value of taking action a in state s, θ Q is a Q network parameter, y j is a target Q value, r j is a reward r obtained by taking action a in state s, γ is a decay rate, Q T (s′,a′; θ T ) is a Q value of each action a′ determined in state s′, max a′ Q T (s′,a′; θ T ) is a maximum Q value in state s′, a′ is each action taken in the future, θ T is a target Q network parameter;

[0061] According to the predicted Q value, the target Q value, and a Q network loss function, the weight of the Q network is updated;

[0062] S4, determining whether the number of times of updating the Q network reaches a second preset number of times, if not, repeating step S3, if yes, updating the Q network weight to the target weight;

[0063] S5, determining whether the number of cycles of steps S3 to S4 is greater than a preset threshold or whether a continuous M number of prints obtain a continuous uniform coaxial fiber, if not, repeating steps S3 to S4, if yes, saving the target network as a trained expected return prediction model.

[0064] As a further embodiment of the present application, updating the expected return prediction model according to the feedback reward value comprises:

[0065] According to the feedback reward value, calculating the target Q value of the target network;

[0066] According to the last identified fiber feature and the last action taken, determining the predicted Q value of the Q network;

[0067] According to the target Q value of the target network and the predicted Q value of the Q network, updating the loss function;

[0068] Using the loss function, updating the weight of the target network.

[0069] The second aspect of the present application provides a coaxial fiber printing control device based on reinforcement learning, comprising:

[0070] an interaction unit, configured to receive device parameters of the coaxial printing device and coaxial fiber images collected by a camera on the coaxial printing device;

[0071] an identification unit, configured to identify fiber features from the coaxial fiber images, wherein the fiber features include inner and outer layer fiber morphology features and fiber size features;

[0072] a prediction unit, configured to determine a current state from the fiber features and the device parameters, and input the current state into a pre-trained expected return prediction model to obtain expected return values of each action, the expected return prediction model being part of reinforcement learning and being trained by an experience pool, and being used to predict expected returns of each action taken in a given state, each training sample in the experience pool including: a state s, an action a, a reward r, and a new state s', the action being a printing parameter, the reward being calculated by a reward function set according to coaxial fiber printing requirements, and the expected return for training being calculated by using the reward in the training sample;

[0073] a first control unit, configured to control the coaxial printing device to run by using the action with the maximum expected return value;

[0074] a second control unit, configured to determine whether a print head of the coaxial printing device runs to an end point, if not, return to execute the steps of receiving the device parameters and the coaxial fiber images and the following steps, and if yes, shut down the coaxial printing device.

[0075] The third aspect of the present application provides a computer device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, and the processor implements the method of any one of the preceding embodiments when executing the computer program.

[0076] The fourth aspect of the present application provides a computer storage medium, which stores a computer program, and the computer program is executed by a processor of a computer device to implement the method of any one of the preceding embodiments.

[0077] The fifth aspect of the present application provides a computer program product, which includes a computer program, and the computer program is executed by a processor of a computer device to implement the method of any one of the preceding embodiments.

[0078] The coaxial fiber printing control method and device based on reinforcement learning provided in the application can solve the problems of complex printing parameter adjustment and machine unable to adaptively adjust when viscosity changes in semi-solid extrusion printing under normal temperature conditions. Specifically, a camera is installed to collect coaxial fiber images during the operation of the coaxial printing device, and fiber characteristics reflecting fiber phase state and fiber size changes are identified from the coaxial fiber images, so that the change of slurry viscosity can be indirectly monitored. The current state is determined from the fiber characteristics and the device parameters, and the current state is input into a pre-trained expected return prediction model to obtain the expected return value of each action. The action with the maximum expected return value is used to control the operation of the coaxial printing device. It is judged whether the print head of the coaxial printing device has reached the end point. If not, the steps of receiving the device parameters and the coaxial fiber images and the subsequent steps are returned to execute. If yes, the coaxial printing device is turned off. The neural network is part of the reinforcement learning algorithm, which is used to identify the state to obtain the expected return of each action, can simplify the search for printing parameters that meet the printing requirements, realize the real-time adjustment of the printing parameters of the inner and outer channels at the same time, reduce human errors, and the determination of the printing parameters also considers the fiber characteristics reflecting the change of the viscosity of the slurry of the normal temperature solidifiable material with time, and can also realize controllable and efficient printing of the normal temperature solidifiable biological slurry, and improve the printing quality.

[0079] In order to make the above and other objects, features and advantages of the present application more apparent, the following will describe a preferred embodiment in detail, and the accompanying drawings will be described as follows. BRIEF DESCRIPTION OF DRAWINGS

[0080] In order to more clearly illustrate the technical solutions in the embodiments of the present application or the prior art, the following will briefly introduce the drawings needed to be used in the embodiments or prior art description. Obviously, the drawings in the following description are only some embodiments of the present application, and for those skilled in the art, other drawings can also be obtained without creative labor.

[0081] Figure 1 The schematic diagram of the coaxial fiber printing control system based on reinforcement learning in the embodiment of the present application is shown;

[0082] Figure 2 The structural diagram of the coaxial printing device in the embodiment of the present application is shown;

[0083] Figure 3 The flowchart of the return function configuration process in the present application is shown;

[0084] Figure 4 The flowchart of the process of establishing an expected return prediction model based on reinforcement learning by the server in the embodiment of the present application is shown;

[0085] Figure 5A first flowchart of a coaxial fiber printing control method based on reinforcement learning of an embodiment of the present application is shown;

[0086] Figure 6 A second flowchart of a coaxial fiber printing control method based on reinforcement learning of an embodiment of the present application is shown;

[0087] Figure 7 A flowchart of a process of identifying a coaxial fiber image of an embodiment of the present application is shown;

[0088] Figure 8 A structure diagram of a coaxial fiber printing control device based on reinforcement learning of an embodiment of the present application is shown;

[0089] Figure 9 A structure diagram of a computer device of an embodiment of the present application is shown.

[0090] Explanation of the drawing symbols:

[0091] 101, coaxial printing device;

[0092] 102, client;

[0093] 103, server;

[0094] 801, interaction unit;

[0095] 802, identification unit;

[0096] 803, prediction unit;

[0097] 804, first control unit;

[0098] 805, second control unit;

[0099] 902, computer device;

[0100] 904, processor;

[0101] 906, memory;

[0102] 908, driving mechanism;

[0103] 910, input / output module;

[0104] 912, input device;

[0105] 914, output device;

[0106] 916, presentation device;

[0107] 918, graphical user interface;

[0108] 920, network interface;

[0109] 922、communication link;

[0110] 924、communication bus. DETAILED DESCRIPTION

[0111] The technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the drawings in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work are within the scope of protection of the present application.

[0112] It should be noted that the terms "first", "second", and the like in the specification and claims of the present application and the above-described drawings are used to distinguish similar objects, and do not necessarily indicate a specific order or a chronological sequence. It should be understood that the data thus used can be interchanged under appropriate circumstances, so that the embodiments of the present application described herein can be implemented in an order other than that illustrated or described herein. In addition, the terms "include" and "have" and any variations thereof are intended to cover non-exclusive inclusion, for example, a process, method, device, product, or apparatus including a series of steps or units does not have to be limited to only those steps or units clearly listed, but can include other steps or units not clearly listed or inherent to the process, method, product, or apparatus.

[0113] The present specification provides method operation steps as described in the embodiments or flowcharts, but can include more or fewer operation steps based on routine or non-creative work. The order of steps listed in the embodiments is only one of the many step execution orders, and does not represent the only execution order. In actual system or device product execution, the method order shown in the embodiments or drawings can be executed in sequence or in parallel.

[0114] It should be noted that the coaxial fiber printing control method and device based on reinforcement learning of the present application can be used in the field of medical materials, such as printing biomedical electronic devices, such as strain sensors, multi-channel neural probes, electrochemical biosensor arrays, and biophysical sensor arrays, and can also be used in any field other than medical materials, and the application field of the coaxial fiber printing control method and device based on reinforcement learning of the present application is not limited.

[0115] The existing coaxial printing device based on semi-solid extrusion printing technology cannot monitor the slurry viscosity in real time. Even some conventional test methods such as the falling ball viscometer and the rotary viscometer cannot be well integrated into the coaxial printing equipment to realize rapid and continuous monitoring of the slurry viscosity. Therefore, the existing coaxial fiber printing has the problems of small width and discontinuity, and is not suitable for scenarios with high requirements for printing results, such as medical material field, thereby limiting the application range of semi-solid extrusion printing technology. Moreover, the printing parameters of the existing coaxial printing device based on semi-solid extrusion printing technology need to be repeatedly tested according to the material properties, and then the most suitable printing parameter combination is searched out. This kind of way has the problems of large workload and non-adaptability. In subsequent embodiments, unless otherwise specified, the coaxial printing device refers to a printing device using semi-solid extrusion printing technology. The room temperature curing slurry used by the coaxial printing device includes but is not limited to: platinum silica gel, platinum silica gel modified, and their composite materials mixed with other organic and inorganic materials; protein-based materials include silk protein, collagen, gelatin, etc., and their composites with other materials.

[0116] To solve the above technical problems, in an embodiment of the present application, a coaxial fiber printing control system based on reinforcement learning is provided, as shown in Figure 1 The coaxial printing device 101, the client 102 and the server 103. In specific implementation, the coaxial printing device 101, the client 102 and the server 103 can be integrated into one, thereby realizing the functions of the integrated parts.

[0117] A camera (the performance of the camera can be determined according to the clarity of the collected image) is arranged beside the coaxial printing head of the coaxial printing device 101, which is used for real-time detection of the coaxial fiber printing process and sends the coaxial fiber image to the server 103. In some embodiments, as shown in Figure 2As shown, the coaxial printing device 101 includes a main control board and a 5V power supply, a camera, a rotary ring encoder, a stepper motor driver, an LED display screen, a limit switch, and a communication module connected to the main control board. The stepper motor driver is connected to the stepper motor and is powered by 24V DC, used to drive the stepper motor to work according to the control of the main control board. Specifically, the main control board sends pulse signals to the stepper motor driver to control the speed and direction of the stepper motor. The rotary encoder is used to control functions and adjust parameters. The OLED display is used to display the working state and function selection interface in real time, wherein the working state includes the current fiber characteristics and device parameters, and the working selection interface is used for the user to configure the initial extrusion speed, select the printing head moving speed, and control the running end, etc. The limit switch is placed in the end holder of the coaxial printing device, used to prevent the sliding table from moving beyond the safe range. The communication module communicates with the server 103, used to receive the printing parameters determined by the server 103, and send the printing parameters to the main control board to adjust the moving speed of the coaxial printing head by the main control board. The camera is placed in the end holder, used to monitor the fiber printing in real time, and the coaxial fiber image collected by the communication module is sent to the server 103 for analysis of the coaxial fiber image by the server 103, and the printing parameters are determined based on the analysis result.

[0118] The client 102 is used for the user to set the configuration information of the reinforcement learning algorithm, and send the configuration information related to the coaxial printing device 101 to the coaxial printing device 101, and send the configuration information related to the server 103 to the server 103. Specifically, it includes:

[0119] (1) Configure the state space S, including: the inner and outer fiber morphology C, the fiber size characteristics (the inner and outer diameters d 内 , d 外 and the width) of the coaxial printing head, the height h of the coaxial printing head, the inner and outer layer extrusion speeds v 内 , v 外 and the printing head moving speed v 移 .

[0120] The state in the state space satisfies the Markov property, that is, the transition of the next state only depends on the current state and the action taken.

[0121] (2) Configure the action space including: adjusting the height h of the coaxial printing head, the inner and outer layer extrusion speeds v 内 , v 外 and the printing head moving speed v 移 . Ensure that the action space can cover all possible adjustments in the printing process, and each action is feasible. The actions that can be taken are: h±2cm, v 内 ±0.1mm / s, v 外 ±0.1mm / s, v 移±1mm / s.

[0122] (3) Set the reward function R(s, a): The goal of the present application is to achieve continuous and uniform straight line printing of coaxial fibers, so when the fiber morphology meets the continuous and uniform straight line, a reward will be obtained, otherwise a penalty will be given.

[0123] In an embodiment, as shown in Figure 3 the reward function is set according to the printing requirements of coaxial fibers, which includes:

[0124] Step 301, according to the morphological characteristics of the inner and outer fibers, a first reward term representing the reward of the printing morphology of the inner fibers and a second reward term representing the reward of the printing morphology of the outer fibers are set.

[0125] Specifically, the first reward term can be represented by the following formula:

[0126] R1(s, a) = α1·C1;

[0127] Wherein, R1(s, a) is the first reward value; C1 is the expected result when the inner fiber morphology is a continuous straight line, which can be taken as 1 here, and is an unexpected result when the inner fiber morphology is a non-continuous straight line, which can be taken as -1; α1 is a hyperparameter greater than 0, which can be set according to actual requirements, and the present application does not limit the value range thereof.

[0128] Specifically, the second reward term is represented by the following formula:

[0129] R2(s, a) = α2·C2;

[0130] Wherein, R2(s, a) is the second reward value; C2 is the expected result when the outer fiber morphology is a continuous straight line, which can be taken as 1 here, and is an unexpected result when the outer fiber morphology is a non-continuous straight line, which can be taken as -1; α2 is a hyperparameter greater than 0, which can be set according to actual requirements, and the present application does not limit the value range thereof.

[0131] Step 302, a penalty term is set according to the change difference of the printing fiber size of the inner and outer channels.

[0132] In actual printing, we hope that the change difference of the printing fiber size is smaller, therefore, the greater the change difference, the more the corresponding penalty, based on this, the penalty term can be represented by the following formula:

[0133] P1(s, a) = -α3·abs(Δd 内 )-α4·abs(Δd 外 );

[0134] Wherein, P1(s, a) is the penalty term; Δd 内is the inner channel fiber diameter difference; ad 外 is the outer channel fiber diameter difference; a3, a4 are super parameters greater than 0, which can be set according to actual requirements, and the present application does not limit the value range thereof.

[0135] Step 303, the first reward item, the second reward item and the penalty item are added to obtain the return function.

[0136] Specifically, the return function can be expressed as follows:

[0137] R(s, a) = R1+ R2+ P1

[0138] (4) define the expected return prediction model structure, the expected return prediction model is used to approximate the Q-value function, and is used to predict the expected return (i.e. Q value) of each action in a given state.

[0139] Specifically, the expected return prediction model structure includes the number of neurons and the connection mode of the input layer, the hidden layer and the output layer. In some embodiments, the expected return prediction model can be trained by using a DQN network, and the DQN network includes a Q network and a target network with the same structure. The DQN can calculate the Q value of different states and different behaviors under a complex model, and can quickly learn from feedback to adjust its own decision-making. In other embodiments, the expected return prediction model can also be trained by using other deep learning algorithms. Specifically, the other models include improved algorithms based on DQN, including Multi-step DQN, Dueling DQN and Double Q-learning.

[0140] In a specific embodiment, the number of neurons from the input layer to the output layer is 7, 25, 90 and 16 respectively, the input of the input layer is the state of a certain period, the activation function of the hidden layer is the ReLU function (f(x) = max(0, x)), and the activation function of the output layer is the softmax function The output of the output layer is the expected return of each action. The actions include the height of the print head, the speed of the print head, the extrusion speed of channel one and the extrusion speed of channel two. The action with the highest expected return is selected from the four actions to control the coaxial printing device.

[0141] (5) configure the initialization neural network parameters during model training. Taking the DQN network training the expected return prediction model as an example, the initialized parameters include the weights in the Q network and the target network, the starting device, and the setting of the printing path.

[0142] In some embodiments of the present disclosure, the client 102 can be a desktop computer, a tablet computer, a notebook computer, a smart phone, a digital assistant, a smart wearable device, etc. The smart wearable device can include a smart bracelet, a smart watch, smart glasses, a smart helmet, etc. Of course, the client is not limited to the above-mentioned electronic devices with certain entities, and can also be software running in the above-mentioned electronic devices.

[0143] On the basis of completing the above information configuration, the server 103 can train the expected return prediction model.

[0144] The server 103 is configured to pre-train an expected return prediction model, and control the coaxial fiber printing based on the expected return prediction model. The control process includes: receiving device parameters of the coaxial printing device and coaxial fiber images collected by a camera on the coaxial printing device; identifying the coaxial fiber images to obtain fiber features, wherein the fiber features include inner and outer fiber morphology features and fiber size features; determining a current state from the fiber features and the device parameters, inputting the current state into the pre-trained expected return prediction model to obtain expected return values of each action; controlling the coaxial printing device to run by using the action with the maximum expected return value; determining whether the print head of the coaxial printing device has run to the end point, if not, returning to execute the steps of receiving the device parameters and the coaxial fiber images and the subsequent steps, and if yes, closing the coaxial printing device.

[0145] Further, in order to ensure the continuous accuracy of the expected return prediction model, the server 103 can continuously learn and improve the accuracy of the model after controlling the coaxial printing device to run by using the action with the maximum expected return value. Specifically, the learning process includes: obtaining device parameters of the coaxial printing device and coaxial fiber images collected by a camera on the coaxial printing device; identifying the coaxial fiber images to obtain fiber features; determining a latest current state according to the fiber features identified this time and the device parameters; calculating a feedback reward value according to the latest current state and a return function; and updating the expected return prediction model according to the feedback reward value.

[0146] The embodiment realizes real-time controllable adjustment of printing parameters by real-time monitoring of fiber features in the printing process through visual information and combining the self-adaptive learning ability of the neural network, thereby saving operation time and improving the printing quality of the semi-solid extrusion printed coaxial fiber. Continuous learning can also improve the accuracy of the expected return prediction model. In summary, the embodiment can improve the printing efficiency and quality, and also increase the adaptability of the system to complex printing environments.

[0147] In an embodiment of the present application, the DQN algorithm is selected to learn the complex nonlinear function relationship between the rheological properties of the slurry, the printing parameters of the device and the printing effect in the printing process, and the expected return prediction model is obtained. Specifically, as shown in FIG. 4, the DQN algorithm is used to learn the complex nonlinear function relationship between the rheological properties of the slurry, the printing parameters of the device and the printing effect in the printing process, and the expected return prediction model is obtained. Figure 4As shown, the training process using a DQN network includes:

[0148] Step S1: Initialize the Q network, target network, and initial states, and set the print path. During this step, a random initialization method (such as Xavier initialization) can be used to initialize the Q network weights θ. Q And set the weights θ of the target network. T =θ Q The initial state can also be set randomly. The print path can be designed according to requirements, such as a 20cm straight line.

[0149] Step S2: Generate the experience pool. This step can be implemented using the experience replay method to construct an experience pool of fixed length containing sufficiently diverse samples to address the problem of excessively high sample correlation.

[0150] Specifically, step S2 includes steps S21 to S24.

[0151] Step S21: Based on the current state s, select action a using a preset strategy, and use action a to control the operation of the coaxial printing device. In this step, an ε-greedy strategy is used, selecting action a with a probability ε = 0.7. The ε-greedy strategy balances utilization and exploration; the part that selects the action with the highest expected return is the utilization part, while there is still a probability of searching for the global optimal solution, which is the exploration part.

[0152] Step S22: Identify the fiber image captured by the camera on the coaxial printing device to obtain a new state s', calculate the reward r after the action according to the reward function, calculate the predicted Q value of each action using the Q network, and select the action with the largest Q value as the new action a'.

[0153] Step S23: Store state s, action a, reward r, and new state s' as a sample data in the experience pool.

[0154] Step S24: Identify and determine whether the coaxial print head on the coaxial printing device has reached the end of the path. If yes, generate the next batch of data, reset the starting state, and repeat step S2 until the first preset number of times is reached. If no, update the new state s' to state s, update the new action a' to action a, and use action a to control the operation of the coaxial printing device, repeating the identification of fiber images and subsequent steps.

[0155] In this step, the first preset number of times can be adjusted according to the subsequent training situation. In some implementations, the first preset number of times is, for example, thirty times.

[0156] Step S3, train the network parameters, specifically including the following steps S31 to S32.

[0157] Step S31, sample data is obtained from the experience pool, and the sample data is input into the Q network and the target network to obtain a predicted Q value and a target Q value represented as follows, respectively:

[0158] Q(s,a; θ Q ),

[0159]

[0160] wherein Q(s,a; θ Q ) is a predicted Q value of taking action a in state s, θ Q is a Q network parameter, y j is a target Q value, r j is a reward r obtained by taking action a in state s, γ is a decay rate, Q T (s',a'; θ T ) is a Q value of each action a' determined in state s', max a′ Q T (s',a'; θ T ) is a maximum Q value in state s', a' is each action to be taken in the future, θ T is a target Q network parameter. The decay rate is a preconfigured quantity.

[0161] In this step, a batch of samples is randomly obtained from the experience pool, and the size is N, wherein N is a positive integer.

[0162] Step S32, the weight of the Q network is updated according to the predicted Q value, the target Q value and the Q network loss function.

[0163] In detail, the Q network loss function is the root mean square error of the target Q value of the target network and the predicted Q value of the Q network, and the calculation formula is as follows:

[0164] L(w) =∑(y j -Q(s,a; θ Q )) / N 2 .

[0165] wherein L(w) is a loss function, y j is a target Q value, and Q(s,a; θ Q ) is a predicted Q value.

[0166] In a specific implementation, a stochastic gradient descent method can be used to solve the new weight θ Q of the Q network.

[0167] Step S4, it is judged whether the number of Q network updates reaches a second preset number, if not, S3 is repeatedly executed, and if yes, the Q network weight is updated to a target weight.

[0168] The second preset number of times in this step can be set according to actual needs. By fixing the parameters in the target network within the second preset number of times, the purpose of stabilizing the learning target can be achieved.

[0169] Step S5, whether the number of cycles of steps S3 to S4 is greater than a preset threshold or whether continuous M times of printing obtain continuous uniform coaxial fibers, if not, repeat steps S3 to S4, if yes, save the target network as a trained expected return prediction model. Wherein, M is a positive integer.

[0170] In this step, the preset threshold and M can be set and adjusted according to actual needs. In some embodiments, the preset threshold is, for example, 200, and M is, for example, 50.

[0171] Through the setting of step S5, the training accuracy of the expected return prediction model can be improved.

[0172] After the expected return prediction model is trained, it can be applied to the coaxial fiber printing control. In an embodiment of the present application, as shown in Figure 5 The coaxial fiber printing control method based on reinforcement learning includes:

[0173] Step 501, receiving device parameters of a coaxial printing device and coaxial fiber images collected by a camera on the coaxial printing device. In detail, the device parameters include: the current coaxial printing head moving speed of the coaxial printing device, the coaxial printing head height, and the extrusion speed of each channel.

[0174] Step 502, identifying the coaxial fiber images to obtain fiber features, wherein the fiber features include inner and outer fiber morphology features and fiber size features. In detail, the inner and outer fiber morphology features are used to indicate whether the outer fiber morphology is a straight line or a non-straight line (curve or broken line), and whether the inner fiber morphology is a straight line or a non-straight line. The fiber size features include: inner fiber diameter, outer fiber diameter, and fiber width.

[0175] Step 503, determining the current state from the fiber features and device parameters, and inputting the current state into a pre-trained expected return prediction model to obtain the expected return value of each action.

[0176] In this step, the fiber features and device parameters together constitute the current state, and the current state is processed into the format required by the expected return prediction model and then input into the expected return prediction model.

[0177] The expected return prediction model is part of reinforcement learning and is trained using an experience pool, and is used to predict the expected return of each action in a given state. The expected return prediction model training process can refer to the foregoing embodiments, which will not be described in detail here.

[0178] The action is a printing parameter, including: the coaxial printing device next step to be executed by the coaxial printing head moving speed, coaxial printing head height, each channel extrusion speed.

[0179] Step 504, the action control coaxial printing device running with the maximum expected return value. In detail, the maximum overdue return value of the action indicates that the action can make the coaxial printing device obtain a large expected return, and then make the printing effect meet the demand.

[0180] Step 505, determine whether the printing head of the coaxial printing device runs to the end, if not, return to step 501 and continue to execute, if yes, turn off the coaxial printing device.

[0181] In this step, the end point is pre-configured information for indicating whether to stop printing. When the printing head of the coaxial printing device does not run to the end point, it means that the printing is not completed, and then returns to step 501 and continues to execute. When the printing head of the coaxial printing device runs to the end point, it means that the printing is completed, and the coaxial printing device is related.

[0182] In this embodiment, the coaxial fiber image is collected by the camera during the operation of the coaxial printing device, the fiber features reflecting the fiber phase state and the fiber size change are determined by identifying the coaxial fiber image, so that the change of the slurry viscosity can be indirectly monitored. The expected return prediction model can simplify the search of the printing parameters meeting the printing demand, realize the real-time adjustment of the printing parameters of the two channels, reduce the human error, and the determination of the printing parameters considers the fiber features reflecting the change of the viscosity of the slurry of the room temperature curable material with time, and the controllable and efficient printing of the room temperature curable biological slurry can be realized, and the printing quality is improved.

[0183] In further embodiments, as shown in Figure 6 Step 504, the action control coaxial printing device running with the maximum expected return value. In detail, the maximum overdue return value of the action indicates that the action can make the coaxial printing device obtain a large expected return, and then make the printing effect meet the demand.

[0184] Step 601, obtain the device parameters of the coaxial printing device and the coaxial fiber image collected by the camera on the coaxial printing device.

[0185] Step 602, identify the fiber features from the coaxial fiber image.

[0186] Step 603, determine the latest current state according to the fiber features identified this time and the device parameters. In this step, the fiber features identified this time are the fiber features identified in step 603, and the fiber features identified in step 603 and the device parameters together constitute the latest current state.

[0187] Step 604, calculate the feedback reward value according to the latest current state and the return function.

[0188] Step 605, updating the expected return prediction model according to the feedback reward value. Specifically, the implementation process of step 605 includes:

[0189] (1) First, calculate the target Q value of the target network according to the feedback reward value. The calculation process can refer to the calculation formula in the aforementioned step S31.

[0190] (2) Determine the predicted Q value of the Q network according to the last identified fiber feature and the last action taken.

[0191] In the implementation of this step, the last identified fiber feature and the obtained device parameters are input into the Q network to obtain the predicted Q value of each action, and the predicted Q value of the last action taken is selected to obtain the predicted Q value of the Q network.

[0192] (3) Update the loss function according to the target Q value of the target network and the predicted Q value of the Q network. The loss function in this step can refer to the calculation formula in the aforementioned step S32.

[0193] (4) Update the weight of the target network using the loss function. In the implementation of this step, the network weight can be solved using the stochastic gradient descent method.

[0194] This embodiment realizes real-time controllable adjustment of printing parameters by real-time monitoring of fiber features in the printing process through visual information and combining the adaptive learning ability of neural networks, thereby saving operation time and improving the printing quality of semi-solid extrusion printing coaxial fibers. Continuous learning can also improve the accuracy of the expected return prediction model and improve the prediction and decision-making ability of the model. In summary, this embodiment can improve printing efficiency and quality and also increase the adaptability of the system to complex printing environments.

[0195] In an embodiment of the present application, as shown in Figure 7 The above step 502 of identifying the fiber feature of the coaxial fiber image includes:

[0196] Step 701, pre-processing the coaxial fiber image to obtain a binary image.

[0197] The implementation process of this step includes: performing adaptive threshold processing on the coaxial fiber image to obtain a first binary image; performing denoising processing on the first binary image to obtain a second binary image; performing morphological opening operation and closing operation on the second binary image to obtain a binary image.

[0198] In the denoising processing, a filter (such as a Gaussian filter) is used.

[0199] Step 702, extracting contour information from the binary image using a feature detection operator.

[0200] The feature detection operator used in this step is, for example, Roberts Cross, Sobel or Canny edge detector. The contour information includes boundary pixel coordinate information.

[0201] In step 703, a boundary tracking algorithm is used to extract region boundaries from the contour information.

[0202] The boundary tracking algorithm used in this step is, for example, a contour finding algorithm, and the specific implementation steps of the algorithm can refer to the prior art.

[0203] In step 704, the fiber types of the region boundaries are determined according to the inclusion relationship between the region boundaries, wherein the fiber types include inner layer fibers and outer layer fibers.

[0204] In this step, if one boundary is contained in another boundary, it is determined that the former is an inner layer fiber and the latter is an outer layer fiber.

[0205] In step 705, inner layer fiber images and outer layer fiber images are obtained according to the region boundaries and their fiber types.

[0206] In the implementation of this step, the original fiber image can be divided into inner layer fiber images and outer layer fiber images through filling or masking operations between edges.

[0207] In step 706, Hough transform is used to analyze the inner layer fiber images and the outer layer fiber images to obtain the inner and outer layer fiber morphological features.

[0208] In this step, the specific implementation process of Hough Transform can refer to the prior art, which will not be described in detail here.

[0209] In step 707, the circular contours of the inner and outer layer fibers are fitted according to the inner and outer layer fiber morphological features.

[0210] In the implementation of this step, Hough Circle Transform can be used to fit the circular contours of the fibers.

[0211] In step 708, fiber size features are calculated according to the circular contours of the inner and outer layer fibers.

[0212] In the implementation of this step, the fiber width is calculated according to the fiber diameter d t-1 calculated from the previous fiber image and the fiber diameter d t calculated from the current fiber image, and the specific calculation formula is: Δd = d t -d t-1 , and Δd is the fiber width.

[0213] The fiber feature recognition method provided by the embodiment can improve the accuracy of fiber feature recognition.

[0214] In an embodiment of the present application, after the step 504 of controlling the coaxial printing device to act according to the printing parameters, the method further comprises:

[0215] The method further comprises the steps of: judging whether the sliding table moves to the limit switch at the end of the bracket, and stopping the operation of the coaxial printing device if yes. The printing head is fixed on the sliding table, and the extrusion of the printing head is controlled by the movement of the sliding table on the sliding rail.

[0216] The setting of the limit switch can make the coaxial printing device operate in a safe range.

[0217] Based on the same inventive concept, the present application also provides a coaxial fiber printing control device based on reinforcement learning, as described in the following embodiments. Since the principle of solving problems of the coaxial fiber printing control device based on reinforcement learning is similar to that of the coaxial fiber printing control method based on reinforcement learning, the implementation of the coaxial fiber printing control device based on reinforcement learning can be referred to the coaxial fiber printing control method based on reinforcement learning, and the repeated parts will not be described here.

[0218] Specifically, as shown in Figure 8 The coaxial fiber printing control device based on reinforcement learning comprises:

[0219] An interaction unit 801 is configured to receive device parameters of a coaxial printing device and coaxial fiber images collected by a camera on the coaxial printing device.

[0220] An identification unit 802 is configured to identify fiber features from the coaxial fiber images, wherein the fiber features include inner and outer fiber morphological features and fiber size features.

[0221] A prediction unit 803 is configured to determine a current state from the fiber features and the device parameters, input the current state into a pre-trained expected return prediction model to obtain expected return values of each action, and the expected return prediction model is part of reinforcement learning and is obtained by training using an experience pool, and is used to predict the expected return of taking each action in a given state. Each training sample in the experience pool comprises a state s, an action a, a reward r, and a new state s'. The action is a printing parameter, and the reward is calculated by using a return function set according to the coaxial fiber printing requirements. The expected return for training is calculated using the reward in the training sample.

[0222] A first control unit 804 is configured to control the operation of the coaxial printing device by using the action with the maximum expected return value.

[0223] The second control unit 805 is configured to determine whether the coaxial printing head of the coaxial printing device has reached the end point, and if not, return to execute the steps of receiving the device parameters and the coaxial fiber image and the subsequent steps, and if so, turn off the coaxial printing device.

[0224] The coaxial fiber printing control method and device based on reinforcement learning provided by the present application can achieve the following technical effects:

[0225] (1) Simplified search for optimal printing parameters: The present application introduces a closed-loop control system combined with an intelligent adjustment mechanism, greatly simplifying the process of finding the optimal printing parameters for the operator. The camera monitors the fiber printing situation in real time, and the monitoring results are fed back to the computer. The computer calculates the current optimal control parameters based on the previous training results and feeds them back to the control system of the coaxial printing device. This intelligent closed-loop control system greatly simplifies the manual search process for optimal parameters by the operator, making the printing process more intelligent and efficient.

[0226] (2) Visual feedback of printing situation: The present application monitors the fiber printing situation in real time through the camera, extracts the current coaxial fiber printing state through image processing and analysis, and extracts fiber characteristics such as the shape evolution and diameter change of the inner and outer fibers.

[0227] (3) Non-invasive real-time monitoring of printing state: The present application monitors the fiber printing situation in real time through the camera, avoiding the use of traditional monitoring methods that cannot monitor viscosity in real time or may contaminate the sample. This non-invasive monitoring method maintains the integrity of the printing process without direct contact with the printing material, thereby avoiding possible damage to the sample.

[0228] (4) Reinforcement learning closed-loop adjustment: Reinforcement learning algorithms can learn in complex situations and improve their choices. When the viscosity of the slurry changes, the reinforcement learning algorithm can handle uncertainty, maintain the stability of the printing system through learning and adjustment, and expand the application range of semi-solid extrusion printing technology.

[0229] (5) Realize uniform and continuous printing of slurry with changing rheological properties over time: The rheological properties of ambient-curable slurry change over time. In the present application, the printing situation can be adjusted in real time to adjust the working parameters of the printing device, realizing uniform and continuous printing of ambient-curable slurry.

[0230] (6) Feedback on the printability of the material: The printability of the slurry depends largely on the matching between the force provided by the printing device and the rheological properties of the slurry. Through the intelligent decision-making of the present application, the printability of the slurry can be monitored and evaluated in real time, providing a reference for adjusting the properties of the slurry.

[0231] In an embodiment of the present application, a computer device is provided, as shown in Figure 9 As shown, computer device 902 can include one or more processors 904, such as one or more central processing units (CPUs), each of which can implement one or more hardware threads. Computer device 902 can also include any memory 906 for storing any kind of information, such as code, settings, data, etc. Without limitation, for example, memory 906 can include any one or combination of: any type of RAM, any type of ROM, a flash memory device, a hard disk, an optical disk, etc. More generally, any memory can use any technology for storing information. Further, any memory can provide volatile or non-volatile retention of information. Further, any memory can represent a fixed or removable component of computer device 902. In one case, computer device 902 can perform any operation of the associated instructions when processor 904 executes the associated instructions stored in any memory or combination of memories. Computer device 902 also includes one or more drive mechanisms 908 for interacting with any memory, such as a hard disk drive mechanism, an optical disk drive mechanism, etc.

[0232] Computer device 902 can also include an input / output module 910 (I / O) for receiving various inputs (via input devices 912) and for providing various outputs (via output devices 914). One particular output mechanism can include a presentation device 916 and an associated graphical user interface 918 (GUI). In other embodiments, input / output module 910 (I / O), input devices 912, and output devices 914 can also not be included, merely serving as one computer device in a network. Computer device 902 can also include one or more network interfaces 920 for exchanging data with other devices via one or more communication links 922. One or more communication buses 924 couple the above-described components together.

[0233] Communication links 922 can be implemented in any manner, such as through a local area network, a wide area network (e.g., the Internet), a point-to-point connection, etc., or any combination thereof. Communication links 922 can include any combination of hardwired links, wireless links, routers, gateway functionality, name servers, etc., governed by any protocol or combination of protocols.

[0234] The embodiment of the present application further provides a computer readable storage medium, and the computer readable storage medium stores a computer program, and the computer program is executed by a processor to perform the steps of the method.

[0235] The embodiment of the present application further provides a computer readable instruction, and when the processor executes the instruction, the program in the instruction makes the processor execute the method of any one of the preceding embodiments.

[0236] The embodiment of the present application further provides a computer program product, which comprises a computer program, and the computer program is executed by a processor of a computer device to realize the method.

[0237] It should be understood that the size of the sequence number of the above processes does not mean the order of execution in various embodiments of the present application, and the execution order of the processes should be determined according to its function and inherent logic, and should not constitute any limitation on the implementation process of the embodiments of the present application.

[0238] It should also be understood that in the embodiments of the present application, the term "and / or" only describes the association relationship of the associated objects, and can exist in three relationships. For example, A and / or B can represent three cases of A alone, A and B together, and B alone. In addition, the character " / " in the present application generally represents an "or" relationship between the associated objects before and after it.

[0239] Those skilled in the art can realize that the units and algorithm steps of the examples described in combination with the embodiments disclosed in the present application can be realized by electronic hardware, computer software or a combination of both. In order to clearly illustrate the interchangeability of hardware and software, the components and steps of the examples have been described in general terms in the above description. Whether the functions are performed in hardware or software depends on the specific application and design constraints of the technical solution. A person skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of the present application.

[0240] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working process of the system, device and unit described above can refer to the corresponding process in the foregoing method embodiments, which will not be repeated here.

[0241] In several embodiments provided in the present application, it should be understood that the disclosed system, device and method can be implemented in other ways. For example, the device embodiments described above are only schematic, and the division of the units is only a logical function division, and actual implementation can have another division manner, for example, a plurality of units or components can be combined or integrated into another system, or some features can be ignored or not executed. In addition, the coupling or direct coupling or communication connection between the units shown or discussed can be indirect coupling or communication connection through some interfaces, devices or units, and can also be electrical, mechanical or other forms of connection.

[0242] The units described as separate components may or may not be physically separate, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Part or all of the units can be selected according to actual needs to achieve the purpose of the embodiments of the application.

[0243] In addition, the functional units in each embodiment of the application can be integrated in one processing unit, or each unit can exist physically, or two or more units can be integrated in one unit. The integrated unit can be realized in the form of hardware or in the form of a software functional unit.

[0244] The integrated unit, if realized in the form of a software functional unit and sold or used as an independent product, can be stored in a computer readable storage medium. Based on such understanding, the technical solutions of the application essentially or the part of the prior art that contributes to the technical solutions, or all or part of the technical solutions can be embodied in the form of a software product. The computer software product is stored in a storage medium, including a plurality of instructions for causing a computer device (which can be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the method described in each embodiment of the application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various program code storage media.

[0245] The principles and implementation manners of the application are described in the specific embodiments in the application, and the above embodiment descriptions are only used to help understand the method of the application and its core idea; at the same time, for those skilled in the art, according to the idea of the application, the specific implementation manner and application range will be changed, and the above description of the specification should not be understood as a limitation of the application.

Claims

1. A coaxial fiber printing control method based on reinforcement learning, characterized in that, include: Receives equipment parameters of the coaxial printing device and coaxial fiber images captured by the camera on the coaxial printing device; The fiber features are obtained by identifying the coaxial fiber image, wherein the fiber features include the morphological features of the inner and outer layers of fibers and the fiber size features; The current state is determined by the fiber features and the device parameters. The current state is then input into a pre-trained expected reward prediction model to obtain the expected reward value for each action. The expected reward prediction model is part of reinforcement learning and is trained using an experience pool. It is used to predict the expected reward for taking each action in a given state. Each training sample in the experience pool includes: state s, action a, reward r, and new state s'. The action is the printing parameter. The reward is calculated using a reward function set according to the coaxial fiber printing requirements. The expected reward for training is calculated using the rewards in the training samples. The coaxial printing unit is controlled to operate using actions that maximize the expected return value. Determine whether the print head of the coaxial printing device has reached the end point. If not, return to the steps of receiving device parameters and coaxial fiber image and subsequent steps. If yes, shut down the coaxial printing device. The printing parameters include: the coaxial print head moving speed, coaxial print head height, and extrusion speed of each channel to be executed in the next step of the coaxial printing device. The fiber size characteristics include: inner fiber diameter, outer fiber diameter, and fiber width; The equipment parameters include: the current coaxial printhead moving speed, coaxial printhead height, and extrusion speed of each channel of the coaxial printing device.

2. The method as described in claim 1, characterized in that, After utilizing the motion control of the coaxial printing unit to achieve the maximum expected return, the following is also included: Obtain the equipment parameters of the coaxial printing device and the coaxial fiber images captured by the camera on the coaxial printing device; Fiber features are obtained by identifying the coaxial fiber image; Based on the fiber characteristics and equipment parameters identified in this study, the latest current status is determined; Calculate the reward value based on the latest current status and reward function; The expected return prediction model is updated based on the feedback reward value.

3. The method as described in claim 1, characterized in that, Identifying the fiber features obtained from the coaxial fiber image includes: The coaxial fiber image is preprocessed to obtain a binary image; Contour information is extracted from the binary image using a feature detection operator; The region boundary is extracted from the contour information using a boundary tracking algorithm; The fiber type of the region boundary is determined based on the inclusion relationship between the region boundaries, wherein the fiber type includes inner layer fibers and outer layer fibers; Based on the region boundaries and their fiber types, inner fiber images and outer fiber images are obtained; The inner and outer fiber images were analyzed using Hough transform to obtain the morphological characteristics of the inner and outer fibers. Based on the morphological characteristics of the inner and outer fiber layers, a circular outline of the inner and outer fiber layers is obtained by fitting. Calculate fiber size characteristics based on the circular outlines of the inner and outer fibers.

4. The method as described in claim 3, characterized in that, Preprocessing the coaxial fiber image to obtain a binary image includes: The coaxial fiber image is subjected to adaptive thresholding to obtain a first binary image; The first binary image is denoised to obtain the second binary image. The second binary image is subjected to morphological opening and closing operations to obtain a binary image.

5. The method as described in claim 1, characterized in that, After controlling the coaxial printing device's operation according to the printing parameters, the following is also included: Determine whether the slide has moved to the limit switch of the end slot. If so, stop the operation of the coaxial printing device.

6. The method as described in claim 1, characterized in that, The reward function is set according to the coaxial fiber printing requirements, including: Based on the morphological characteristics of the inner and outer fiber layers, a first reward item representing the printing morphology reward of the inner fiber layer and a second reward item representing the printing morphology reward of the outer fiber layer are set. Set a penalty item based on the difference in fiber size between the inner and outer channels; The reward function is obtained by summing the first reward, the second reward, and the penalty.

7. The method as described in claim 6, characterized in that, The first reward item is represented by the following formula: ； The second reward item is represented by the following formula: ； The penalty item is represented by the following formula: )； in, The first reward value; This is the second reward value; This is a penalty item; The value is 1 when the inner fiber morphology is a continuous straight line, and -1 when the inner fiber morphology is a discontinuous straight line. The value is 1 when the outer fiber morphology is a continuous straight line, and -1 when the outer fiber morphology is a discontinuous straight line. The difference in fiber diameter within the inner channel; The difference in fiber diameter in the outer channel; This is a hyperparameter.

8. The method as described in claim 1, characterized in that, The expected return prediction model is trained using a DQN network, which includes a Q network and a target network with identical structures. The Q network and the target network include an input layer, a hidden layer, and an output layer. The input of the input layer is the state at a certain time period, the activation function of the hidden layer is the ReLU function, the activation function of the output layer is the softmax function, and the output of the output layer is the expected reward for each action.

9. The method as described in claim 8, characterized in that, The process of training an expected return prediction model using a DQN network includes: S1, initialize the Q network, target network and initial state, and set the print path; S2, generate the experience pool: Based on the current state s, select action a using a preset strategy, and use action a to control the operation of the coaxial printing device. The new state s' is obtained by recognizing the fiber image captured by the camera on the coaxial printing device. The reward r after the action is calculated according to the reward function. The predicted Q value of each action is calculated using the Q network and the action with the largest Q value is selected as the new action a'. Store state s, action a, reward r, and new state s' as a sample data in the experience pool; Determine whether the coaxial print head on the coaxial printing device has reached the end of the path. If yes, reset the starting state and repeat S2 until the first preset number of times is reached. If no, update the new state s' to state s and the new action a' to action a. Use action a to control the operation of the coaxial printing device and repeat the steps of identifying the fiber image and subsequent steps. S3, Training network parameters: Sample data is obtained from the experience pool and input into the Q-network and the target network, respectively, to obtain the predicted Q-value and the target Q-value as shown below: ， in, Let be the predicted Q-value for taking action a in state s, and be the Q-network parameters. For the target Q value, Let r be the reward r obtained by taking action a in state s. The attenuation rate, In the state The determined actions Q value, In the state The maximum Q value, The actions to be taken for the future, The target Q network parameters; Update the weights of the Q-network based on the predicted Q-value, the target Q-value, and the Q-network loss function; S4, determine whether the number of Q network updates has reached the second preset number. If not, repeat S3. If yes, update the Q network weights to the target weights. S5, determine whether the number of iterations of S3 to S4 is greater than a preset threshold or whether continuous and uniform coaxial fibers are obtained by printing M times. If not, repeat S3 to S4. If yes, save the target network as a trained expected return prediction model.

10. The method as described in claim 2, characterized in that, Updating the expected return prediction model based on the feedback reward value includes: Calculate the target Q-value of the target network based on the feedback reward value; The predicted Q value of the Q network is determined based on the fiber features identified in the previous test and the actions taken in the previous test. Update the loss function based on the target Q-value of the target network and the predicted Q-value of the Q-network; The weights of the target network are updated using the loss function.

11. A coaxial fiber printing control device based on reinforcement learning, characterized in that, include: The interaction unit is used to receive equipment parameters of the coaxial printing device and coaxial fiber images captured by the camera on the coaxial printing device. The identification unit is used to identify the coaxial fiber image to obtain fiber features, wherein the fiber features include inner and outer layer fiber morphological features and fiber size features; The prediction unit is used to determine the current state based on the fiber features and the device parameters, and input the current state into a pre-trained expected reward prediction model to obtain the expected reward value of each action. The expected reward prediction model is part of reinforcement learning and is trained using an experience pool. It is used to predict the expected reward of taking each action in a given state. Each training sample in the experience pool includes: state s, action a, reward r, and new state s'. The action is the printing parameter, and the reward is calculated using a reward function set according to the coaxial fiber printing requirements. The expected reward for training is calculated using the rewards in the training samples. The first control unit is used to control the operation of the coaxial printing unit by utilizing the action with the maximum expected return value. The second control unit is used to determine whether the coaxial print head of the coaxial printing device has reached the end point. If not, it returns to the steps of receiving device parameters and coaxial fiber images and subsequent steps. If yes, it shuts down the coaxial printing device. The printing parameters include: the coaxial print head moving speed, coaxial print head height, and extrusion speed of each channel to be executed in the next step of the coaxial printing device. The fiber size characteristics include: inner fiber diameter, outer fiber diameter, and fiber width; The equipment parameters include: the current coaxial printhead moving speed, coaxial printhead height, and extrusion speed of each channel of the coaxial printing device.

12. A computer device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method according to any one of claims 1 to 10.

13. A computer storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor of a computer device, it implements the method according to any one of claims 1 to 10.

14. A computer program product, characterized in that, The computer program product includes a computer program that, when executed by a processor of a computer device, implements the method according to any one of claims 1 to 10.

Images