Decision-making method and device based on deep reinforcement learning, equipment and medium

Abstract

The invention provides a decision-making method, device and equipment based on deep reinforcement learning and a medium, and the method comprises the steps: obtaining multi-source data which comprises user blood glucose data, medication record data, user behavior data and user physiological data; a deep reinforcement learning model is called to carry out virtual simulation and medication decision making on the multi-source data, corresponding decision making information is output, the decision making information comprises medication suggestions and risk early warning, and the deep reinforcement learning model comprises a state space, an action space and a reward function. The state space is used for indicating feature data of the multi-source data, the action space is used for determining medication suggestions of the user, and the reward function is used for guiding and optimizing medication decisions of the deep reinforcement learning model. In this way, the technical problem that in the prior art, blood sugar control is not stable due to the fact that doctor experience or fixed dosage is relied on can be solved.

Description

Technical Field

[0001] This application relates to the field of medical artificial intelligence technology, specifically to a decision-making method, device, equipment, and medium based on deep reinforcement learning. Background Technology

[0002] With rapid socio-economic development, people's lifestyles are undergoing tremendous changes. Lifestyle is crucial to people's health; neglecting it can lead to various chronic diseases in the long run. Among them, diabetes is one of the most common chronic diseases.

[0003] Currently, most existing diabetes medication regimens rely on doctors' experience or fixed dosages, and cannot be adjusted in real time based on the patient's dynamic changes in blood glucose data, lifestyle habits, and other factors, leading to unstable blood glucose control. Summary of the Invention

[0004] In view of this, the embodiments of this application aim to provide a decision-making method, apparatus, device and medium based on deep reinforcement learning, which can solve the technical problem of unstable blood glucose control caused by reliance on doctor experience or fixed dosage in the prior art.

[0005] Firstly, this application provides a decision-making method based on deep reinforcement learning, including: Acquire multi-source data, including user blood glucose data, medication record data, user behavior data, and user physiological data, wherein the user blood glucose data is blood glucose sequence data recorded at different times; A deep reinforcement learning model is invoked to perform virtual simulation and medication decision-making on the multi-source data, and output corresponding decision information, including medication recommendations and risk warnings. The deep reinforcement learning model includes a state space, an action space, and a reward function. The state space is used to indicate the feature data of the multi-source data, the action space is used to determine the medication recommendations for the user, and the reward function is used to guide and optimize the medication decision-making of the deep reinforcement learning model.

[0006] In some embodiments, the virtual simulation and medication decision-making based on the multi-source data, and the output of corresponding decision information, include: The multi-source data is preprocessed to obtain corresponding processed data. The preprocessing includes at least missing value processing, outlier filtering, and normalization processing. Based on the processed data, a corresponding state space and action space are constructed; Based on a preset reward function, simulation calculations and medication decisions are performed on the state space and the action space to obtain the decision information.

[0007] In some embodiments, the reward function is associated with at least the user's blood glucose data and the medication record data.

[0008] In some embodiments, the deep reinforcement learning model includes an Actor-Critic network, and the simulation calculation and medication decision-making for the state space and the action space include: Based on the Actor-Critic network, action evaluation and medication decisions are performed on the state space and the action space.

[0009] In some embodiments, the deep reinforcement learning model includes a state encoder, a data embedding layer, a feature fusion layer, an Actor-Critic network, and an action constraint module. The state space includes temporal and non-temporal data. The action evaluation and medication decision based on the Actor-Critic network of the state space and the action space includes: Based on the state encoder, feature extraction is performed on the temporal data in the state space to obtain the corresponding temporal feature data; Based on the data embedding layer, feature extraction is performed on the non-temporal data in the state space to obtain the corresponding non-temporal feature data; Based on the feature fusion layer, the temporal feature data and the non-temporal feature data are fused to obtain the corresponding fused feature data; Based on the Actor-Critic network, the fused feature data is used to predict and evaluate medication actions in the action space to obtain corresponding decision actions, which include at least the medication recommendations. The decision action is modified based on the action constraint module to obtain the decision information.

[0010] In some embodiments, the medication record data includes at least insulin dose and compliance score, and the user physiological data includes at least complication markers.

[0011] In some embodiments, the reward function is: Among them, G target For the preset target blood glucose value, G t The current blood glucose value in the user's blood glucose data, ΔD represents the change in insulin dosage, and H represents the value at that time. compliance The compliance score is given.

[0012] Secondly, this application provides a decision-making device based on deep reinforcement learning, comprising: The acquisition module is used to acquire multi-source data, including user blood glucose data, medication record data, user behavior data, and user physiological data. The user blood glucose data is blood glucose sequence data recorded at different times. The processing module is used to call a deep reinforcement learning model to perform virtual simulation and medication decision-making on the multi-source data, and output corresponding decision information, including medication recommendations and risk warnings. The deep reinforcement learning model includes a state space, an action space, and a reward function. The state space is used to indicate the feature data of the multi-source data, the action space is used to determine the medication recommendations for the user, and the reward function is used to guide and optimize the medication decision-making of the deep reinforcement learning model.

[0013] For any content not introduced or described in the embodiments of this application, please refer to the relevant descriptions in the foregoing method embodiments; they will not be repeated here.

[0014] Thirdly, this application provides an electronic device, including: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement the steps of the above-described deep reinforcement learning-based decision-making method.

[0015] Fourthly, this application provides a computer-readable storage medium storing computer program instructions thereon, which, when executed by a processor, implement the steps of the decision-making method based on deep reinforcement learning described above.

[0016] The technical solution provided in this application embodiment can include the following beneficial effects: This application acquires multi-source data, including user blood glucose data, medication record data, user behavior data, and user physiological data. The user blood glucose data is blood glucose sequence data recorded at different times. A deep reinforcement learning model is invoked to perform virtual simulation and medication decision-making on the multi-source data, outputting corresponding decision information, including medication recommendations and risk warnings. The deep reinforcement learning model includes a state space, an action space, and a reward function. The state space is used to indicate the feature data of the multi-source data, the action space is used to determine the user's medication recommendations, and the reward function is used to guide and optimize the medication decision-making of the deep reinforcement learning model. In this way, a deep reinforcement learning model can be used to analyze blood glucose fluctuation trends and, combined with other multi-source user data, dynamically adjust and decide on corresponding medication recommendations and risk warnings, assisting doctors in treating diabetic patients better and more accurately, thereby improving the accuracy and reliability of diabetes medication decisions. It also solves the problems of unstable blood glucose control caused by reliance on doctor experience or fixed dosages in existing technologies.

[0017] The above description is only an overview of the technical solution of this application. In order to better understand the technical means of this application and to implement it in accordance with the contents of the specification, and to make the above and other objects, features and advantages of this application more obvious and understandable, the following are specific embodiments of this application. Attached Figure Description

[0018] Various other advantages and benefits will become apparent to those skilled in the art upon reading the following detailed description of preferred embodiments. The accompanying drawings are for illustrative purposes only and are not intended to limit the scope of this application. Furthermore, the same reference numerals denote the same parts throughout the drawings.

[0019] Figure 1 This is a flowchart illustrating a decision-making method based on deep reinforcement learning, provided as an embodiment of this application.

[0020] Figure 2 This is a schematic diagram of the structure of a deep reinforcement learning model provided in an embodiment of this application.

[0021] Figure 3 This is a schematic diagram of a decision-making device based on deep reinforcement learning, provided as an embodiment of this application.

[0022] Figure 4 This is a schematic diagram of another decision-making device based on deep reinforcement learning provided in an embodiment of this application.

[0023] Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0024] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0025] Unless otherwise defined, the technical or scientific terms used in the embodiments of this specification shall have the ordinary meaning understood by one of ordinary skill in the art to which this specification pertains. The terms "first," "second," and similar terms used in the embodiments of this specification do not indicate any order, quantity, or importance, but are merely used to avoid confusion of constituent elements.

[0026] Unless the context otherwise requires, throughout this specification, "a plurality of" means "at least two," and "including" is interpreted as open-ended or encompassing, that is, "including, but not limited to." In the description of this specification, terms such as "one embodiment," "some embodiments," "exemplary embodiment," "example," "specific example," or "some examples" are intended to indicate that a particular feature, structure, material, or characteristic associated with that embodiment or example is included in at least one embodiment or example of this specification. The illustrative representations of the above terms do not necessarily refer to the same embodiment or example.

[0027] Exemplary embodiments of the present application will now be described in more detail with reference to the accompanying drawings. While exemplary embodiments of the present application are shown in the drawings, it should be understood that the present application may be implemented in various forms and should not be limited to the embodiments set forth herein. Rather, these embodiments are provided so that this application will be thorough and complete, and will fully convey the scope of the present application to those skilled in the art.

[0028] Please see Figure 1 This is a flowchart illustrating a decision-making method based on deep reinforcement learning provided in an embodiment of this application. Figure 1 The method shown may include the following implementation steps: S101. Obtain multi-source data, including user blood glucose data, medication record data, user behavior data, and user physiological data, wherein the user blood glucose data is blood glucose sequence data recorded at different times.

[0029] The aforementioned user blood glucose data in this application can refer to blood glucose values ​​obtained by the user through continuous monitoring using a blood glucose meter at different time series, or simply blood glucose time series. The aforementioned medication record data can refer to relevant data in the user's medication records, which may include, but is not limited to, data such as insulin type, injection time, insulin dosage, and patient compliance score. The aforementioned user behavior data can refer to data used to describe / record user behavior, which may include, but is not limited to, data such as the user's dietary intake (e.g., carbohydrate intake), exercise data (e.g., exercise intensity), and sleep data (e.g., sleep quality). The aforementioned user physiological data can refer to data used to describe / record user physiological parameters, which may include, but is not limited to, data such as body mass index (BMI) and complication marker data (e.g., presence of kidney disease or retinopathy), etc. This application does not further limit or elaborate on these.

[0030] S102. Invoke the deep reinforcement learning model to perform virtual simulation and medication decision-making on the multi-source data, and output corresponding decision information. The decision information includes medication recommendations and risk warnings. The deep reinforcement learning model includes a state space, an action space, and a reward function. The state space is used to indicate the feature data of the multi-source data. The action space is used to determine the medication recommendations for the user. The reward function is used to guide and optimize the medication decision-making of the deep reinforcement learning model.

[0031] This application does not limit the specific type of the aforementioned deep reinforcement learning model, which may include, but is not limited to, proximal policy optimization (PPO), deep Q-network (DQN), actor-critic network framework model, or other custom network models. This application will not impose further limitations or details on this.

[0032] By implementing the embodiments of this application, this application acquires multi-source data, including user blood glucose data, medication record data, user behavior data, and user physiological data. The user blood glucose data is blood glucose sequence data recorded at different times. A deep reinforcement learning model is invoked to perform virtual simulation and medication decision-making on the multi-source data, outputting corresponding decision information, including medication recommendations and risk warnings. The deep reinforcement learning model includes a state space, an action space, and a reward function. The state space is used to indicate the feature data of the multi-source data, the action space is used to determine the user's medication recommendations, and the reward function is used to guide and optimize the medication decision-making of the deep reinforcement learning model. In this way, a deep reinforcement learning model can be used to analyze blood glucose fluctuation trends and, combined with other multi-source user data, dynamically adjust and decide on corresponding medication recommendations and risk warnings, assisting doctors in treating diabetic patients better and more accurately, thereby improving the accuracy and reliability of diabetes medication decisions. It also solves the problems of unstable blood glucose control caused by reliance on doctor experience or fixed dosage in existing technologies.

[0033] The following describes some specific and optional embodiments related to this application.

[0034] In step S101, this application does not limit the implementation method of obtaining the above-mentioned multi-source data. For example, blood glucose data such as fasting blood glucose, postprandial blood glucose and nighttime blood glucose can be continuously monitored and obtained by a blood glucose monitor; medication record data of the user can be obtained from the corresponding hospital platform system; user behavior data or user physiological data can be obtained from the corresponding device (such as wearable device, etc.). This application does not limit or elaborate on these aspects.

[0035] In step S102, this application does not limit the specific internal processing implementation of the aforementioned deep reinforcement learning model. For example, this application can first preprocess the aforementioned multi-source data to obtain corresponding processed data, which corresponds one-to-one with the aforementioned multi-source data, such as processed user blood glucose data, medication record data, user behavior data, and user physiological data. The aforementioned preprocessing is a pre-defined processing method set by the system according to the actual situation, which may include, but is not limited to, missing value processing, outlier filtering, normalization processing, or other custom processing. Specifically, for example, for missing values ​​in user blood glucose data, this application can use linear interpolation to fill in the missing values; for other data (such as medication record data, user behavior data, or user physiological data), this application can use the mean or mode to fill in the missing values. For the aforementioned multi-source data (such as blood glucose values ​​less than 2.2 or greater than 22 mmol / L in user blood glucose data), this application can use a preset sliding window for smoothing and outlier filtering processing. Regarding normalization, this application can normalize the above multi-source data to a preset reference range. For example, the reference range for blood glucose data can be [2.2, 22], the reference range for insulin dosage can be [0, 50], and the reference range for body mass index can be [15, 40]. This application will not impose further limitations or details on this.

[0036] Next, this application can construct the corresponding state space and action space based on the above-mentioned processed data. The state space can be represented by a multi-dimensional vector, containing normalized feature data for the current time step, specifically expressed as shown in the following formula (1): Formula (1) Among them, S T G represents the state space. t This represents the normalized blood glucose level at the current moment. This represents the recent normalized average blood glucose level, with T for example, 6 hours. This represents the rate of change in blood glucose, such as the slope of the normalized blood glucose value over the past hour calculated using linear regression. (D) cum This indicates the cumulative insulin dose, such as the total insulin dose injected in the past 12 hours. BMI represents the normalized body mass index. k This represents the Kth complication flag, where K is a positive integer that can be set according to the actual situation.

[0037] The aforementioned action space defines insulin adjustment strategies, such as adjusting insulin dose and / or administration time. This action space may include action types and action values, such as the insulin dose adjustment amount (e.g., ±0-5U) and administration time adjustment (e.g., ±0-30 minutes) corresponding to continuous or discrete actions, which are not further limited in this application.

[0038] Next, this application can perform simulation calculations and medication decisions on the aforementioned state space and action space based on a preset reward function to obtain corresponding decision information. The aforementioned reward function is a function that the system pre-defines according to the actual situation, and it is at least associated with the aforementioned user blood glucose data and medication record data. For example, the aforementioned reward function can be shown in the following formula (2): Formula (2) Where R represents the reward function. G target For the preset target blood glucose value, G t The current blood glucose value in the user's blood glucose data, ΔD represents the change in insulin dosage, and H represents the value at that time. compliance For patient compliance scoring. α, β, and γ are preset weighting coefficients, which can be pre-trained in the model, such as α=1.0, β=0.5, γ=0.3, etc. This application does not impose further restrictions or details on this.

[0039] This application does not impose any limitations on the internal structure of the aforementioned deep reinforcement learning models; for example, please refer to [link to relevant documentation]. Figure 2 This is a schematic diagram of the internal structure of a possible deep reinforcement learning model provided in an embodiment of this application. Figure 2 In this deep reinforcement learning model, a state encoder, a data embedding layer, a feature fusion layer, an Actor-Critic network, and an action constraint module may be included. The state space may specifically include temporal data (e.g., continuous blood glucose data) and non-temporal data (e.g., BMI, complication markers, etc.). In specific implementations, this application can utilize the state encoder to extract features from the temporal data in the state space to obtain corresponding temporal feature data (also known as hidden state data). t The aforementioned state encoder can be an LSTM encoder. Next, a data embedding layer is used to extract features from the non-temporal data in the aforementioned state space, obtaining the corresponding non-temporal feature data e. s For example, the Sigmoid activation function can be used to process non-time-series data and output non-time-series feature data e. s This application does not impose further limitations or elaborate on these aspects. Furthermore, this application can utilize a feature fusion layer to process the aforementioned time-series feature data h. t And the aforementioned non-time-series feature data e s Feature fusion can be performed, for example, by using a fully connected ReLU layer to connect and fuse the aforementioned time-series feature data h. t And the aforementioned non-time-series feature data e s Thus, the corresponding fused feature data F is obtained. tNext, this application can utilize an Actor-Critic network to predict and evaluate medication actions in the action space of the aforementioned fused feature data. Specifically, for example, an Actor network can be used to predict and evaluate medication actions in the aforementioned fused feature data F. t The process involves predicting medication actions and scaling them to a preset range (e.g., [-5, +5]) using a Tanh activation function. A Critic network is then used to evaluate the Q-value of these predicted actions, and those that pass the evaluation are taken as the corresponding decision actions. These decision actions and decision information can correspond to each other, for example, they may include medication recommendations. Next, the application can use an action constraint module to modify the decision actions, thereby obtaining corresponding decision information. This decision information may include user medication recommendations and risk warnings, such as repeating 3U of rapid-acting insulin before meals. For example, the application can modify the decision actions based on complications and blood glucose trends. Specifically, if the user's blood glucose drops too quickly (e.g., blood glucose change rate less than -0.5 mmol / L), the application can modify the insulin dose reduction to 0. Similarly, if the user has kidney disease (e.g., the kidney disease complication flag is 1), the application can modify the insulin dose to 0.6 times the original dose. This application does not impose further limitations on these modifications.

[0040] By implementing the embodiments of this application, this application acquires multi-source data, including user blood glucose data, medication record data, user behavior data, and user physiological data. The user blood glucose data is blood glucose sequence data recorded at different times. A deep reinforcement learning model is invoked to perform virtual simulation and medication decision-making on the multi-source data, outputting corresponding decision information, including medication recommendations and risk warnings. The deep reinforcement learning model includes a state space, an action space, and a reward function. The state space is used to indicate the feature data of the multi-source data, the action space is used to determine the user's medication recommendations, and the reward function is used to guide and optimize the medication decision-making of the deep reinforcement learning model. In this way, a deep reinforcement learning model can be used to analyze blood glucose fluctuation trends and, combined with other multi-source user data, dynamically adjust and decide on corresponding medication recommendations and risk warnings, assisting doctors in treating diabetic patients better and more accurately, thereby improving the accuracy and reliability of diabetes medication decisions. It also solves the problems of unstable blood glucose control caused by reliance on doctor experience or fixed dosage in existing technologies.

[0041] Based on the above embodiments, please refer to Figure 3 This is a schematic diagram of a decision-making device based on deep reinforcement learning, provided in an embodiment of this application. Figure 3 The device 300 shown may include an acquisition module 301 and a processing module 302, wherein: The acquisition module 301 is used to acquire multi-source data, including user blood glucose data, medication record data, user behavior data and user physiological data, wherein the user blood glucose data is blood glucose sequence data recorded at different times; The processing module 302 is used to call a deep reinforcement learning model to perform virtual simulation and medication decision-making on the multi-source data, and output corresponding decision information. The decision information includes medication recommendations and risk warnings. The deep reinforcement learning model includes a state space, an action space, and a reward function. The state space is used to indicate the feature data of the multi-source data. The action space is used to determine the medication recommendations for the user. The reward function is used to guide and optimize the medication decision-making of the deep reinforcement learning model.

[0042] In some embodiments, the processing module 302 is specifically used for: The multi-source data is preprocessed to obtain corresponding processed data. The preprocessing includes at least missing value processing, outlier filtering, and normalization processing. Based on the processed data, a corresponding state space and action space are constructed; Based on a preset reward function, simulation calculations and medication decisions are performed on the state space and the action space to obtain the decision information.

[0043] In some embodiments, the reward function is associated with at least the user's blood glucose data and the medication record data.

[0044] In some embodiments, the deep reinforcement learning model includes an Actor-Critic network, and the processing module 302 is specifically used for: Based on the Actor-Critic network, action evaluation and medication decisions are performed on the state space and the action space.

[0045] In some embodiments, the deep reinforcement learning model includes a state encoder, a data embedding layer, a feature fusion layer, an Actor-Critic network, and an action constraint module. The state space includes temporal and non-temporal data. The processing module 302 is specifically used for: Based on the state encoder, feature extraction is performed on the temporal data in the state space to obtain the corresponding temporal feature data; Based on the data embedding layer, feature extraction is performed on the non-temporal data in the state space to obtain the corresponding non-temporal feature data; Based on the feature fusion layer, the temporal feature data and the non-temporal feature data are fused to obtain the corresponding fused feature data; Based on the Actor-Critic network, the fused feature data is used to predict and evaluate medication actions in the action space to obtain corresponding decision actions, which include at least the medication recommendations. The decision action is modified based on the action constraint module to obtain the decision information.

[0046] In some embodiments, the medication record data includes at least insulin dose and compliance score, and the user physiological data includes at least complication markers.

[0047] In some embodiments, the reward function is: Among them, G target For the preset target blood glucose value, G t The current blood glucose value in the user's blood glucose data, ΔD represents the change in insulin dosage, and H represents the value at that time. compliance The compliance score is given.

[0048] Please see Figure 4 This is a schematic diagram of another decision-making device based on deep reinforcement learning provided in an embodiment of this application. Figure 4 The device shown can be a mobile phone, computer, digital broadcasting terminal, messaging device, game console, tablet device, medical device, fitness equipment, personal digital assistant, or other electronic device.

[0049] Reference Figure 4 The device 400 may include one or more of the following components: processing component 402, memory 404, power supply component 406, multimedia component 408, audio component 410, input / output interface 412, sensor component 414, and communication component 416.

[0050] Processing component 402 typically controls the overall operation of device 400, such as operations associated with display, telephone calls, data communication, camera operation, and recording. Processing component 402 may include one or more processors 420 to execute instructions to complete all or part of the steps of the deep reinforcement learning-based decision-making method described above. Furthermore, processing component 402 may include one or more modules to facilitate interaction between processing component 402 and other components. For example, processing component 402 may include a multimedia module to facilitate interaction between multimedia component 408 and processing component 402.

[0051] Memory 404 is configured to store various types of data to support the operation of device 400. Examples of such data include instructions for any application or method operating on device 400, contact data, phonebook data, messages, pictures, videos, etc. Memory 404 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk.

[0052] Power supply component 406 provides power to various components of device 400. Power supply component 406 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to device 400.

[0053] Multimedia component 408 includes a screen that provides an output interface between the device 400 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touchscreen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors may sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation. In some embodiments, multimedia component 408 includes a front-facing camera and / or a rear-facing camera. When the device 400 is in an operating mode, such as a shooting mode or a video mode, the front-facing camera and / or the rear-facing camera may receive external multimedia data. Each front-facing camera and rear-facing camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

[0054] Audio component 410 is configured to output and / or input audio signals. For example, audio component 410 includes a microphone (MIC) configured to receive external audio signals when device 400 is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 404 or transmitted via communication component 416. In some embodiments, audio component 410 also includes a speaker for outputting audio signals.

[0055] Input / output interface 412 provides an interface between processing component 402 and peripheral interface modules, which may be keyboards, click wheels, buttons, etc. These buttons may include, but are not limited to, home buttons, volume buttons, start buttons, and lock buttons.

[0056] Sensor assembly 414 includes one or more sensors for providing status assessments of various aspects of device 400. For example, sensor assembly 414 may detect the on / off state of device 400, the relative positioning of components such as the display and keypad of device 400, changes in the position of device 400 or a component of device 400, the presence or absence of user contact with device 400, the orientation or acceleration / deceleration of device 400, and temperature changes of device 400. Sensor assembly 414 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. Sensor assembly 414 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, sensor assembly 414 may also include an accelerometer, a gyroscope, a magnetometer, a pressure sensor, or a temperature sensor.

[0057] Communication component 416 is configured to facilitate wired or wireless communication between device 400 and other devices. Device 400 can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 416 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 416 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.

[0058] In an exemplary embodiment, the apparatus 400 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic components to perform the aforementioned deep reinforcement learning-based decision-making method.

[0059] Understandably, the processor 420 in this application embodiment can be an integrated circuit chip with signal processing capabilities. During implementation, each step of the above method embodiment can be completed by integrated logic circuits in the processor's hardware or by instructions in software form. The processor described above can be a general-purpose processor, a digital signal processor (DSP), an application-specific integrated circuit (ASIC), a field-programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0060] Understandably, the memory 404 in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. The non-volatile memory can be read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (EEPROM), or flash memory. The volatile memory can be random access memory (RAM), which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDRSDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous linked dynamic random access memory (SLDRAM), and direct rambus RAM (DR RAM). It should be noted that the memory used in the systems and methods described herein is intended to include, but is not limited to, these and any other suitable types of memory.

[0061] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 404 including instructions, which can be executed by a processor 420 of the device 400 to complete the aforementioned upper-level decision-making method based on deep reinforcement learning. For example, the non-transitory computer-readable storage medium may be a ROM, random access memory (RAM), CD-ROM, magnetic tape, floppy disk, and optical data storage device, etc.

[0062] The aforementioned device can be a standalone electronic device or a part of a standalone electronic device. For example, in one embodiment, the device can be an integrated circuit (IC) or a chip, wherein the integrated circuit can be a single IC or a collection of multiple ICs. The chip can include, but is not limited to, the following types: GPU (Graphics Processing Unit), CPU (Central Processing Unit), FPGA (Field Programmable Gate Array), DSP (Digital Signal Processor), ASIC (Application Specific Integrated Circuit), and SoC (System on Chip). The aforementioned integrated circuit or chip can be used to execute executable instructions (or code) to implement the aforementioned deep reinforcement learning-based decision-making method. The executable instructions can be stored in the integrated circuit or chip or obtained from other devices or equipment. For example, the integrated circuit or chip includes a processor, memory, and an interface for communicating with other devices. The executable instructions can be stored in the memory, and when the executable instructions are executed by the processor, the above-mentioned decision-making method based on deep reinforcement learning can be implemented; or, the integrated circuit or chip can receive the executable instructions through the interface and transmit them to the processor for execution to implement the above-mentioned decision-making method based on deep reinforcement learning.

[0063] Please see Figure 5 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. For example, as shown... Figure 5 As shown, the electronic device 500 includes a memory 501 and a processor 502. The memory 501 stores executable program code 5011, and the processor 502 is used to call and execute the executable program code 5011 to perform a decision-making method based on deep reinforcement learning.

[0064] This application embodiment can divide the electronic device into functional modules according to the above method embodiment. For example, each function can be assigned to a separate module, or two or more functions can be integrated into a processing module. The integrated module can be implemented in hardware. It should be noted that the module division in this embodiment is illustrative and only represents one logical functional division. In actual implementation, there may be other division methods. When dividing each functional module according to its corresponding function, the electronic device may include: a processing module and a communication module, etc.

[0065] It should be noted that all relevant content of each step involved in the above method embodiments can be referenced from the functional description of the corresponding functional module, and will not be repeated here. The electronic device provided in this embodiment is used to execute the above-described decision-making method based on deep reinforcement learning, and therefore can achieve the same effect as the above implementation method.

[0066] In another exemplary embodiment, a computer program product is also provided, the computer program product comprising a computer program executable by a programmable device, the computer program having a code portion for executing the aforementioned deep reinforcement learning-based decision-making method when executed by the programmable device.

[0067] It should be noted that the descriptions of the above embodiments of storage media, devices, and equipment are similar to the descriptions of the above method embodiments, and have similar beneficial effects. For technical details not disclosed in the embodiments of storage media, devices, and equipment of this application, please refer to the descriptions of the method embodiments of this application for understanding.

[0068] Other embodiments of this application will readily occur to those skilled in the art upon consideration of the specification and practice of this application. This application is intended to cover any variations, uses, or adaptations of this application that follow the general principles of this application and include common knowledge or customary techniques in the art not disclosed in this application. The specification and embodiments are to be considered exemplary only, and the true scope and spirit of this application are indicated by the following claims.

[0069] The above description is merely a preferred embodiment of this application and is not intended to limit this application. Any modifications or equivalent substitutions made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A decision-making method based on deep reinforcement learning, characterized in that, include: Acquire multi-source data, including user blood glucose data, medication record data, user behavior data, and user physiological data, wherein the user blood glucose data is blood glucose sequence data recorded at different times; A deep reinforcement learning model is invoked to perform virtual simulation and medication decision-making on the multi-source data, and output corresponding decision information, including medication recommendations and risk warnings. The deep reinforcement learning model includes a state space, an action space, and a reward function. The state space is used to indicate the feature data of the multi-source data, the action space is used to determine the medication recommendations for the user, and the reward function is used to guide and optimize the medication decision-making of the deep reinforcement learning model.

2. The method according to claim 1, characterized in that, The virtual simulation and medication decision-making process based on the multi-source data, and the output of corresponding decision information, include: The multi-source data is preprocessed to obtain corresponding processed data. The preprocessing includes at least missing value processing, outlier filtering, and normalization processing. Based on the processed data, a corresponding state space and action space are constructed; Based on a preset reward function, simulation calculations and medication decisions are performed on the state space and the action space to obtain the decision information.

3. The method according to claim 2, characterized in that, The reward function is associated with at least the user's blood glucose data and the medication record data.

4. The method according to claim 2, characterized in that, The deep reinforcement learning model includes an Actor-Critic network, and the simulation calculation and medication decision-making for the state space and the action space include: Based on the Actor-Critic network, action evaluation and medication decisions are performed on the state space and the action space.

5. The method according to claim 4, characterized in that, The deep reinforcement learning model includes a state encoder, a data embedding layer, a feature fusion layer, an Actor-Critic network, and an action constraint module. The state space includes temporal and non-temporal data. The action evaluation and medication decision based on the Actor-Critic network in the state space and the action space includes: Based on the state encoder, feature extraction is performed on the temporal data in the state space to obtain the corresponding temporal feature data; Based on the data embedding layer, feature extraction is performed on the non-temporal data in the state space to obtain the corresponding non-temporal feature data; Based on the feature fusion layer, the temporal feature data and the non-temporal feature data are fused to obtain the corresponding fused feature data; Based on the Actor-Critic network, the fused feature data is used to predict and evaluate medication actions in the action space to obtain corresponding decision actions, which include at least the medication recommendations. The decision action is modified based on the action constraint module to obtain the decision information.

6. The method according to any one of claims 1-5, characterized in that, The medication record data includes at least insulin dosage and compliance score, and the user physiological data includes at least complication markers.

7. The method according to claim 6, characterized in that, The reward function is: Among them, G target For the preset target blood glucose value, G t The current blood glucose value in the user's blood glucose data, ΔD represents the change in insulin dosage, and H represents the value at that time. compliance The compliance score is given.

8. A decision-making device based on deep reinforcement learning, characterized in that, include: The acquisition module is used to acquire multi-source data, including user blood glucose data, medication record data, user behavior data, and user physiological data. The user blood glucose data is blood glucose sequence data recorded at different times. The processing module is used to call a deep reinforcement learning model to perform virtual simulation and medication decision-making on the multi-source data, and output corresponding decision information, including medication recommendations and risk warnings. The deep reinforcement learning model includes a state space, an action space, and a reward function. The state space is used to indicate the feature data of the multi-source data, the action space is used to determine the medication recommendations for the user, and the reward function is used to guide and optimize the medication decision-making of the deep reinforcement learning model.

9. A computer device, characterized in that, include: processor; A memory for storing processor-executable instructions; wherein the processor is configured to execute the executable instructions to implement the steps of the method according to any one of claims 1-7.

10. A computer-readable storage medium having computer program instructions stored thereon, characterized in that, When the computer program instructions are executed by the processor, they implement the steps of the method according to any one of claims 1-7.

Images