Pet intelligent product structure generation method based on multi-agent reinforcement learning

Abstract

The method comprises the following steps: 1) constructing a virtual three-dimensional environment containing geometric constraints of a target pet product shell, and predefining an appearance model for digital generation, wherein specific components are marked as static components with fixed positions; 2) modeling functional modules to be laid out as agents with autonomous decision-making capabilities, wherein the state space of each agent at least includes position and attitude information in the virtual three-dimensional environment, and the action space includes translation, rotation and connection request operations in the environment; 3) defining a unified global reward function for all agents, which is a weighted sum of a layout reward item, a connection relationship reward item and a system-level reward item, wherein the connection relationship reward item is calculated according to the layout length and complexity of the predetermined electrical and logical connection path between modules; 4) each agent performs collaborative exploration and training in the virtual three-dimensional environment based on a multi-agent reinforcement learning algorithm, learns an optimal collaborative strategy by maximizing the long-term cumulative global reward function; and 5) after training convergence, the final three-dimensional layout position, attitude and connection topology relationship of all functional modules in the restricted shell space are determined according to the action sequence output by the final strategy of each agent, and a complete manufacturable structure scheme is generated.

Description

Technical Field

[0001] This invention relates to the field of interdisciplinary technology of artificial intelligence and intelligent manufacturing, specifically a method for generating intelligent pet product structures based on multi-agent reinforcement learning. It is particularly suitable for achieving automated layout and integrated optimization of multifunctional electronic and mechanical components within a predefined irregular pet product shell space. Background Technology

[0002] With the rapid development of the smart pet product market, product functions are becoming increasingly complex and integrated. Traditional product structure design methods mainly rely on engineers to manually lay out components. Within a given irregular product shell space, manual layout is difficult to simultaneously meet multiple constraints such as electrical connection, heat dissipation management, and signal integrity. Secondly, compatibility conflicts and physical interference between components are often only discovered in the later stages of design, resulting in long design cycles and high costs. In addition, traditional optimization algorithms are prone to getting stuck in local optima when dealing with complex layout problems with multiple objectives and constraints, and cannot adaptively learn optimization strategies.

[0003] Existing automatic layout methods are mostly based on traditional optimization algorithms such as genetic algorithms or simulated annealing. These methods perform well in solving single-objective optimization problems, but they struggle to effectively coordinate multiple conflicting design goals and cannot learn and evolve from historical design experience. Furthermore, traditional methods treat the layout problem as a holistic optimization, failing to reflect the collaborative and competitive relationships between components, resulting in layout schemes with low feasibility in practical engineering. Summary of the Invention

[0004] The purpose of this invention is to overcome the shortcomings of the prior art and provide a method for generating intelligent pet product structures based on multi-agent reinforcement learning. This method regards each functional component as an intelligent agent with autonomous decision-making ability, and automatically generates the optimal product structure layout scheme under complex constraints through multi-agent collaborative learning.

[0005] The present invention provides a method for generating the structure of intelligent pet products based on multi-agent reinforcement learning, comprising the following steps: S1: Construct a virtual 3D environment containing the geometric constraints of the target pet product shell. The virtual 3D environment is digitally generated based on a predefined appearance model of the product, wherein specific components are marked as static components with fixed positions. S2: The functional modules to be laid out are modeled as intelligent agents with autonomous decision-making capabilities. The state space of each intelligent agent includes at least its position and posture information in the virtual three-dimensional environment, and its action space includes translation, rotation and connection request operations in the environment. S3: Define a unified global reward function R for all agents. This function R is the layout reward term. Rlayout Connection Relationship Rewards R connection With system-level rewards R system The weighted sum, i.e. R = αR layout + βR connection + γ R system The connection relationship reward item R connection Calculations are performed based on the length and complexity of the predetermined electrical and logical connection paths between each module; S4: Each agent collaborates and trains in the virtual 3D environment based on a multi-agent reinforcement learning algorithm, maximizing the long-term accumulated global reward function. R Learn the optimal collaborative strategy; S5: After training convergence, based on the action sequence output by each agent's final policy, determine the final three-dimensional layout position, orientation, and connection topology of all functional modules within the confined shell space, and generate a complete manufacturable structure scheme.

[0006] Preferably, the construction of the virtual three-dimensional environment further includes: importing a triangular mesh or boundary representation model of the pet product shell, and defining its internal cavity as a restricted space in which the intelligent agent can move; according to the product architecture definition, pre-setting one or more components among the motor mounting base, power interface position, and main structural support points as static entities whose position and orientation cannot be changed, and using them as part of the environmental constraints.

[0007] Preferably, the layout reward item R layout The calculation considers at least the following factors: imposing penalties for geometric interference between components or between components and the shell, and rewarding space utilization exceeding a set threshold; the connection relationship reward item. R connectio The calculation should consider at least the following factors: penalties for actual connection path lengths exceeding the protocol's maximum allowed length, penalties for excessive path turns, and penalties for insufficient reserved cable tray space; the system-level reward item. R system The calculation should at least consider the following factors: penalties for overly concentrated layout of heat-generating components, proximity of sensitive components to interference sources, and obstruction of maintenance channels.

[0008] Preferably, the multi-agent reinforcement learning algorithm adopts a framework of centralized training and distributed execution. During the training phase, a central critic network calculates the expected value of the global reward R based on the global state and the actions of all agents. The actor network corresponding to each agent outputs an action policy based on its local observations and updates it through the policy gradient method to maximize the value evaluated by the central critic network.

[0009] Preferably, a conflict resolution mechanism is introduced during training: when the central commentator network detects that the actions of multiple agents cause the critical constraints of the task to be continuously violated, the global reward function is dynamically adjusted. R Medium weight α , β , γ The allocation of resources, or the application of additional exploration guidance to relevant agents, can be used to encourage the agent group to escape local optimal strategies.

[0010] Compared with existing technologies, this invention effectively solves the component layout problem under complex constraints through multi-agent collaborative learning. The generated structural scheme outperforms traditional methods in terms of connection length, heat dissipation performance, and signal integrity, while completely avoiding component interference. By modeling inter-component interactions using an attention mechanism, it can adaptively learn the cooperative and competitive relationships between components, improving the engineering feasibility of the layout scheme. The reinforcement learning framework can learn from historical design data to achieve continuous optimization. This invention achieves full-process automation from requirements to manufacturing, shortening the design cycle and significantly improving the development efficiency of smart pet products. Attached Figure Description

[0011] Figure 1 This is an overall flowchart of a method for generating the structure of a smart pet product according to an embodiment of the present invention.

[0012] Figure 2 This is the reward curve for the optimization process in one embodiment of the present invention.

[0013] Figure 3 This is a layout of components for a smart pet product in one embodiment of the present invention. Detailed Implementation

[0014] To make the objectives, technical solutions, and advantages of this invention clearer, the following detailed description of the embodiments of this invention is provided using a structural example of a typical smart pet product: a smart cat litter box. Those skilled in the art will understand that the specific examples described herein are for illustrative purposes only and are not intended to limit the scope of protection of this invention.

[0015] Taking a "smart litter box suitable for multi-cat households, with health monitoring functions and automatic cleaning support" as an example, the system loads information on all standard functional modules related to the "smart litter box." These modules constitute a candidate set of agents. Modules include, but are not limited to: UV sterilization lamp module, deodorizing fan module, main control board module, weight sensor module, infrared sensor module, roller motor module, Wi-Fi module, and power supply module. Each module is instantiated as an independent agent in the system, and its attributes (such as physical size, interface type, power consumption, heat generation, and installation constraints) are digitized.

[0016] Subsequently, based on the specific industrial design of the cat litter box shell, a three-dimensional virtual confined space environment is generated. This environment is a digital twin model that precisely describes the internal geometry of the shell, fixed mounting points, obstacle areas, and immutable constraints such as heat dissipation channels. This environment serves as the stage for all intelligent agents to learn and interact together.

[0017] Each functional module agent is endowed with independent learning capabilities. Its state space S is defined as its coordinates, orientation, and connection status with other agents in the virtual environment. Its action space A is defined as performing small translations or rotations within a confined space, or issuing requests to establish connections with other agents.

[0018] The core lies in designing a unified global reward function for all agents. R This function guides the evolution of group behavior towards the optimal structure. It embodies a multi-objective trade-off, and its basic form is:

[0019] in, α , β , γ These are weighting coefficients used to balance the importance of different optimization objectives. Specifically: R layout (Layout reward) is manifested as follows: when the space occupied by the intelligent agent does not collide with the shell or other intelligent agents, and the layout is compact and the center of gravity is stable, a positive reward is obtained; when a collision occurs or the agent deviates significantly from the reasonable area, a negative penalty is obtained. R connection (Connectivity Reward) manifests as follows: when the estimated wiring path between agent A and agent B, which needs to connect, is shorter, has fewer turns, and has sufficient space reserved in the path, a high reward is obtained. The closer the ratio of the connection length to the maximum length allowed by the communication protocol is to the upper limit, the lower the reward becomes, even to the point of being negative. This reward mechanism forces agents to consider "how to connect with partners" as a core consideration when moving. R system(System-level performance rewards) encode the engineering experience of IoT products. For example, a reward is given when a heat-generating agent is placed in a location that facilitates heat dissipation and is far away from temperature-sensitive agents. A reward is also given when easily maintainable agents are placed in locations that are easy for users to access. Conversely, penalties are imposed if the layout leads to potential heat buildup or maintenance difficulties.

[0020] After training begins, all agents explore the virtual environment in parallel. In the initial phase, their actions are random, which may lead to a large number of collisions and chaotic states where they cannot connect, resulting in very low global rewards.

[0021] Through multi-agent reinforcement learning algorithms, each agent progressively learns its policy network π. This policy network can select actions that maximize long-term cumulative rewards based on the current environment and the states of other agents. In this process, complex cooperative and competitive relationships spontaneously form among the agents. For example: The weight sensor agent wants to be closer to the main control board agent to shorten the data cable (improve efficiency). R connection However, getting too close may cause physical interference (reducing...) R layout It may also be affected by the heat of the main control board (reducing...) R system Therefore, it learns to find a balance point, such as moving to a position at the bottom of the basin that allows for a direct connection to the main control board while maintaining a safe distance and good ventilation. The position of the roller motor agent is strictly constrained by the mechanical structure of the casing (fixed position), preventing it from moving significantly. However, it influences the position of the drive circuit agent connected to it through signals, prompting the latter to move closer to optimize the connection. To avoid electromagnetic interference from the motor's operation, the Wi-Fi module agent learns to move to the other side of the space, even if this increases wiring length, but avoids the penalty of decreased system performance.

[0022] Through millions of simulation iterations, the entire swarm continuously tries and learns, eventually converging to one or more stable states with high rewards. This state represents a Pareto optimal component integration scheme within a given shell, considering multiple dimensions such as layout rationality, connection feasibility, and system reliability. Once the agent swarm converges, the system derives the final pose and connection relationship information.

[0023] For an example of a smart litter box, the final solution might look like this: the weighing sensor is optimized and placed in a specific location on the box for accurate measurement; the main control board is located high on the side wall, away from the water source and close to the Wi-Fi antenna; the power module is independently placed in the bottom ventilation area; all cable paths are clear, the length is controllable, and they avoid moving parts and high-temperature areas.

Claims

1. A method for generating the structure of intelligent pet products based on multi-agent reinforcement learning, characterized in that, Includes the following steps: S1: Construct a virtual 3D environment containing the geometric constraints of the target pet product shell. The virtual 3D environment is digitally generated based on a predefined appearance model of the product, wherein specific components are marked as static components with fixed positions. S2: The functional modules to be laid out are modeled as intelligent agents with autonomous decision-making capabilities. The state space of each intelligent agent includes at least its position and posture information in the virtual three-dimensional environment, and its action space includes translation, rotation and connection request operations in the environment. S3: Define a unified global reward function R for all agents. This function R is the layout reward term. R layout Connection Relationship Rewards R connection With system-level rewards R system The weighted sum, wherein the connection reward item R connection Calculations are performed based on the length and complexity of the predetermined electrical and logical connection paths between each module; S4: Each agent collaborates and trains in the virtual 3D environment based on a multi-agent reinforcement learning algorithm, maximizing the long-term accumulated global reward function. R Learn the optimal collaborative strategy; S5: After training convergence, based on the action sequence output by each agent's final policy, determine the final three-dimensional layout position, orientation, and connection topology of all functional modules within the confined shell space, and generate a complete manufacturable structure scheme.

2. The method according to claim 1, characterized in that, In step S1, the construction of the virtual three-dimensional environment further includes: importing a triangular mesh or boundary representation model of the pet product shell and defining its internal cavity as a restricted space in which the intelligent agent can move; according to the product architecture definition, pre-setting one or more components among the motor mounting base, power interface position, and main structural support points as static entities whose position and orientation cannot be changed, and using them as part of the environmental constraints.

3. The method according to claim 1, characterized in that, In step S3, the global reward function R This is used to guide group behavior towards the optimal structure, and takes the following form: in, α , β , γ These are weighting coefficients used to balance the importance of different optimization objectives.

4. The method according to claim 3, characterized in that, In step S3, the layout reward item R layout The calculation should at least consider the following factors: penalize geometric interference between components or between components and the shell, and reward space utilization that exceeds a set threshold; The connection relationship reward item R connectio The calculation should consider at least the following factors: penalties for actual connection path lengths exceeding the protocol's maximum allowed length, penalties for excessive path turns, and penalties for insufficient reserved cable tray space; the system-level reward item. R system The calculation should at least consider the following factors: penalties for overly concentrated layout of heat-generating components, proximity of sensitive components to interference sources, and obstruction of maintenance channels.

5. The method according to claim 1, characterized in that, In step S4, the multi-agent reinforcement learning algorithm adopts a framework of centralized training and distributed execution. During the training phase, a central critic network calculates the expected value of the global reward R based on the global state and the actions of all agents. The actor network corresponding to each agent outputs the action policy based on its local observations and updates it through the policy gradient method to maximize the value evaluated by the central critic network.

6. The method according to claim 4, characterized in that, A conflict resolution mechanism is introduced during training: when the central commentator network detects that the actions of multiple agents cause continuous violations of task-critical constraints, the global reward function is dynamically adjusted. R Medium weight α , β , γ The allocation of resources, or the application of additional exploration guidance to relevant agents, can be used to encourage the agent group to escape local optimal strategies.

Images