Model data processing method, apparatus, device, storage medium, and program product

Abstract

The application provides a model data processing method and device, relates to the technical field of Internet, can be applied to various scenes such as cloud technology, artificial intelligence, intelligent transportation and auxiliary driving, and comprises the following steps: in response to a connection request carrying a target random identifier sent by a first end, a connection response for establishing a target transmission connection with the first end is sent to the first end; the target random identifier is used for identifying the target transmission connection; in response to a model data request carrying the target random identifier sent by the first end through the target transmission connection, corresponding target model data is acquired; a data response carrying the target model data is sent to the first end through the target transmission connection, so that the first end performs corresponding model training based on the target model data; and in the case that the model data request carries model update data generated by the first end, the model database is updated based on the model update data. The application can effectively improve the data connection efficiency and transmission efficiency, and further improve the training efficiency.

Description

Technical Field

[0001] This application relates to the field of Internet technology, and in particular to a model data processing method, apparatus, device, storage medium, and program product. Background Technology

[0002] With the development of artificial intelligence technology, machine learning and deep learning models are widely used in fields such as computer vision, speech processing, natural language processing, autonomous driving, and smart healthcare. These applications involve the training and storage of numerous models. Taking information flow recommendation scenarios in computer vision applications as an example, machine learning models score articles, images, and videos, and then apply the scores to content ranking, allowing users to access valuable content. Related information flow products require training various machine learning models for the ranking and recall stages of recommendation systems. Typically, model training services and storage services for training data, such as parameters, are deployed on different servers, and training data needs to be stored and updated across multiple servers. Therefore, there are significant model data processing requirements during model training. Due to this distributed deployment, if the training data storage service experiences deployment updates or anomalies, the training server needs to update its communication address, increasing reconnection time costs and potentially requiring a restart of the training service.

[0003] Therefore, a model data processing solution is needed to solve the above-mentioned problems in the existing technology and improve the model training efficiency. Summary of the Invention

[0004] This application provides a model data processing method, apparatus, device, storage medium, and program product, which can effectively reduce the speed of model data processing and optimize web performance.

[0005] On the one hand, this application provides a model data processing method, the method comprising: The terminal receives a page loading request, which includes page information. Determine the simplified style data associated with the page information; the simplified style data is obtained by simplifying the full style data corresponding to the type information of the matched page data. Obtain target simplified style data that matches the type information of the target page data from the associated simplified style data; A target page file is generated based on the target page data and the target simplified style data. The target page file is used to generate the target page corresponding to the page loading request.

[0006] On the other hand, a model data processing apparatus is provided, the apparatus comprising: Request receiving module: used to receive page loading requests sent by the terminal, the page loading request including page information; Page data acquisition module: used to acquire the target page data corresponding to the page information and the type information of the target page data; The associated style data determination module is used to determine the simplified style data associated with the page information; the simplified style data is obtained by simplifying the full style data corresponding to the type information of the matched page data. Target style data acquisition module: used to acquire target simplified style data that matches the type information of the target page data from the associated simplified style data; Model data processing module: used to generate a target page file based on the target page data and the target simplified style data, wherein the target page file is used to generate the target page corresponding to the page loading request.

[0007] On the other hand, a model data processing device is provided, the device including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the model data processing method as described above.

[0008] On the other hand, a computer-readable storage medium is provided, wherein at least one instruction or at least one program is stored therein, the at least one instruction or the at least one program being loaded and executed by a processor to implement the model data processing method as described above.

[0009] On the other hand, a computer device is provided, the computer device including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the model data processing method as described above.

[0010] On the other hand, a server is provided, the server including a processor and a memory, the memory storing at least one instruction or at least one program, the at least one instruction or the at least one program being loaded and executed by the processor to implement the model data processing method as described above.

[0011] On the other hand, a computer program product or computer program is provided, which includes computer instructions that, when executed by a processor, implement the model data processing method as described above.

[0012] The model data processing method, apparatus, equipment, storage medium, and server provided in this application have the following technical effects: In response to a connection request carrying a target random identifier sent by a first end, this application sends a connection response to the first end to establish a target transmission connection; the target random identifier is used to identify the target transmission connection; in response to a model data request carrying a target random identifier sent by the first end through the target transmission connection, the application obtains the corresponding target model data; and sends a data response carrying the target model data to the first end through the target transmission connection, so that the first end can perform corresponding model training based on the target model data; and when the model data request carries model update data generated by the first end, the application updates the model database based on the model update data. The transmission connection establishment process of this application has low latency and is concise, effectively improving connection efficiency. In multi-node transmission scenarios for model training, it can effectively improve data connection efficiency and transmission efficiency, thereby improving training efficiency. Attached Figure Description

[0013] To more clearly illustrate the technical solutions and advantages in the embodiments of this application or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0014] Figure 1 This is a schematic diagram of the structure of a system provided in an embodiment of this application; Figure 2 This is a flowchart illustrating a model data processing method provided in an embodiment of this application; Figure 3 This is a framework diagram of a system provided in an embodiment of this application; Figure 4 This is a communication framework diagram of a system provided in an embodiment of this application; Figure 5 This is a schematic diagram of a model data processing flow provided in an embodiment of this application; Figure 6 This is a flowchart illustrating a model data processing method provided in an embodiment of this application; Figure 7 This is a flowchart illustrating a model data processing method provided in an embodiment of this application; Figure 8 This is a schematic diagram of the structure of a model data processing device provided in an embodiment of this application; Figure 9 This is a schematic diagram of the structure of a model data processing device provided in an embodiment of this application; Figure 10 This is a hardware structure block diagram of a server for a model data processing method provided in an embodiment of this application; Figure 11 This is a schematic diagram of the structure of a blockchain system provided in an embodiment of this application. Detailed Implementation

[0015] Cloud computing refers to the delivery and usage model of IT infrastructure, meaning obtaining necessary resources in an on-demand and easily scalable manner through a network. In a broader sense, cloud computing also refers to the delivery and usage model of services, meaning obtaining necessary services in an on-demand and easily scalable manner through a network. These services can be IT and software related, internet-related, or other services. Cloud computing is a product of the development and integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing.

[0016] The technical solutions of this application embodiment can utilize cloud computing and cloud storage technologies to provide resource data services such as style data for model data processing. Cloud storage is a new concept that extends and develops from the concept of cloud computing. A distributed cloud storage system (hereinafter referred to as a storage system) refers to a storage system that uses cluster applications, grid technology, and distributed storage file systems to aggregate a large number of storage devices (also called storage nodes) of various types in the network through application software or application interfaces to work together and jointly provide data storage and business access functions to the outside world.

[0017] Currently, the storage method in storage systems is as follows: Logical volumes are created. During creation, physical storage space is allocated to each logical volume. This physical storage space may consist of a single storage device or the disks of several storage devices. Clients store data on a logical volume, which means storing the data on the file system. The file system divides the data into many parts, each part being an object. Each object contains not only the data but also additional information such as a data identifier (ID, ID entity). The file system writes each object to the physical storage space of that logical volume and records the storage location information of each object. Therefore, when a client requests access to data, the file system can allow the client to access the data based on the storage location information of each object.

[0018] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those of ordinary skill in the art without creative effort are within the scope of protection of this application.

[0019] It should be noted that the terms "first," "second," etc., in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or server that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or devices.

[0020] Please see Figure 1 , Figure 1 This is a schematic diagram of a system provided in an embodiment of this application, such as... Figure 1 As shown, the system may include at least a first terminal 01 and a second terminal 02. In practical applications, the first terminal 01 and the second terminal 02 can be directly or indirectly connected via wired or wireless communication to achieve interaction between them; this application does not impose any limitations on this. Specifically, the first terminal 01 and the second terminal 02 may include physical devices such as servers, or software running on the physical devices, such as applications.

[0021] In this embodiment, the server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network), and big data and artificial intelligence platforms. Specifically, the server can include physical devices, such as network communication units, processors, and memory, and software running on the physical devices, such as applications.

[0022] In this embodiment, the first end 01 is used to provide model training services, including at least one first end node for performing model training, and a training computation process is running on the first end node. The first end 01 can send a connection request carrying a target random identifier to the second end 02, and after establishing a transmission connection between the first end node and the second end node, it sends a model data request and model update data to the second end 02 through the transmission connection to obtain model data, and performs corresponding model training based on the model data.

[0023] In this embodiment of the application, the second end 02 is used to provide model data services, including at least one second end node for providing data services. After receiving a connection request carrying a target random identifier, the second end 02 sends a connection response to the first end 01 to establish a transmission connection; and in response to the model data request sent by the first end 01, it obtains the corresponding model data and updates the model database according to the model update data.

[0024] In addition, it should be noted that, Figure 1 The system shown is merely one example of a model data processing system. In practical applications, it may include more or fewer nodes, such as terminals for receiving model training instructions or managing model data. Terminals include, but are not limited to, mobile phones, computers, smart voice interaction devices, smart home appliances, and vehicle terminals. This application does not impose any restrictions on these.

[0025] The following describes a model data processing method based on the aforementioned system. Embodiments of this application can be applied to various scenarios, including but not limited to cloud technology, artificial intelligence, smart transportation, and assisted driving. This specification provides method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive labor, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only execution order. In actual system or server product execution, the method can be executed sequentially according to the embodiments or accompanying drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). For details, please refer to... Figure 2 The methods may include: S201: The first end sends a connection request carrying a target random identifier to the second end.

[0026] In this embodiment, the first end is a training server, which can be built based on a training server cluster to provide model training services, including at least one first-end node for performing model training. Specifically, at least one training computation process (training computation node) runs on a first-end node. The second end is a data server, which can be built based on a model data server cluster to provide model data services, including at least one second-end node for providing data services, and at least one data service process (data service node) runs on a second-end node.

[0027] In some embodiments, please refer to Figure 3 , Figure 3 A system architecture diagram is shown, illustrating how bidirectional transmission connections can be established between the training computation processes on the training server and the second-end nodes on the data server, based on the model data processing method provided in this application. Further details can be found in the following references. Figure 4 , Figure 4 This diagram illustrates a communication architecture framework for a system based on this application. The computation program corresponding to the training computation process and the computation program corresponding to the second end node establish a bidirectional transmission connection through the underlying framework and the upper-layer transmission protocol. In one specific embodiment, the underlying framework includes, but is not limited to, an RPC framework, and the transmission protocol includes, but is not limited to, the QUIC protocol.

[0028] In practical applications, after the first end initiates model training, each training computation process generates multiple connection requests carrying random identifiers based on the aforementioned upper-layer transmission protocol to establish transmission connections with each second-end node. The random identifier is used to identify the transmission connection between the first and second ends, with each transmission connection corresponding to a random identifier. That is, the first end uses different random identifiers to mark different transmission connections. When a training computation process establishes multiple transmission connections with multiple second-end nodes, a corresponding random identifier is generated for each transmission connection. Specifically, this random identifier is a unique connection ID, which can be a random number with a preset number of bits, such as a 32-bit or 64-bit random number, randomly generated by the training computation process. In one embodiment, a training computation process can establish at least one transmission connection with one second-end node.

[0029] S203: In response to the connection request sent by the first end, which carries a target random identifier, the second end sends a connection response to the first end to establish a target transmission connection with the first end; the target random identifier is used to identify the target transmission connection.

[0030] In this embodiment, after receiving a connection request carrying a target random identifier, the second-end node generates and sends a connection response corresponding to the connection request to the training computation process of the first end based on the aforementioned upper-layer transmission protocol. Upon receiving the connection response, the first end completes the establishment of the target transmission connection. After the target transmission connection is established, the corresponding first-end node and second-end node can train data through their transmission models. Thus, by utilizing the upper-layer transmission protocol, the transmission connection is established in the initial communication between the first and second ends based on the connection request carrying a random identifier and the corresponding connection response, effectively improving communication efficiency.

[0031] S205: The first end sends a model data request carrying a target random identifier to the second end through the target transmission connection.

[0032] In some cases, after the first node receives a connection response from the second node, a target transmission connection is established between the corresponding first and second node. Specifically, this is a transmission connection between the training computation process and the data service process. In other cases, establishing the target transmission connection requires connection verification, which can be achieved using existing encryption algorithms such as the Diffie-Hellman algorithm. Specifically, after receiving a connection request, the second node generates a connection response carrying service configuration information including service public key information, encryption algorithm information, and certificate information, and sends it to the first node. Upon receiving this connection response, the first node sends its first model data request, which includes the process public key information. Both the first and second nodes verify the received public key information; successful verification establishes the target transmission connection. After the target transmission connection is established, the training computation process and the second node transmit model data requests and responses through this connection using the aforementioned upper-layer transmission protocol.

[0033] Furthermore, assuming the system as a whole is trustworthy, or if the training computation process caches or persists the service configuration information and target connection identifier, the connection request also carries model data request information. In step S201, the training computation process at the first end directly sends a connection request carrying the target random identifier and data request information. In step S203, the second-end node at the second end responds to the data request information in the connection request, obtains the corresponding target model data, and sends a connection response carrying the target model data to the training computation process at the first end, thereby establishing the target transmission connection. After the target transmission connection is established, the training computation process and the second-end node transmit model data requests and data responses through this target transmission connection.

[0034] S207: The second end responds to the model data request carrying the target random identifier and obtains the corresponding target model data.

[0035] In this embodiment, the second end node retrieves the target model data corresponding to the model data request from the model database based on the parsing result. Specifically, the model database may store model training parameter data and model configuration data, etc.

[0036] S209: The second end sends a data response carrying the target model data to the first end through the target transmission connection.

[0037] In this embodiment, the second end node can determine the corresponding target transmission connection based on the target random identifier, generate a data response corresponding to the model data request, encapsulate the target model data in the data response, and send it to the corresponding first end node through the target transmission connection.

[0038] S211: When the model data request carries model update data generated by the first end, the second end updates the model database based on the model update data.

[0039] In this embodiment, the model update data is the model data generated by the first end node during model training iterations, which may include, but is not limited to, updated model parameters and updated model configurations. That is, when the model data request is generated after the first end node begins model training, the first end node generates model update data, encapsulates the model update data in the model data request, and sends it to the corresponding second end node through the target transmission connection. After parsing the model update data, the second end node stores it in the corresponding storage location of the model database to update the model database.

[0040] S213: The first end responds to the data response carrying the target model data and performs corresponding model training based on the target model data.

[0041] In this embodiment of the application, after the training computation process receives the target model data, it performs model training based on the target model data to generate new model update data, and then generates a model data request carrying the target random identifier and model update data again, repeating the above steps S205 to S213.

[0042] The transmission connection established in this application has low latency and a short process, which effectively improves connection efficiency. In multi-node transmission scenarios for model training, it can greatly reduce connection time, improve data connection efficiency and transmission efficiency, and thus improve model training efficiency.

[0043] In this embodiment, communication failures or interruptions may occur between the training computation process and the second-end node during the training process. The reasons may include, but are not limited to: restarting the computation process due to abnormal exit of the training computation process; restarting or redeploying the first-end node due to abnormality or failure of the first-end node; restarting the service process due to abnormal exit of the data service process; and restarting or redeploying the second-end node due to abnormality or failure of the second-end node. All of these situations will result in a change in the connection address, i.e., a change in the connection address corresponding to the target transmission connection. Accordingly, in the case of a change in the connection address corresponding to the target transmission connection, before S209, the method further includes S215: the second end generates a data response carrying the changed connection address and target model data.

[0044] In this embodiment, the connection request, connection response, model data request, and data response also carry a connection address. The connection address includes the address of the first end node and / or the training computation process, and the address of the second end node and / or the data service process. This connection address is typically encapsulated in the request header and response header, and may include a source IP address, a source port number, a destination IP address, and a destination port number. Specifically, the source IP address can be the IP address of the first end node, the source port number can be the computation process port number of the training computation process, the destination IP address can be the IP address of the second end node, and the destination port number can be the service process port number of the data service process or the node port number of the second end node.

[0045] In some embodiments, if the connection address change is a change in the address of the first end node corresponding to the target transport connection, before S215, the method further includes: S301: The first end determines the changed connection address based on the updated address of the first end node and the current address of the second end node.

[0046] S303: The first end sends a model data request carrying the changed connection address to the second end through the target transmission connection.

[0047] Specifically, the first-end node is the node running on the first end for performing model training; the second-end node is the node running on the second end for performing model data services.

[0048] Specifically, in cases such as abnormal termination of the training computation process leading to a restart, or abnormality or failure of the first-end node leading to a restart or redeployment of the first-end node, the addresses of the first-end node and / or the training computation process will change. For example, if the training computation process restarts abnormally, the port number of the first-end node's computation process will change; if the first-end node restarts or is redeployed, both the IP address of the first-end node and the port number of the computation process will change.

[0049] Specifically, after the address of the first end node and / or the training computation process changes, the training computation process directly generates a new connection address based on the updated address of the first end node and the current address of the second end node (i.e., the address of the second end node before the address change), and generates a model data request carrying the parameters of the new connection address based on the above-mentioned upper-layer transmission protocol, and sends it to the data service node through the target transmission connection without having to re-establish the transmission connection.

[0050] Accordingly, S215 specifically refers to: the second end node responding to the model data request carrying the changed connection address, generating a data response carrying the changed connection address, and encapsulating the target model data in the data response and sending it together with the target transmission connection to the training computation process of the first end node.

[0051] In other embodiments, if the connection address change is a change in the address of the second end node corresponding to the target transport connection, the method further includes, before S215,... S305: The first end generates a model data request carrying the current address and address retrieval information of the first end node; S307: The second end retrieves information based on the address and determines the changed connection address according to the current address of the first end node and the updated address of the second end node.

[0052] Specifically, the address retrieval information is used to instruct the reacquisition of the address of the second-end node. In cases such as abnormal exit of the data service process leading to a service process restart, or abnormality or failure of the second-end node leading to a restart or redeployment of the second-end node, the addresses of the second-end node and / or the data service process will change. For example, if the data service process restarts abnormally, the service process port number of the second-end node will change; if the second-end node restarts or is redeployed, both the IP address of the second-end node and the service process port number / node port number will change.

[0053] Specifically, in the event of a communication anomaly caused by the aforementioned abnormality at the second end, the model data request sent by the training computation process cannot reach the corresponding second-end node. If the training computation process does not receive the corresponding data response within a preset time period, a communication anomaly is determined. Based on the upper-layer transport protocol, a model data request carrying the current address of the first-end node (i.e., the original address of the first-end node) and address fetch information is generated and sent to the second-end node. The second-end node parses the address fetch information, determines the changed connection address based on the changed IP address of the second-end node and / or the service process port number / node port number, and then generates a data response carrying the changed connection address. The target model data is encapsulated in this data response and sent together with the target transport connection to the training computation process of the first-end node.

[0054] In this embodiment of the application, before the second end obtains the corresponding target model data, the method further includes: S401: The second end responds to the model data request carrying the target random identifier by parsing the request header of the model data request and obtaining the header time information.

[0055] In practical applications, the request header of the model data request includes a header timestamp. The second-end node parses the request header to obtain the header timestamp, and then obtains the header time information.

[0056] S403: The second end determines the request delay for model data requests based on the packet header time information and the current time.

[0057] S405: The second end determines whether the request delay is greater than or equal to the timeout limit value corresponding to the model data request.

[0058] In practical applications, the request delay is the time interval between the packet header time and the current time. The timeout limit is either preset or calculated in real time; for the latter, the timeout limit varies depending on the data request model.

[0059] S407: If the request delay is greater than or equal to the timeout limit for the model data request, the second end discards the request data packet corresponding to the model data request.

[0060] In practical applications, the timeout limit represents the expiration time of a model data request. If the request delay is greater than or equal to the corresponding timeout limit, the second-end node determines that the model data request is expired, discards the corresponding request data packet, and does not execute the steps of obtaining the target model data and sending the corresponding data response. If the request delay is less than the corresponding timeout limit, the second-end node executes the steps of obtaining the corresponding target model data, and then generates and sends a data response carrying the target model data. In this way, congestion control can be achieved during data transmission, avoiding invalid requests and data blocking the transmission connection.

[0061] In practical applications, the request header for model data requests also includes information about the number of training samples; prior to S405, the method also included: S501: The second end determines the training time threshold and / or transmission delay threshold corresponding to the model data request based on the training sample size information.

[0062] S503: The second end determines the timeout limit value based on at least one of the training time consumption threshold and the transmission delay threshold.

[0063] Specifically, the training time threshold represents the maximum time required for the training computation process corresponding to the target transmission connection to train the model based on the training sample size information mentioned above. The transmission latency threshold represents the maximum time required for the training computation process to transmit data in the target transmission connection based on the training sample size information mentioned above. The calculation of the above training time threshold and / or transmission latency threshold can be obtained using existing deep learning models based on forward or backward computation, and this application does not impose any specific limitations.

[0064] Specifically, the timeout limit can be the smaller of the training time threshold and the transmission latency threshold, the larger of the two, a weighted average of the training time threshold and the transmission latency threshold, or a weighted sum of the training time threshold and the transmission latency threshold. It is understood that the aforementioned weighted average includes the average of the training time threshold and the transmission latency threshold, and the aforementioned weighted sum includes the sum of the training time threshold and the transmission latency threshold.

[0065] The timeout limit is determined based on the training sample size information, enabling independent control of data transmission for each transmission connection, as well as independent control of data transmission for each model data request within each transmission connection. Data streams formed by different transmission connections and different data requests within the same transmission connection are independent and do not interfere with each other. When multiple data streams are created simultaneously in the same transmission connection, each data stream can independently request a batch of model data and send the updated model data after iteration. Even if one data stream fails to transmit, it will not affect the data transmission of other model streams.

[0066] In some cases, data backup and transfer can be performed. Accordingly, following S407, the method may also include: S409: The second end sends a request failure message to the first end, the request failure message carrying a request identifier of the model data request where the request data packet was discarded.

[0067] S411: The first end responds to the request failure information by resending the model data request corresponding to the request identifier.

[0068] Specifically, after the second end generates a request failure message based on the upper-layer transport protocol, it sends the request failure message to the training computation process at the first end. The training computation process, responding to the request identifier carried in the request failure message, determines the failed model data request, and then regenerates the model data request based on the upper-layer transport protocol. It then sends the regenerated request to the original data service process via the originally established transport connection, or to another data service process that has established a transport connection with the training computation process. In some cases, the regenerated model data request includes not only the original request identifier but also retry encoding information, which can represent the number of retries for the request.

[0069] Based on some or all of the above implementation methods, in this embodiment of the application, before the second end discards the request data packet corresponding to the model data request, the method further includes: S601: The second end determines the current packet loss ratio based on the number of dropped packets and the number of arriving packets in the target transmission connection within the first unit time.

[0070] S603: The second end determines whether the current packet loss ratio is less than or equal to the preset packet loss ratio.

[0071] Specifically, the number of dropped packets is the number of request packets dropped by the second end node in the same transmission connection within the first unit time; the number of arriving packets is the total number of request packets arriving at the second end node through the same transmission connection within the first unit time. The packet loss ratio is the ratio of the number of dropped packets to the number of arriving packets in the same transmission connection within the first unit time. After determining that the request latency is greater than or equal to the timeout limit corresponding to the model data request, the second end node also calculates the ratio of the current number of dropped packets to the number of arriving packets in the target transmission connection to obtain the corresponding current packet loss ratio; and compares it with the preset packet loss ratio. If the current packet loss ratio is less than the preset packet loss ratio, the second end performs the step of dropping the request packets corresponding to the model data request; if the current packet loss ratio is greater than or equal to the preset packet loss ratio, it can be predicted that one or more of the current overall network latency, training time threshold, and transmission latency will increase. In this case, request packets are no longer dropped to avoid affecting the model training effect, and the second end performs the step of obtaining the corresponding target model data. For example, the preset packet loss ratio can be 60%.

[0072] In this way, the second end detects the number of dropped packets and the number of arriving packets in each transmission connection in real time. After determining that there is a request delay greater than or equal to the corresponding timeout limit in any transmission connection, it determines whether to continue to drop request data packets based on the current packet loss ratio. This achieves packet loss error correction control, optimizes model data transmission, and avoids excessive packet loss from affecting model training.

[0073] Based on some or all of the above implementation methods, in this embodiment of the application, before the second end discards the request data packet corresponding to the model data request, the method further includes: S701: When the second end detects that the number of arriving request packets in the target transmission connection within the second unit time is greater than or equal to the first packet quantity threshold, or the current load of the second end is greater than or equal to the first load threshold, it obtains the request order of the header time of the model data request in the target transmission connection.

[0074] In practical applications, the second unit time and the first unit time can be the same, or they can be set to different values ​​according to actual needs. The first packet volume threshold is a preset value or calculated in real time based on the current network conditions, representing the maximum number of arriving packets that the transmission connection can handle in the second unit time. The first load threshold is a preset maximum load that the second end node can handle, including CPU utilization and memory utilization, etc.

[0075] If the number of arriving packets is greater than or equal to a first packet volume threshold and / or the current load is greater than or equal to a first load threshold, the second end node sorts each transmission connection, that is, it groups and sorts the model data requests sent by the training computation processes corresponding to each transmission connection. Specifically, it sorts the requests according to the time sequence of the packet header timestamps of the model data requests that have arrived in each transmission connection, thus obtaining the request order for each transmission connection. Understandably, the earlier the packet header time, the higher the request's order.

[0076] S703: Determine whether the request sorting meets the preset discard conditions.

[0077] In practical applications, if the requested sorting time is greater than or equal to the sequence number threshold, the header time of the corresponding model data request is determined to have timed out, satisfying the preset discarding condition; otherwise, it is not satisfied. In some cases, the sequence number threshold can be preset. In other cases, the sequence number threshold can be determined based on a first packet volume threshold, such as a preset proportion of the first packet volume threshold, for example, 60% of the first packet volume threshold; or, the sequence number threshold can be determined based on a first load threshold, such as the average packet volume corresponding to each transmission connection when the current load is reduced to the first load threshold, which is used as the sequence number threshold. That is, when the packet volume in each transmission connection is below the average packet volume, the current load of the first end node can reach below the first load threshold.

[0078] S705: If the request sorting meets the preset discarding conditions, the second end describes the steps of discarding the request data packet corresponding to the model data request, and sends a first indication message to the first end to instruct the first end node corresponding to the target transmission connection to lower the sending window.

[0079] In practical applications, the first indication information includes the maximum receive window that the target transmission connection of the second end node can carry. In some cases, the first indication information also includes the header time or sequence number of the request data packet with the preset highest order (i.e., the latest header time) that the second end node discards in the target transmission connection.

[0080] S707: The first end responds to the first indication information by lowering the sending window of the first end node.

[0081] In practical applications, after receiving the first instruction information, the training computation process reduces the sending window in the target transmission connection to below the corresponding maximum receiving window, thereby reducing the data flow in the connection and achieving a reduction in traffic.

[0082] S709: When the second end detects that the number of arriving packets in the second unit time is less than the second packet volume threshold, or the current load of the second end is less than the second load threshold, it executes the step of obtaining the corresponding target model data and sends a second indication message to the first end to instruct the first end node to increase the sending window.

[0083] In practical applications, the second packet volume threshold is less than or equal to the first packet volume threshold. The second packet volume threshold can be set according to actual needs, and this application does not impose specific limitations. The second indication information includes the target receiving window corresponding to the second packet volume threshold. The second indication information also includes the header time or sequence number of the request data packet with the preset maximum order (i.e., the latest header time) that the second end node discards in the target transmission connection.

[0084] S711: The first end responds to the second instruction information by increasing the sending window of the first end node.

[0085] In practical applications, after receiving the second instruction information, the training computation process increases the sending window in the target transmission connection to above the corresponding target receiving window, thereby improving the data flow in the connection and realizing real-time traffic control.

[0086] In conjunction with the specific implementation methods described above, please refer to the embodiments of this application. Figure 3 and 4 Based on the aforementioned system structure and communication architecture, please refer to... Figure 5 The model data processing flow of the system in this application may specifically include: deploying the training computation process at the first end; deploying the model data server at the second end; establishing a transmission connection between the first end and the second end based on the upper-layer transmission protocol, such as a QUIC connection; assigning a random identifier (connection ID) to the transmission connection at the first end; the first end communicating with the second end for model data based on the established transmission connection; the training computation process at the first end training and updating the model data based on the received model data; and then the first end pushing the updated model data to the model data server at the second end for storage.

[0087] The following describes a model data processing method based on the above system, which is applied to the second end. Figure 6This is a flowchart illustrating a model data processing method provided in an embodiment of this application. This specification provides method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive methods, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or server product execution, the method can be executed sequentially according to the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown... Figure 6 As shown, the method may include: S801: In response to the connection request sent by the first end carrying a target random identifier, send a connection response to the first end for establishing a target transmission connection with the first end; the target random identifier is used to identify the target transmission connection; S803: In response to a model data request carrying a random identifier sent by the first end through the target transmission connection, obtain the corresponding target model data; S805: Send a data response carrying the target model data to the first end through the target transmission connection, so that the first end can perform corresponding model training based on the target model data; S807: When the model data request carries model update data generated by the first end, update the model database based on the model update data.

[0088] In this embodiment of the application, if the connection address corresponding to the target transmission connection changes, before S805, the method further includes: generating a data response carrying the changed connection address and target model data.

[0089] In this embodiment of the application, when the connection address changes to the address of the first end node corresponding to the target transmission connection, the model data request also carries the updated address of the first end node; before generating a data response carrying the changed connection address and the target model data, the method further includes: determining the changed connection address based on the updated address of the first end node and the current address of the second end node; the first end node is a node running on the first end for performing model training.

[0090] In this embodiment of the application, when the connection address changes to the address of the second end node corresponding to the target transmission connection, the model data request also carries the current address and address fetch information of the first end node; before generating a data response carrying the changed connection address and target model data, the method further includes: determining the changed connection address based on the address fetch information, according to the current address of the first end node and the updated address of the second end node; the second end node is a node running on the second end for performing model data services.

[0091] In this embodiment of the application, before obtaining the corresponding target model data, the method further includes: S809: In response to a model data request carrying a target random identifier, parse the request header of the model data request to obtain the header time information; S811: Determine the request delay for model data requests based on the header time information and the current time; S813: If the request delay is greater than or equal to the timeout limit for the model data request, discard the request data packet corresponding to the model data request.

[0092] In this embodiment of the application, the request header of the model data request also includes training sample size information; before S813, the method further includes: S815: Determine the training time threshold and / or transmission delay threshold corresponding to the model data request based on the training sample size information; S817: Determine the timeout limit based on at least one of the training time threshold and the transmission delay threshold.

[0093] In this embodiment of the application, before discarding the request data packet corresponding to the model data request, the method further includes: S819: Determine the current packet loss ratio based on the number of dropped and arrived packets of the request data packets in the target transmission connection within the first unit of time.

[0094] If the current packet loss ratio is less than the preset packet loss ratio, execute the step of discarding the request data packet corresponding to the model data request; if the current packet loss ratio is greater than or equal to the preset packet loss ratio, execute the step of obtaining the corresponding target model data.

[0095] In this embodiment of the application, before discarding the request data packet corresponding to the model data request, the method further includes: S821: If the number of arriving packets of the request data packet in the target transmission connection within the second unit time is greater than or equal to the first packet quantity threshold, or the current load of the second end is greater than or equal to the first load threshold, obtain the request order of the packet header time of the model data request in the target transmission connection. S823: If the request sorting meets the preset discarding conditions, execute the step of discarding the request data packet corresponding to the model data request, and send the first indication information to the first end to instruct the first end node corresponding to the target transmission connection to lower the sending window.

[0096] In this embodiment of the application, the method further includes: S823: If the number of arriving packets within the second unit time is less than the second packet volume threshold, or the current load of the second end is less than the second load threshold, the step of obtaining the corresponding target model data is executed, and a second indication message is sent to the first end to instruct the first end node to increase the sending window.

[0097] The following describes a model data processing method based on the above system, which is applied to the first end. Figure 7 This is a flowchart illustrating a model data processing method provided in an embodiment of this application. This specification provides method operation steps as shown in the embodiments or flowcharts, but based on conventional or non-inventive methods, more or fewer operation steps may be included. The order of steps listed in the embodiments is merely one possible execution order among many and does not represent the only possible execution order. In actual system or server product execution, the method can be executed sequentially according to the embodiments or drawings, or in parallel (e.g., in a parallel processor or multi-threaded processing environment). Specifically, as shown... Figure 7 As shown, the method may include: S901: Send a connection request carrying a target random identifier to the second end, so that the first end responds to the connection request by sending a connection response for establishing the target transmission connection; the target random identifier is used to identify the target transmission connection. S903: Send a model data request carrying a target random identifier to the second end through the target transmission connection so that the second end can obtain the corresponding target model data; S905: When model update data has been generated, the model data request also carries model update data so that the second end can update the model database based on the model update data; S907: In response to the data response carrying the target model data sent by the second end through the target transmission connection, perform corresponding model training based on the target model data.

[0098] In this embodiment of the application, the method further includes: S909: The first end reduces the sending window of the first end node in response to the first indication information. The first indication information is generated and sent by the second end when it detects that the number of arriving packets of requested data packets in the target transmission connection within a second unit time is greater than or equal to a first packet volume threshold, or the current load of the second end is greater than or equal to a first load threshold.

[0099] S911: The first end increases the sending window of the first end node in response to the second indication information. The second indication information is generated and sent by the second end when it detects that the number of arriving packets in a second unit time is less than a second packet volume threshold, or the current load of the second end is less than a second load threshold.

[0100] This application embodiment also provides a model data processing device 10, applied to a second end, such as... Figure 8 As shown, Figure 8 This illustration shows a schematic diagram of a model data processing apparatus provided in an embodiment of this application. The apparatus may include: Connection response sending module 11: In response to a connection request sent by the first end carrying a target random identifier, it sends a connection response to the first end to establish a target transmission connection with the first end; the target random identifier is used to identify the target transmission connection; Model data acquisition module 12: used to acquire the corresponding target model data in response to a model data request carrying a target random identifier sent by the first end through the target transmission connection; Data response sending module 13: used to send a data response carrying target model data to the first end through the target transmission connection, so that the first end can perform corresponding model training based on the target model data; Model data update module 14: Used to update the model database based on the model update data when the model data request carries model update data generated by the first end.

[0101] In this embodiment of the application, when the connection address corresponding to the target transmission connection changes, the method further includes a data response generation module: used to generate a data response carrying the changed connection address and target model data before sending a data response carrying target model data to the first end through the target transmission connection.

[0102] In this embodiment, when the connection address changes to the address of the second end node corresponding to the target transmission connection, the model data request also carries the current address and address retrieval information of the first end node; the device further includes a change address determination module: used to determine the changed connection address based on the address retrieval information, according to the current address of the first end node and the updated address of the second end node, before generating a data response carrying the changed connection address and target model data; the second end node is a node running on the second end for performing model data services.

[0103] In this embodiment of the application, the device further includes: Request header parsing module: Before obtaining the corresponding target model data, it responds to the model data request carrying the target random identifier by parsing the request header of the model data request to obtain the header time information; Request Delay Determination Module: Used to determine the request delay of model data requests based on the packet header time information and the current time; Packet Dropping Module: If the request delay is greater than or equal to the timeout limit for the model data request, the request packet corresponding to the model data request will be dropped.

[0104] In this embodiment of the application, the request header of the model data request also includes training sample size information; the device further includes: Time threshold determination module: used to determine the training time threshold and / or transmission delay threshold corresponding to the model data request based on the training sample size information before discarding the request data packet corresponding to the model data request if the request delay is greater than or equal to the timeout limit value corresponding to the model data request. Timeout upper limit determination module: used to determine the timeout upper limit based on at least one of the training time consumption threshold and the transmission delay threshold.

[0105] In this embodiment of the application, the apparatus further includes a packet loss ratio determination module: configured to determine the current packet loss ratio based on the number of dropped packets and the number of arriving packets in the target transmission connection within a first unit time before discarding the request data packet corresponding to the model data request. Furthermore, it is configured to, if the current packet loss ratio is less than a preset packet loss ratio, execute the step of discarding the request data packet corresponding to the model data request; and if the current packet loss ratio is greater than or equal to the preset packet loss ratio, execute the step of obtaining the corresponding target model data.

[0106] In this embodiment of the application, the device further includes: Request sorting acquisition module: Before discarding the request data packet corresponding to the model data request, if it is detected that the number of arriving request data packets in the target transmission connection within the second unit time is greater than or equal to the first packet quantity threshold, or the current load of the second end is greater than or equal to the first load threshold, the module acquires the request sorting of the packet header time of the model data request in the target transmission connection. First indication information sending module: If the request sorting meets the preset discarding conditions, execute the step of discarding the request data packet corresponding to the model data request, and send a first indication information to the first end to instruct the first end node corresponding to the target transmission connection to reduce the sending window.

[0107] In this embodiment of the application, the device further includes: The second instruction information sending module is used to execute the step of obtaining the corresponding target model data when the number of arriving packets within the second unit time is less than the second packet volume threshold, or the current load of the second end is less than the second load threshold, and to send the second instruction information to the first end to instruct the first end node to increase the sending window.

[0108] The above-described apparatus and method embodiments are based on the same implementation methods.

[0109] This application embodiment also provides a model data processing device 20, applied to the first end, such as... Figure 9 As shown, Figure 9 This illustration shows a schematic diagram of a model data processing apparatus provided in an embodiment of this application. The apparatus may include: Connection request sending module 21: used to send a connection request carrying a target random identifier to the second end, so that the first end responds to the connection request sending a connection response for establishing a target transmission connection; the target random identifier is used to identify the target transmission connection; Data request sending module 22: is used to send a model data request carrying a target random identifier to the second end through the target transmission connection so that the second end can obtain the corresponding target model data; and is used to, when model update data is generated, also carry model update data in the model data request so that the second end can update the model database based on the model update data; Model training module 23: Used to respond to the data response carrying the target model data sent by the second end through the target transmission connection, and to perform corresponding model training based on the target model data.

[0110] In this embodiment of the application, the device further includes: Sending window reduction module: Used to reduce the sending window of the first end node in response to the first indication information. The first indication information is generated and sent by the second end when it detects that the number of arriving packets of requested data packets in the target transmission connection within a second unit time is greater than or equal to a first packet volume threshold, or the current load of the second end is greater than or equal to a first load threshold.

[0111] Sending window addition module: Used to add a sending window for the first end node in response to a second indication message. The second indication message is generated and sent by the second end when it detects that the number of arriving packets within a second unit time is less than a second packet volume threshold, or that the current load of the second end is less than a second load threshold.

[0112] This application provides a model data processing device, which includes a processor and a memory. The memory stores at least one instruction or at least one program segment, which is loaded and executed by the processor to implement the model data processing method provided in the above method embodiments.

[0113] Memory is used to store software programs and modules. The processor executes these stored software programs and modules to perform various functional applications and data processing. Memory can primarily consist of a program storage area and a data storage area. The program storage area stores the operating system, application programs required for functionality, etc.; the data storage area stores data created based on device usage, etc. Furthermore, memory can include high-speed random access memory (RAM) and non-volatile memory, such as at least one disk storage device, flash memory device, or other volatile solid-state storage device. Accordingly, memory can also include a memory controller to provide the processor with access to the memory.

[0114] The methods and embodiments provided in this application can be executed on mobile terminals, computer terminals, servers, or similar computing devices. Taking running on a server as an example, Figure 10 This is a hardware structure block diagram of a server for a model data processing method provided in an embodiment of this application. For example... Figure 10 As shown, the server 800 can vary significantly due to different configurations or performance. It may include one or more central processing units (CPUs) 810 (CPUs 810 may include, but are not limited to, microprocessors such as MCUs or programmable logic devices such as FPGAs), a memory 830 for storing data, and one or more storage media 820 (e.g., one or more mass storage devices) for storing application programs 823 or data 822. The memory 830 and storage media 820 may be temporary or persistent storage. The program stored in the storage media 820 may include one or more modules, each module may include a series of instruction operations on the server. Furthermore, the CPU 810 may be configured to communicate with the storage media 820 and execute the series of instruction operations stored in the storage media 820 on the server 800. Server 800 may also include one or more power supplies 860, one or more wired or wireless network interfaces 850, one or more input / output interfaces 840, and / or one or more operating systems 821, such as Windows Server™, Mac OS X™, Unix™, Linux™, FreeBSD™, etc.

[0115] The input / output interface 840 can be used to receive or send data via a network. Specific examples of the network described above may include a wireless network provided by the communication provider of server 800. In one example, the input / output interface 840 includes a network interface controller (NIC), which can connect to other network devices via a base station to communicate with the Internet. In another example, the input / output interface 840 may be a radio frequency (RF) module used for wireless communication with the Internet.

[0116] Those skilled in the art will understand that Figure 10 The structure shown is for illustrative purposes only and does not limit the structure of the aforementioned electronic device. For example, server 800 may also include... Figure 10 The more or fewer components shown, or having the same Figure 10 The different configurations shown.

[0117] Embodiments of this application also provide a computer-readable storage medium, which can be disposed in a server to store at least one instruction or at least one program related to implementing a model data processing method in the method embodiments. The at least one instruction or the at least one program is loaded and executed by the processor to implement the model data processing method provided in the above method embodiments.

[0118] Optionally, in this embodiment, the storage medium may be located at at least one of the multiple network servers in a computer network. Optionally, in this embodiment, the storage medium may include, but is not limited to, various media capable of storing program code, such as USB flash drives, read-only memory (ROM), random access memory (RAM), portable hard drives, magnetic disks, or optical disks.

[0119] In the embodiments of this application, the model data processing method or apparatus disclosed in this application can operate in, for example, Figure 11 One or more servers may be configured to form a blockchain, which can then provide data storage and other services for model data processing methods or devices. For example, the model training data or sample data of the model training in this application may be stored in the aforementioned blockchain, and the servers are nodes on the blockchain. Figure 9 This is a schematic diagram of the structure of a blockchain system provided in an embodiment of this application. Figure 11 As shown, the server can be a node 920 in the distributed system 910, which can be a blockchain system. The blockchain system can be a distributed system formed by multiple nodes connected through network communication. The nodes can form a peer-to-peer (P2P) network. Any form of computer device, such as a server, terminal 02, or other electronic device, can become a node in the blockchain system by joining the peer-to-peer network. The blockchain includes a series of blocks that are sequentially generated according to their time sequence. Once a new block is added to the blockchain, it will not be removed. The blocks record the data submitted by the nodes in the blockchain system.

[0120] Blockchain is an emerging application model of computer technologies such as distributed data storage, peer-to-peer transmission, consensus mechanisms, and encryption algorithms. Essentially, it's a decentralized database, a chain of data blocks linked using cryptographic methods. Each data block contains information about a batch of network transactions, used to verify the validity of the information (anti-counterfeiting) and generate the next block. A blockchain can include a blockchain underlying platform, a platform product service layer, and an application service layer. The blockchain underlying platform can include processing modules such as user management, basic services, smart contracts, and operational monitoring. The user management module is responsible for managing the identity information of all blockchain participants, including maintaining public and private key generation (account management), key management, and maintaining the correspondence between user real identities and blockchain addresses (access management). Under authorization, it manages and audits transactions of certain real identities, providing risk control rule configuration (risk control audit). The basic service module is deployed on all blockchain node devices to verify the validity of business requests. After consensus is reached on valid requests, they are recorded in storage. For a new business request, the basic service first performs interface adaptation parsing and authentication (interface adaptation), and then encrypts the business information using a consensus algorithm (consensus management). The blockchain process involves several key steps: First, encryption and secure transmission of data to the shared ledger (network communication) for recording and storage. Second, the smart contract module handles contract registration, issuance, triggering, and execution. Developers can define contract logic using a programming language and publish it to the blockchain (contract registration). Execution is triggered by calling keys or other events based on the contract terms, completing the contract logic. The module also provides contract upgrade and cancellation functionality. Third, the operation and monitoring module handles deployment, configuration modification, contract settings, cloud adaptation, and real-time visualization of the product's operational status, including alerts, network status monitoring, and node device health monitoring. The platform's product service layer provides basic capabilities and implementation frameworks for typical applications. Developers can leverage these capabilities and add business characteristics to implement business logic on the blockchain. Finally, the application service layer provides blockchain-based application services to business stakeholders.

[0121] According to one aspect of this application, a computer program product or computer program is provided, comprising computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the methods provided in the various alternative implementations described above.

[0122] As can be seen from the embodiments of the model data processing method, apparatus, device, server, or storage medium provided in this application, in response to a connection request carrying a target random identifier sent by a first end, this application sends a connection response to the first end for establishing a target transmission connection with the first end; the target random identifier is used to identify the target transmission connection; in response to a model data request carrying a target random identifier sent by the first end through the target transmission connection, the corresponding target model data is obtained; a data response carrying the target model data is sent to the first end through the target transmission connection, so that the first end performs corresponding model training based on the target model data; when the model data request carries model update data generated by the first end, the model database is updated based on the model update data. The transmission connection establishment of this application has low connection latency and a short process, effectively improving connection efficiency. In multi-node transmission scenarios for model training, it can effectively improve data connection efficiency and transmission efficiency, thereby improving training efficiency.

[0123] It should be noted that the order of the embodiments described above is merely for descriptive purposes and does not represent the superiority or inferiority of the embodiments. Furthermore, the above description focuses on specific embodiments of this application. Other embodiments are within the scope of the appended claims. In some cases, the actions or steps described in the claims can be performed in a different order than that shown in the embodiments and still achieve the desired results. Additionally, the processes depicted in the drawings do not necessarily require a specific or sequential order to achieve the desired results. In some implementations, multitasking and parallel processing are also possible or may be advantageous.

[0124] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device, equipment, and storage medium embodiments are basically similar to the method embodiments, so the descriptions are relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0125] Those skilled in the art will understand that all or part of the steps of the above embodiments can be implemented by hardware, or by a program instructing the relevant hardware to implement them. The program can be stored in a computer-readable storage medium, such as a read-only memory, a disk, or an optical disk.

[0126] The above are merely preferred embodiments of this application and are not intended to limit this application. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the protection scope of this application.

Claims

1. A model data processing method, applied to a second end, characterized in that, The method includes: In response to a connection request sent by the first end, carrying a target random identifier and model data request information, a connection response is sent to the first end to establish a target transmission connection with the first end. The connection response also carries target model data corresponding to the model data request information, so that the first end can establish the target transmission connection based on the connection response during the initial communication with the second end. The target random identifier is used to identify the target transmission connection. The connection response includes a connection address, which includes the address of the first end node and the address of the second end node. In response to a model data request carrying a target random identifier sent by the first end through the target transmission connection, the corresponding target model data is obtained; if the address of the first end node corresponding to the target transmission connection changes, the model data request is a request carrying a changed connection address sent by the first end to the second end through the target transmission connection, and the changed connection address is determined by the first end based on the updated address of the first end node and the current address of the second end node; if the address of the second end node corresponding to the target transmission connection changes, and the first end node does not receive a data response corresponding to the current model data request within a preset time period, a model data request carrying the current address and address retrieval information of the first end node is generated and sent to the second end; The second end sends a data response carrying the target model data to the first end through the target transmission connection, so that the first end can perform corresponding model training based on the target model data; if the model data request carries the changed connection address, the data response carries the changed connection address and the target model data; if the model data request carries the address retrieval information, the second end determines the changed connection address based on the address retrieval information, according to the current address of the first end node and the updated address of the second end node, and generates a data response carrying the changed connection address and the target model data; If the model data request carries model update data generated by the first end, the model database is updated based on the model update data.

2. The method according to claim 1, characterized in that, In the event of a change in the connection address corresponding to the target transmission connection, before sending a data response carrying the target model data to the first end via the target transmission connection, the method further includes: Generate a data response carrying the modified connection address and the target model data.

3. The method according to claim 2, characterized in that, When the connection address changes to the address of the second end node corresponding to the target transmission connection, the model data request also carries the current address and address retrieval information of the first end node; Before generating the data response carrying the modified connection address and the target model data, the method further includes: Based on the address retrieval information, the changed connection address is determined according to the current address of the first end node and the updated address of the second end node.

4. The method according to claim 1, characterized in that, Before obtaining the corresponding target model data, the method further includes: In response to the model data request carrying the target random identifier, the request header of the model data request is parsed to obtain the header time information; The request delay for the model data request is determined based on the packet header time information and the current time; If the request delay is greater than or equal to the timeout limit corresponding to the model data request, the request data packet corresponding to the model data request is discarded.

5. The method according to claim 4, characterized in that, The request header of the model data request also includes training sample size information; before discarding the request data packet corresponding to the model data request if the request delay is greater than or equal to the timeout limit value corresponding to the model data request, the method further includes: Based on the training sample size information, determine the training time threshold and / or transmission latency threshold corresponding to the model data request; The timeout limit is determined based on at least one of the training time threshold and the transmission delay threshold.

6. The method according to claim 4 or 5, characterized in that, Before discarding the request data packet corresponding to the model data request, the method further includes: The current packet loss ratio is determined based on the number of dropped and arriving packets of the requested data packets in the target transmission connection within the first unit time. If the current packet loss ratio is less than the preset packet loss ratio, execute the step of discarding the request data packet corresponding to the model data request; If the current packet loss ratio is greater than or equal to the preset packet loss ratio, the step of obtaining the corresponding target model data is executed.

7. The method according to claim 4 or 5, characterized in that, Before discarding the request data packet corresponding to the model data request, the method further includes: If the number of arriving request packets in the target transmission connection within the second unit time is greater than or equal to the first packet volume threshold, or the current load of the second end is greater than or equal to the first load threshold, the request order of the packet header time of the model data request in the target transmission connection is obtained. If the request sorting meets the preset discarding conditions, the step of discarding the request data packet corresponding to the model data request is executed, and a first indication message is sent to the first end to instruct the first end node corresponding to the target transmission connection to reduce the sending window.

8. The method according to claim 7, characterized in that, The method further includes: If the number of arriving packets within the second unit time is less than the second packet volume threshold, or the current load of the second end is less than the second load threshold, the step of obtaining the corresponding target model data is executed, and a second indication message is sent to the first end to instruct the first end node to increase the sending window.

9. A model data processing method, applied to a first end, characterized in that, The method includes: A connection request carrying a target random identifier and model data request information is sent to the second end, so that the second end responds to the connection request by sending a connection response to establish a target transmission connection. The connection response also carries the target model data corresponding to the model data request information, so that the first end completes the establishment of the target transmission connection based on the connection response during the initial communication with the second end. The target random identifier is used to identify the target transmission connection. The connection response includes a connection address, which includes the address of the first end node and the address of the second end node. A model data request carrying the target random identifier is sent to the second end through the target transmission connection, so that the second end can obtain the corresponding target model data; if the address of the first end node corresponding to the target transmission connection changes, the model data request is a request carrying the changed connection address sent by the first end to the second end through the target transmission connection, and the changed connection address is determined by the first end based on the updated address of the first end node and the current address of the second end node; if the address of the second end node corresponding to the target transmission connection changes, and the first end node does not receive the data response corresponding to the current model data request within a preset time period, a request carrying the first end node's random identifier is generated. The system requests model data with the current address of the node and address retrieval information, and sends this data to the second end. If model update data has been generated, the model data request also carries the model update data, so that the second end updates the model database based on the model update data. If the model data request carries the changed connection address, the data response carries the changed connection address and the target model data. If the model data request carries the address retrieval information, the second end determines the changed connection address based on the address retrieval information, according to the current address of the first end node and the updated address of the second end node, and generates a data response carrying the changed connection address and the target model data. In response to a data response carrying the target model data sent by the second end through the target transmission connection, corresponding model training is performed based on the target model data.

10. The method according to claim 9, characterized in that, The method further includes: In response to the first indication information, the sending window of the first end node is reduced; the first indication information is generated and sent by the second end when it detects that the number of arriving packets of the requested data packets in the target transmission connection within a second unit time is greater than or equal to a first packet quantity threshold, or the current load of the second end is greater than or equal to a first load threshold. The sending window of the first end node is increased in response to the second indication information; the second indication information is generated and sent by the second end when it detects that the number of arriving packets in the second unit time is less than the second packet volume threshold, or the current load of the second end is less than the second load threshold.

11. A model data processing device, applied at a second end, characterized in that, The device includes: Connection response sending module: In response to a connection request sent by the first end carrying a target random identifier and model data request information, the module sends a connection response to the first end to establish a target transmission connection with the first end. The connection response also carries target model data corresponding to the model data request information, so that the first end can establish the target transmission connection based on the connection response during the initial communication with the second end. The target random identifier is used to identify the target transmission connection. The connection response includes a connection address, which includes the address of the first end node and the address of the second end node. Model data acquisition module: In response to a model data request carrying a random identifier sent by the first end through the target transmission connection, the module acquires the corresponding target model data; in the case of a change in the address of the first end node corresponding to the target transmission connection, the model data request is a request carrying a changed connection address sent by the first end to the second end through the target transmission connection, the changed connection address being determined by the first end based on the updated address of the first end node and the current address of the second end node; in the case of a change in the address of the second end node corresponding to the target transmission connection, and the first end node not receiving a data response corresponding to the current model data request within a preset time period, the module generates a model data request carrying the current address and address retrieval information of the first end node, and sends it to the second end; Data response sending module: used to send a data response carrying the target model data to the first end through the target transmission connection, so that the first end can perform corresponding model training based on the target model data; if the model data request carries the changed connection address, the data response carries the changed connection address and the target model data; if the model data request carries the address retrieval information, the second end determines the changed connection address based on the address retrieval information, according to the current address of the first end node and the updated address of the second end node, and generates a data response carrying the changed connection address and the target model data; Model data update module: used to update the model database based on the model update data when the model data request carries model update data generated by the first end.

12. The apparatus according to claim 11, characterized in that, In the event of a change in the connection address corresponding to the target transmission connection, the device further includes a data response generation module: This is used to generate a data response carrying the modified connection address and the target model data before sending the data response carrying the target model data to the first end through the target transmission connection.

13. The apparatus according to claim 12, characterized in that, When the connection address changes to the address of the second end node corresponding to the target transmission connection, the model data request also carries the current address and address retrieval information of the first end node; the device further includes a change address determination module: Before generating the data response carrying the changed connection address and the target model data, the changed connection address is determined based on the address retrieval information, according to the current address of the first end node and the updated address of the second end node.

14. The apparatus according to claim 11, characterized in that, The device further includes: Request header parsing module: used to, in response to the model data request carrying the random identifier of the target, parse the request header of the model data request to obtain the header time information before obtaining the corresponding target model data; Request delay determination module: used to determine the request delay of the model data request based on the packet header time information and the current time; Packet discarding module: If the request delay is greater than or equal to the timeout limit value corresponding to the model data request, discard the request data packet corresponding to the model data request.

15. The apparatus according to claim 14, characterized in that, The request header of the model data request also includes training sample size information; the device further includes: Time threshold determination module: used to determine the training time threshold and / or transmission delay threshold corresponding to the model data request based on the training sample size information before discarding the request data packet corresponding to the model data request if the request delay is greater than or equal to the timeout limit value corresponding to the model data request; Timeout upper limit determination module: used to determine the timeout upper limit based on at least one of the training time consumption threshold and the transmission delay threshold.

16. The apparatus according to claim 14 or 15, characterized in that, The device further includes a packet loss ratio determination module, used for: Before discarding the request data packet corresponding to the model data request, the current packet loss ratio is determined based on the number of dropped packets and the number of arriving packets in the target transmission connection within a first unit time. If the current packet loss ratio is less than the preset packet loss ratio, execute the step of discarding the request data packet corresponding to the model data request; If the current packet loss ratio is greater than or equal to the preset packet loss ratio, the step of obtaining the corresponding target model data is executed.

17. The apparatus according to claim 14 or 15, characterized in that, The device further includes: Request sorting acquisition module: Before discarding the request data packet corresponding to the model data request, if it is detected that the number of arriving request data packets in the target transmission connection within a second unit time is greater than or equal to a first packet quantity threshold, or the current load of the second end is greater than or equal to a first load threshold, the module acquires the request sorting of the packet header time of the model data request in the target transmission connection. First indication information sending module: if the request sorting meets the preset discarding conditions, execute the step of discarding the request data packet corresponding to the model data request, and send first indication information to the first end to instruct the first end node corresponding to the target transmission connection to reduce the sending window.

18. The apparatus according to claim 17, characterized in that, The device further includes: The second instruction information sending module is used to execute the step of obtaining the corresponding target model data when it is detected that the number of arriving packets in the second unit time is less than the second packet volume threshold, or the current load of the second end is less than the second load threshold, and to send the second instruction information to the first end to instruct the first end node to increase the sending window.

19. A model data processing device, applied at a first end, characterized in that, The device includes: A connection request sending module is used to send a connection request carrying a target random identifier and model data request information to the second end, so that the second end responds to the connection request by sending a connection response for establishing a target transmission connection. The connection response also carries target model data corresponding to the model data request information, so that the first end completes the establishment of the target transmission connection based on the connection response during the initial communication with the second end. The target random identifier is used to identify the target transmission connection. The connection response includes a connection address, which includes the address of the first end node and the address of the second end node. A data request sending module is configured to send a model data request carrying the target random identifier to the second end through the target transmission connection, so that the second end can obtain the corresponding target model data; and, when model update data is generated, the model data request also carries the model update data, so that the second end can update the model database based on the model update data; if the address of the first end node corresponding to the target transmission connection changes, the model data request is a request carrying the changed connection address sent by the first end to the second end through the target transmission connection, the changed connection address being determined by the first end based on the updated address of the first end node and the current address of the second end node; if the The model data request carries the changed connection address, and the data response carries the changed connection address and the target model data. If the address of the second end node corresponding to the target transmission connection changes, and the first end node does not receive the data response corresponding to the current model data request within a preset time period, a model data request carrying the current address and address retrieval information of the first end node is generated and sent to the second end. If the model data request carries the address retrieval information, the second end determines the changed connection address based on the address retrieval information, according to the current address of the first end node and the updated address of the second end node, and generates a data response carrying the changed connection address and the target model data. Model training module: used to respond to the data response carrying the target model data sent by the second end through the target transmission connection, and to perform corresponding model training based on the target model data.

20. The apparatus according to claim 19, characterized in that, The device further includes: Sending window reduction module: used to reduce the sending window of the first end node in response to the first indication information; the first indication information is generated and sent by the second end when it detects that the number of arriving packets of the requested data packets in the target transmission connection within a second unit time is greater than or equal to the first packet quantity threshold, or the current load of the second end is greater than or equal to the first load threshold; Sending window addition module: used to add a sending window for the first end node in response to the second indication information; the second indication information is generated and sent by the second end when it detects that the number of arriving packets in the second unit time is less than the second packet volume threshold, or the current load of the second end is less than the second load threshold.

21. A computer-readable storage medium, characterized in that, The storage medium stores at least one instruction or at least one program segment, which is loaded and executed by a processor to implement the model data processing method as described in any one of claims 1-10.

22. A computer device, the device comprising a processor and a memory, characterized in that, The memory stores at least one instruction or at least one program segment, which is loaded and executed by the processor to implement the model data processing method as described in any one of claims 1-10.

23. A computer program product or computer program, characterized in that, The computer program product or computer program includes computer instructions that, when executed by a processor, implement the model data processing method as described in any one of claims 1-10.

Images