A method, device and electronic device for quickly shielding faulty graphics cards in a cluster

By setting the graphics card failure threshold and shielding duration, the failed graphics cards in the artificial intelligence system are counted and blocked, which solves the problem of failed business request processing caused by the failed graphics card, and improves system stability and user experience.

CN112540880BActive Publication Date: 2024-11-15ZUOYEBANG EDUCATION TECH (BEIJING) CO LTD
View PDF 12 Cites 0 Cited by

Patent Information

Application Number
CN202011522554.X
Authority / Receiving Office
CN · China
Patent Type
Patents(China)
Current Assignee / Owner
Filing Date
2020-12-22
Publication Date
2024-11-15
Estimated Expiration
2040-12-22

AI Technical Summary

Technical Problem

The existing technology cannot quickly block the failed graphics card in the artificial intelligence system, resulting in the failure of business request processing and giving the user a poor experience.

Method used

By pre-setting the graphics card failure threshold, each graphics card in the cluster is counted for faults, and when the number of failures exceeds the threshold, the service requests of the failed graphics card are forwarded to other graphics cards, and the graphics card blocking time is set to resume use.

Benefits of technology

Timely shielding of faulty graphics cards is achieved, system stability is ensured, and user experience and satisfaction are improved.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN112540880B_ABST
    Figure CN112540880B_ABST
Patent Text Reader

Abstract

The present invention belongs to the field of data information processing technology, and provides a method, device, electronic device and recording medium for quickly shielding faulty graphics cards in a cluster, wherein the method comprises: presetting a graphics card fault threshold, counting the faults occurring in each graphics card in the cluster; when the number of faults exceeds the pre-set graphics card fault threshold, shielding the graphics card. The present invention counts the faults of each graphics card, shields the faulty graphics card in time, and transfers the request sent to the faulty graphics card to other graphics cards, thereby ensuring the stability of the system and improving the user experience and satisfaction.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present invention belongs to the technical field of data information processing, and is particularly suitable for real-time data information processing in artificial intelligence. More specifically, it relates to a method, device and electronic equipment for quickly shielding faulty graphics cards in a cluster. Background Art

[0002] With the development of Internet technology and artificial intelligence technology, artificial intelligence is increasingly used to process data information in online services.

[0003] Currently, the amount of data and information that artificial intelligence needs to process is very large. Due to different architectural designs, graphics cards have much higher floating-point computing capabilities than central processing units, so graphics cards are usually used to process data. As the business expands, a large number of graphics cards will need to be added to support artificial intelligence operations.

[0004] However, graphics cards often have poor stability due to high-load computing, and often fail to process data. If not blocked in time, a large number of business requests will fail to be processed, giving users a very bad experience. Summary of the invention

[0005] 1. Technical issues to be resolved

[0006] The present invention aims to solve the problem that in existing online services using artificial intelligence, faulty graphics cards cannot be quickly shielded, resulting in failure in processing business requests and a poor user experience.

[0007] (II) Technical solution

[0008] In order to solve the above technical problems, one aspect of the present invention provides a method for quickly shielding a faulty graphics card in a cluster, comprising:

[0009] Pre-set the graphics card failure threshold and count the failures of each graphics card in the cluster;

[0010] When the number of faults exceeds a preset graphics card fault threshold, the graphics card is shielded.

[0011] According to a preferred embodiment of the present invention, the graphics card is a physical graphics card, and the faults of each graphics card are counted using inter-process communication.

[0012] According to a preferred embodiment of the present invention, the inter-process communication is a shared memory communication mode.

[0013] According to a preferred embodiment of the present invention, counting the failures of each graphics card in the cluster is specifically as follows:

[0014] If a request processing failure message is received from the graphics card, the fault count of the graphics card is increased by 1;

[0015] If the message returned is that the request is processed successfully, the fault count of the graphics card will be reduced by 1.

[0016] According to a preferred embodiment of the present invention, the shielding operation on the graphics card specifically includes shielding the address of the graphics card and forwarding the service request sent to the graphics card to other graphics cards.

[0017] According to a preferred embodiment of the present invention, a shielding time of the graphics card is set. If a shielding operation is performed on the graphics card within the shielding time of the graphics card, a service request is allowed to be sent to the graphics card when the time length exceeds the shielding time of the graphics card.

[0018] A second aspect of the present invention provides a device for quickly shielding a faulty graphics card in a cluster, comprising:

[0019] The fault counting module pre-sets the graphics card fault threshold and counts the faults of each graphics card in the cluster;

[0020] The graphics card shielding module shields the graphics card when the number of faults exceeds a preset graphics card fault threshold.

[0021] According to a preferred embodiment of the present invention, the graphics card is a physical graphics card, and the faults of each graphics card are counted using inter-process communication.

[0022] According to a preferred embodiment of the present invention, the inter-process communication is a shared memory communication mode.

[0023] According to a preferred embodiment of the present invention, counting the failures of each graphics card in the cluster is specifically as follows:

[0024] If a request processing failure message is received from the graphics card, the fault count of the graphics card is increased by 1;

[0025] If the message returned is that the request is processed successfully, the fault count of the graphics card will be reduced by 1.

[0026] According to a preferred embodiment of the present invention, the shielding operation on the graphics card specifically includes shielding the address of the graphics card and forwarding the service request sent to the graphics card to other graphics cards.

[0027] According to a preferred embodiment of the present invention, a shielding time of the graphics card is set. If a shielding operation is performed on the graphics card within the shielding time of the graphics card, a service request is allowed to be sent to the graphics card when the time length exceeds the shielding time of the graphics card.

[0028] A third aspect of the present invention provides an electronic device, comprising a processor and a memory, wherein the memory is used to store a computer executable program, and when the computer program is executed by the processor, the processor executes the described method.

[0029] A fourth aspect of the present invention further proposes a computer-readable medium storing a computer-executable program, wherein when the computer-executable program is executed, the method described is implemented.

[0030] (III) Beneficial effects

[0031] The present invention counts the faults of each graphics card, shields the faulty graphics card in time, and transfers the requests sent to the faulty graphics card to other graphics cards, thereby ensuring the stability of the system and improving the user experience and satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0032] Figure 1 is a schematic diagram of a scenario of a task processing system provided by an embodiment of the present invention;

[0033] Figure 2 It is a flowchart of a method for quickly shielding a faulty graphics card in a cluster according to an embodiment of the present invention;

[0034] Figure 3 It is a flow chart of a client service request processing flow according to an embodiment of the present invention;

[0035] Figure 4 is a schematic diagram of a process of a client detecting whether a graphics card is available according to an embodiment of the present invention;

[0036] Figure 5 It is a schematic diagram of the structure of a device for quickly shielding a faulty graphics card in a cluster according to an embodiment of the present invention;

[0037] Figure 6 is a schematic structural diagram of an electronic device according to an embodiment of the present invention;

[0038] Figure 7 is a schematic diagram of a computer-readable recording medium according to an embodiment of the present invention. DETAILED DESCRIPTION

[0039] In the process of introducing specific embodiments, the detailed description of the structure, performance, effect or other characteristics is to enable those skilled in the art to fully understand the embodiments. However, it does not exclude that those skilled in the art can implement the present invention with a technical solution that does not contain the above-mentioned structure, performance, effect or other characteristics under certain circumstances.

[0040] The flowchart in the accompanying drawings is only an exemplary process demonstration, and does not mean that the solution of the present invention must include all the contents, operations and steps in the flowchart, nor does it mean that it must be executed in the order shown in the figure. For example, some operations / steps in the flowchart can be decomposed, some operations / steps can be combined or partially combined, etc. The execution order shown in the flowchart can be changed according to the actual situation without departing from the main purpose of the present invention.

[0041] Frames in the attached figure Figure 1 Generally, it refers to functional entities, which do not necessarily correspond to physically independent entities. That is, these functional entities can be implemented in software form, or in one or more hardware modules or integrated circuits, or in different networks and / or processing unit devices and / or microcontroller devices.

[0042] The same reference numerals in the drawings represent the same or similar elements, components or parts, and thus the repeated description of the same or similar elements, components or parts may be omitted below. It should also be understood that although the first, second, third and other attributives representing the numbers may be used herein to describe various devices, elements, components or parts, these devices, elements, components or parts should not be limited by these attributives. In other words, these attributives are only used to distinguish one from another. For example, the first device may also be called the second device, but it does not deviate from the essential technical solution of the present invention. In addition, the terms "and / or" and "and / or" refer to all combinations including any one or more of the listed items.

[0043] In recent years, as artificial intelligence (AI) and image processing have developed more and more rapidly, the requirements for graphics card-based computing systems have also increased. The computing systems need to be able to handle task requests of different sizes and be able to process tasks efficiently whether they are busy or not.

[0044] Graphics card, also known as GPU, display core, visual processor, display chip, is a microprocessor that specializes in image computing on personal computers, workstations, game consoles and some mobile devices (such as tablets, smart phones, etc.). Its purpose is to convert and drive the display information required by the computer system, and provide line scanning signals to the display to control the correct display of the display. It is an important component that connects the display and the motherboard of the personal computer. Since the graphics card's ability to process floating-point operations is much higher than that of the central processing unit CPU, now artificial intelligence and machine learning scenes that require a lot of computing are all calculated through graphics cards. The computing efficiency is much higher than that of the central processing unit CPU, and it has gradually become an indispensable device in artificial intelligence.

[0045] Figure 1 A schematic diagram of a task processing system provided in an embodiment of the present application is shown in FIG. Figure 1 As shown, the task processing system includes multiple servers 101, 102, ..., 1XX, each server is provided with multiple graphics cards, in this embodiment, each server is provided with 8 graphics cards, and in other embodiments, 2 graphics cards or 4 graphics cards may also be provided.

[0046] The system includes multiple clients 201, 202, ..., 2XX, where the client can be a smart phone, a tablet computer, a laptop computer, a desktop computer, a smart speaker, a smart watch, etc., but is not limited thereto. The terminal and the server can be directly or indirectly connected via wired or wireless communication, which is not limited in the embodiments of the present application.

[0047] In this embodiment, the client sends a request to the graphics card in the server according to the task type. For example, server 101 is used to identify the user's emotions, and server 102 is used to calculate the user's resource quota, etc. The model type contained in each server can be the same or different. The request sent by the client to the graphics card includes the data to be calculated. The graphics card retrieves the model and parameters from the AI ​​calculation model and inputs the data contained in the request into the model for calculation. After the calculation is completed, the graphics card returns the calculation result to the client.

[0048] The graphics cards in the client and the server are not one-to-one corresponding, and each time they are sent to the graphics cards in different servers according to different tasks. For example, client 201 sends a request to graphics cards 1011 and 1018 in server 101 and graphics card 1022 in server 102, and client 203 sends a request to graphics cards 1022 and 1028 in server 102 and graphics card 1XX8 in server 1XX.

[0049] In traditional task processing systems, the server regularly performs health checks on the performance and status of the graphics card. If the performance or status of the graphics card is found to be abnormal, the operation of the graphics card will be stopped. However, since the health check cycle is relatively long, such as 30 seconds or 1 minute, and the task processing time is often in nanoseconds, when the graphics card fails and the health check time is not reached, a large number of requests sent to the graphics card will fail to be processed, and the client cannot obtain the correct processing results, which brings a very bad experience to users.

[0050] In order to solve the above technical problems, the present invention proposes a method for quickly shielding faulty graphics cards in a cluster. The method flow chart is as follows: Figure 2 As shown, including:

[0051] S101. Preset a graphics card failure threshold, and count the failures of each graphics card in the cluster.

[0052] Further on the basis of the above technical solution, the graphics card is a physical graphics card, and the faults of each graphics card are counted by using inter-process communication.

[0053] In this embodiment, the graphics card used is a physical graphics card, and the cluster providing services to users includes multiple servers, each of which is equipped with multiple graphics cards. Each server is connected to multiple clients, and uses the internally set graphics card to process the service requests sent by the client, and each service request corresponds to a process.

[0054] Use IP address + port number to assign an address to each graphics card. The client sends a service request to the graphics card through this address. If the graphics card is operating normally, the correct processing result will be returned. If the graphics card is abnormal, an error number will be returned. Inter-process communication statistics are used to enable different processes within the same server to communicate with each other. The status of the same graphics card returned to different processes can be counted, so as to quickly determine the status of the graphics card.

[0055] In this implementation, the graphics card fault threshold is set to 60. The graphics card fault threshold can be adjusted according to the business traffic. The graphics card fault threshold can be increased if the business traffic is large, and can be reduced if the business traffic is reduced.

[0056] Further on the basis of the above technical solution, the inter-process communication is a shared memory communication mode.

[0057] In this embodiment, the data structure of the shared memory is a HashTable. This area of ​​the shared memory is visible to all processes processed by the same server. When the process obtains the information returned by the graphics card, the status of the graphics card is recorded in the area of ​​the shared memory.

[0058] On the basis of the above technical solution, the failures of each graphics card in the cluster are further counted as follows:

[0059] If a request processing failure message is received from the graphics card, the fault count of the graphics card is increased by 1;

[0060] If the message returned is that the request is processed successfully, the fault count of the graphics card will be reduced by 1.

[0061] In this embodiment, in many cases, the error code returned by the graphics card may not be caused by a graphics card failure, but may be caused by a model operation error. Therefore, in order to ensure the utilization rate of the graphics card, when the graphics card returns an error code, that is, the request processing fails, it is written in the shared memory, and the failure count of the graphics card is increased by 1. If the graphics card returns the correct processing result, it means that the graphics card request processing is successful. At this time, the shared memory is also written, and the failure count of the graphics card is reduced by 1. In this way, only when the graphics card frequently fails to process requests in a short period of time is it considered a graphics card failure, which ensures the stability of the system, the user is not aware of it, and the graphics card can be fully utilized for calculation processing.

[0062] In its implementation mode, when the graphics card returns an error code, ie, the request processing fails, the fault count of the graphics card is written in the shared memory and increased by 2. In this way, the faulty graphics card can be quickly shielded.

[0063] S102: When the number of faults exceeds a preset graphics card fault threshold, shielding the graphics card.

[0064] In this embodiment, when the client process wants to send a request to the graphics card, it first reads the graphics card fault count of the graphics card to determine whether it exceeds the set threshold. If it exceeds the preset graphics card fault threshold, no request is sent to the graphics card.

[0065] Further on the basis of the above technical solution, the shielding operation on the graphics card specifically includes shielding the address of the graphics card and forwarding the service request sent to the graphics card to other graphics cards.

[0066] In this embodiment, since the graphics card is located in the form of "IP address + port", when the fault count of the graphics card in the shared memory exceeds the preset graphics card fault threshold, the system will no longer allocate the address of the graphics card to the client, and send the address of other graphics cards whose fault counts do not exceed the preset graphics card fault threshold to the client. In other embodiments, load balancing is also performed on other graphics cards, and the number of currently processed requests of other graphics cards whose fault counts do not exceed the preset graphics card fault threshold is determined, and service requests are preferentially sent to the graphics card with the least number of processed requests.

[0067] On the basis of the above technical solution, a shielding time of the graphics card is further set. If a shielding operation is performed on the graphics card within the shielding time of the graphics card, a service request is allowed to be sent to the graphics card when the time length exceeds the shielding time of the graphics card.

[0068] In this embodiment, the shielding time of the graphics card is set. When the graphics card fault count of the graphics card reaches the set graphics card fault threshold, the graphics card is shielded and the timing starts. When the shielding time reaches the set graphics card shielding time, an attempt is made to resend the processing request to the graphics card. At this time, the graphics card fault count is retained. If the service request is successfully processed, the graphics card fault count is reduced by 1 in the shared memory. If it fails, the graphics card is shielded again.

[0069] Because some faults are recoverable minor faults, for example, the graphics card may overheat, resulting in reduced computing power, causing the business request processing to fail. The graphics card will return to normal after cooling down. Therefore, the shielding time should not be set too long. The shielding time ranges from 30 seconds to 5 minutes, usually 1 minute.

[0070] In other implementations, when the graphics card is still in the shielding period, a small proportion of trial can be used to try to send a service request to the graphics card to determine whether the graphics card has recovered. For example, the graphics card can usually process 10 service requests at the same time, and only one service request is sent to the graphics card at the same time for trial. If the graphics card returns a successful service processing message, the shielding of the graphics card is canceled, and the graphics card fault count in the shared memory is reduced by 1.

[0071] The following is a detailed description of the client's service request processing flow through an embodiment. Figure 3 shown.

[0072] Embodiment 1

[0073] S201: The client has a business request to be processed.

[0074] S202: Allocate a graphics card to the client according to load balancing, and the client obtains the address of the graphics card.

[0075] When the client has a business request to be processed, it will send a command to the system, indicating that it has a business request to be processed. The command content includes the business type, the model to be used, and the priority. The system determines the operation time of the business request based on the command content and obtains the business pressure of each graphics card in the current server. The graphics card is allocated to the client according to the load balancing, and the address of the graphics card is sent to the client in the form of IP+port, such as 192.168.50.174:22747.

[0076] When allocating a graphics card to a client, the address of a graphics card whose graphics card failure count exceeds a set graphics card failure threshold is shielded by the system, and the address of the graphics card is not sent to the client.

[0077] S203, the client detects whether the graphics card is available, and if the graphics card is available, execute step S205, otherwise execute step S204.

[0078] The client queries the corresponding shared memory based on the graphics card address to detect whether the graphics card is available.

[0079] S204: Determine whether a retry is required, if yes, return to step S202, otherwise execute step S209.

[0080] When the client detects that the graphics card is unavailable, it can retry according to the needs of the business request, and the system will reallocate the graphics card. If no retry is needed, the process ends directly.

[0081] S205: The client sends a service request to the graphics card.

[0082] The client sends a service request to the graphics card based on the graphics card address. The service request includes operation parameters and model (algorithm).

[0083] S206: The graphics card processes the service request. If the processing is successful, execute step S207; otherwise, execute step S208.

[0084] S207: The client prints the log, writes the result into the shared memory, and reduces the graphics card fault count by 1.

[0085] S208. The client prints the log, writes the result into the shared memory, and increases the graphics card fault count by 1.

[0086] The graphics card processes the business request according to the content of the business request. If the processing is successful, it returns the business processing result. The client writes the result into the shared memory, and the graphics card fault count of the graphics card is reduced by 1. Otherwise, the graphics card returns an error code, and the client adds 1 to the graphics card fault count of the graphics card in the shared memory.

[0087] S209: End. The client service request processing process is terminated.

[0088] The following is a second embodiment to specifically describe the process of S203 client detecting whether the graphics card is available. The flow chart is as follows: Figure 4 shown.

[0089] Embodiment 2

[0090] S2031. Query the shared memory according to the graphics card address.

[0091] The graphics cards are located in the form of "IP address + port number". In the system, a part of the shared memory HashTable code is set for each graphics card in the server to record the graphics card failure count. The corresponding shared memory can be queried through the address of the graphics card.

[0092] S2032. Determine whether there is a record in the shared memory. If there is no record, execute S2035. If there is a record, execute S2033.

[0093] Query the corresponding shared memory according to the address of the graphics card to determine whether there is a record. If there is no record, it means that the graphics card has never failed before or is being used for the first time and does not need to be blocked, indicating that the graphics card is available.

[0094] S2033. If there is a record, determine whether the graphics card failure count in the record exceeds the set graphics card failure threshold. If yes, execute S2034; otherwise, execute S2035.

[0095] S2034, determine whether the current time has exceeded the shielding expiration time, if yes, execute S2035, otherwise execute S2036.

[0096] The shielding expiration time is the time when the shielding of the graphics card is canceled. For example, if the shielding operation is performed on the graphics card at 14:05 and the shielding duration is set to 1 minute, the shielding expiration time is 14:06. Therefore, if the current time exceeds the shielding expiration time, it means that the graphics card is no longer shielded, so step S2035 is executed to determine that the graphics card is available.

[0097] S2035, graphics card is available.

[0098] S2036, Graphics card is unavailable. Figure 5 A device 500 for quickly shielding a faulty graphics card in a cluster according to an embodiment of the present invention comprises:

[0099] The fault counting module 501 pre-sets a graphics card fault threshold and counts the faults occurring in each graphics card in the cluster.

[0100] Further on the basis of the above technical solution, the graphics card is a physical graphics card, and the faults of each graphics card are counted by using inter-process communication.

[0101] In this embodiment, the graphics card used is a physical graphics card, and the cluster providing services to users includes multiple servers, each of which is equipped with multiple graphics cards. Each server is connected to multiple clients, and uses the internally set graphics card to process the service requests sent by the client, and each service request corresponds to a process.

[0102] Use IP address + port number to assign an address to each graphics card. The client sends a service request to the graphics card through this address. If the graphics card is operating normally, the correct processing result will be returned. If the graphics card is abnormal, an error number will be returned. Inter-process communication statistics are used to enable different processes within the same server to communicate with each other. The status of the same graphics card returned to different processes can be counted, so as to quickly determine the status of the graphics card.

[0103] In this implementation, the graphics card fault threshold is set to 60. The graphics card fault threshold can be adjusted according to the business traffic. The graphics card fault threshold can be increased if the business traffic is large, and can be reduced if the business traffic is reduced.

[0104] Further on the basis of the above technical solution, the inter-process communication is a shared memory communication mode.

[0105] In this embodiment, the data structure of the shared memory is a HashTable. This area of ​​the shared memory is visible to all processes processed by the same server. When the process obtains the information returned by the graphics card, the status of the graphics card is recorded in the area of ​​the shared memory.

[0106] On the basis of the above technical solution, the failures of each graphics card in the cluster are further counted as follows:

[0107] If a request processing failure message is received from the graphics card, the fault count of the graphics card is increased by 1;

[0108] If the message returned is that the request is processed successfully, the fault count of the graphics card will be reduced by 1.

[0109] In this embodiment, in many cases, the error code returned by the graphics card may not be caused by a graphics card failure, but may be caused by a model operation error. Therefore, in order to ensure the utilization rate of the graphics card, when the graphics card returns an error code, that is, the request processing fails, it is written in the shared memory, and the failure count of the graphics card is increased by 1. If the graphics card returns the correct processing result, it means that the graphics card request processing is successful. At this time, the shared memory is also written, and the failure count of the graphics card is reduced by 1. In this way, only when the graphics card frequently fails to process requests in a short period of time is it considered a graphics card failure, which ensures the stability of the system, the user is not aware of it, and the graphics card can be fully utilized for calculation processing.

[0110] In its implementation mode, when the graphics card returns an error code, ie, the request processing fails, the fault count of the graphics card is written in the shared memory and increased by 2. In this way, the faulty graphics card can be quickly shielded.

[0111] The graphics card shielding module 502 shields the graphics card when the number of faults exceeds a preset graphics card fault threshold.

[0112] In this embodiment, when the client process wants to send a request to the graphics card, it first reads the graphics card fault count of the graphics card to determine whether it exceeds the set threshold. If it exceeds the preset graphics card fault threshold, no request is sent to the graphics card.

[0113] Further on the basis of the above technical solution, the shielding operation on the graphics card specifically includes shielding the address of the graphics card and forwarding the request sent to the graphics card to other graphics cards.

[0114] In this embodiment, since the graphics card is located in the form of "IP address + port", when the fault count of the graphics card in the shared memory exceeds the preset graphics card fault threshold, the system will no longer allocate the address of the graphics card to the client, and send the address of other graphics cards whose fault counts do not exceed the preset graphics card fault threshold to the client. In other embodiments, load balancing is also performed on other graphics cards, and the number of currently processed requests of other graphics cards whose fault counts do not exceed the preset graphics card fault threshold is determined, and service requests are preferentially sent to the graphics card with the least number of processed requests.

[0115] On the basis of the above technical solution, a shielding time of the graphics card is further set. If the shielding operation is performed on the graphics card within the shielding time of the graphics card, when the time length exceeds the shielding time of the graphics card, the service request is allowed to be sent to the graphics card. In this embodiment, the shielding time of the graphics card is set. When the graphics card fault count of the graphics card reaches the set graphics card fault threshold, the graphics card is shielded and the timing starts at the same time. When the shielding time reaches the set graphics card shielding time, an attempt is made to resend the processing request to the graphics card.

[0116] At this time, the graphics card fault count is retained. If the business request is successfully processed, the graphics card fault count is reduced by 1 in the shared memory. If it fails, the graphics card is shielded again.

[0117] Because some faults are recoverable minor faults, for example, the graphics card may overheat, resulting in reduced computing power, causing the business request processing to fail. The graphics card will return to normal after cooling down. Therefore, the shielding time should not be set too long. The shielding time ranges from 30 seconds to 5 minutes, usually 1 minute.

[0118] In other implementations, when the graphics card is still in the shielding period, a small proportion of trial can be used to try to send a service request to the graphics card to determine whether the graphics card has recovered. For example, the graphics card can usually process 10 service requests at the same time, and only one service request is sent to the graphics card at the same time for trial. If the graphics card returns a successful service processing message, the shielding of the graphics card is canceled, and the graphics card fault count in the shared memory is reduced by 1.

[0119] Figure 6 It is a structural schematic diagram of an electronic device of an embodiment of the present invention, the electronic device includes a processor and a memory, the memory is used to store a computer executable program, when the computer program is executed by the processor, the processor executes a vehicle intelligent power-assisted propulsion method based on rotation angle monitoring.

[0120] like Figure 6 As shown, the electronic device is presented in the form of a general computing device. The processor may be one or more and work in coordination. The present invention does not exclude distributed processing, that is, the processor may be dispersed in different physical devices. The electronic device of the present invention is not limited to a single entity, but may also be the sum of multiple physical devices.

[0121] The memory stores a computer executable program, which is usually a machine-readable code. The computer-readable program can be executed by the processor to enable the electronic device to perform the method of the present invention, or at least some steps in the method.

[0122] The memory includes a volatile memory, such as a random access memory unit (RAM) and / or a cache memory unit, and may also be a non-volatile memory, such as a read-only memory unit (ROM).

[0123] Optionally, in this embodiment, the electronic device further includes an I / O interface, which is used for the electronic device to exchange data with an external device. The I / O interface can represent one or more of several types of bus structures, including a storage unit bus or a storage unit controller, a peripheral bus, a graphics acceleration port, a processing unit, or a local bus using any of a variety of bus structures.

[0124] It should be understood that Figure 7 The electronic device shown is only an example of the present invention, and the electronic device of the present invention may also include elements or components not shown in the above examples. For example, some electronic devices also include display units such as display screens, and some electronic devices also include human-computer interaction elements such as buttons, keyboards, etc. As long as the electronic device can execute the computer-readable program in the memory to implement the method of the present invention or at least part of the steps of the method, it can be considered as an electronic device covered by the present invention.

[0125] Figure 6 Schematic diagram of a computer readable recording medium according to an embodiment of the present invention. Figure 5As shown, a computer executable program is stored in a computer-readable recording medium, and when the computer executable program is executed, the vehicle intelligent power-assisted pushing method based on rotation angle monitoring of the present invention is implemented. The computer-readable storage medium may include a data signal propagated in a baseband or as part of a carrier wave, which carries a readable program code. This propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. The readable storage medium may also be any readable medium other than a readable storage medium, which may send, propagate or transmit a program for use by or in combination with an instruction execution system, device or device. The program code contained on the readable storage medium may be transmitted using any suitable medium, including but not limited to wireless, wired, optical cable, RF, etc., or any suitable combination of the above.

[0126] Program code for performing the operations of the present invention may be written in any combination of one or more programming languages, including object-oriented programming languages ​​such as Java, C++, etc., and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user computing device, partially on the user device, as a separate software package, partially on the user computing device and partially on a remote computing device, or entirely on a remote computing device or server. In cases involving a remote computing device, the remote computing device may be connected to the user computing device through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computing device (e.g., through the Internet using an Internet service provider).

[0127] Through the above description of the implementation mode, it is easy for those skilled in the art to understand that the present invention can be implemented by hardware capable of executing a specific computer program, such as the system of the present invention, and the electronic processing unit, server, client, mobile phone, control unit, processor, etc. contained in the system. The present invention can also be implemented by a vehicle comprising at least a part of the above system or components. The present invention can also be implemented by computer software that executes the method of the present invention, such as control software executed by a microprocessor, an electronic control unit, a client, a server, etc. on the locomotive side. However, it should be noted that the computer software that executes the method of the present invention is not limited to being executed by one or a specific hardware entity, and it can also be implemented in a distributed manner by unspecified specific hardware, such as some method steps executed by the computer program can be executed on the locomotive side, and another part can be executed in a mobile terminal or a smart helmet, etc. For computer software, the software product can be stored in a computer-readable storage medium (which can be a CD-ROM, a U disk, a mobile hard disk, etc.), or it can be distributed and stored on the network, as long as it enables the electronic device to execute the method according to the present invention.

[0128] The specific embodiments described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the present invention is not inherently related to any specific computer, virtual device or electronic device, and various general devices can also implement the present invention. The above description is only a specific embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. A method for quickly shielding a faulty graphics card in a cluster, characterized by: Preset the fault threshold of the graphics card, use inter-process communication to count the faults of each graphics card in the multiple graphics cards set in each server in the cluster with multiple servers, and use the inter-process communication statistics to enable different processes in the same server to communicate with each other and count the status of different processes returned by the same graphics card to quickly determine the status of the graphics card; The client sends a service request to each graphics card through an address assigned to the graphics card, and each graphics card processes the service request sent from the client and each service request corresponds to a process; The fault threshold of the graphics card is adjusted according to the business traffic. If the business traffic is large, the fault threshold is increased, otherwise, the fault threshold is reduced; And, the graphics card is a physical graphics card; Before sending a request to the graphics card, the client process needs to read the fault count of the graphics card to determine whether it exceeds a preset fault threshold. When the fault count of the graphics card exceeds the preset fault threshold, the graphics card is shielded and no request is sent to the graphics card.

2. The method for quickly shielding a faulty graphics card in a cluster as claimed in claim 1, characterized in that: Also includes: If the graphics card operates normally, a correct processing result is returned; if the graphics card is abnormal, an error number is returned.

3. The method for quickly shielding a faulty graphics card in a cluster as claimed in claim 1, characterized in that: The inter-process communication is a shared memory communication method.

4. The method for quickly shielding a faulty graphics card in a cluster as claimed in claim 3, characterized in that: Counting failures of each of multiple graphics cards configured in each server in a cluster having multiple servers using inter-process communication includes: If a request processing failure message is received from the graphics card, the fault count of the graphics card is increased by 1; If the message returned is that the request is processed successfully, the fault count of the graphics card will be reduced by 1.

5. The method for quickly shielding a faulty graphics card in a cluster as claimed in claim 4, characterized in that: The shielding operation on the graphics card includes: shielding the address of the graphics card, and forwarding the service request sent to the graphics card to other graphics cards.

6. The method for quickly shielding a faulty graphics card in a cluster as claimed in claim 5, characterized in that: Also includes: Set the shielding time of the graphics card. Shielding operation is performed on the graphics card within the shielding time. If the time length exceeds the shielding time of the graphics card, service requests are allowed to be sent to the graphics card.

7. A device for quickly shielding a faulty graphics card in a cluster, characterized by: A fault counting module pre-sets a fault threshold of a graphics card, uses inter-process communication to count the faults of each graphics card in a plurality of graphics cards set in each server in a cluster having a plurality of servers, enables different processes in the same server to communicate with each other through inter-process communication statistics, and counts the states of different processes returned by the same graphics card to quickly determine the state of the graphics card; wherein, a client sends a service request to the graphics card through an address assigned to each graphics card, and each graphics card processes the service request sent from the client and each service request corresponds to a process; wherein, the fault threshold of the graphics card is adjusted according to the service flow, if the service flow is large, the fault threshold is increased, otherwise the fault threshold is reduced; and, the graphics card is a physical graphics card; Graphics card shielding module, before the client process needs to send a request to the graphics card, it needs to read the fault count of the graphics card to determine whether it exceeds a preset fault threshold. When the fault count of the graphics card exceeds the preset fault threshold, the graphics card is shielded and no request is sent to the graphics card.

8. The device for quickly shielding a faulty graphics card in a cluster as claimed in claim 7, characterized in that: Also includes: The inter-process communication is a shared memory communication mode; Using inter-process communication to count the faults of each graphics card in a cluster having multiple servers, including: if a message indicating that a request processing failed is received from the graphics card, the fault count of the graphics card is increased by 1, and if a message indicating that the request processing succeeded is returned, the fault count of the graphics card is decreased by 1; The shielding operation on the graphics card includes: shielding the address of the graphics card and forwarding the service request sent to the graphics card to other graphics cards; setting the graphics card shielding time, if the shielding operation is performed on the graphics card within the graphics card shielding time, when the time length exceeds the graphics card shielding time, the service request is allowed to be sent to the graphics card.

9. An electronic device, comprising a processor and a memory, wherein the memory is used to store a computer executable program, wherein: When the computer executable program is executed by the processor, the processor performs the method according to any one of claims 1 to 6.

10. A computer-readable medium storing a computer-executable program, characterized in that: When the computer executable program is executed, the method according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • GPU fault prediction method and device

    CN105988918A

  • Fault processing method and device

    CN106027290A

  • Automatic prediction method, device, apparatus and medium for application service failure

    CN109144835A

  • Cluster resource scheduling method, device and equipment and computer readable storage medium

    CN109634748A

  • A disaster recovery method and device applied to a server

    CN109936613A