Data Processing Method, Apparatus, Device, and Computer Storage Medium

Abstract

The present application provides a data processing method, apparatus, device and computer storage medium, relating to the field of computer technology. The method is applied to a file server and includes: when it is monitored that a first message queue includes a first message sent by a unitized service component, obtaining the first message from the first message queue, where the first message includes first metadata information of a target file, and the first metadata information includes first address information, and the first address information is the compressed package address information of the target file; based on the first address information, downloading the compressed package of the target file from a storage server; decompressing the compressed package to obtain the decompressed target file; uploading the decompressed target file to the storage server; wherein, the physical distance between the file server and the storage server is less than the physical distance between the storage server and the unitized service component. According to the embodiments of the present application, the processing efficiency of file data can be effectively improved.

Description

Technical Field

[0001] This application belongs to the field of computer technology, and particularly relates to a data processing method, device, equipment and computer storage medium. Background Art

[0002] At present, in urban vehicle management, after the in-vehicle computer takes a parking picture, due to the large size of the parking picture, multiple parking pictures are often packed, the generated compressed package is uploaded to the storage server, and the address of the compressed package is sent to the database of the corresponding unit. When the unitized service component in the unit detects that the database contains a file to be decompressed, it will download the compressed package of the file to be decompressed from the storage server and decompress it. After decompression, multiple parking pictures are uploaded to the storage server, and the storage addresses of multiple parking pictures are saved in the database. In this way, when the user terminal requests a parking picture later, it can directly view the picture content according to the storage address of the parking picture saved in the database.

[0003] However, in the actual scenario, the data between the above storage server and the unitized service component is transmitted through the public network, and the data transmission efficiency between the two depends on the speed of the fiber optic network. Thus, when the deployment regions of the above storage server and the above unitized service component are inconsistent or the distance between the two is too large, it will affect the speed of the fiber optic network, increase the service communication delay between the two places, and thus increase the time consumption of file downloading and uploading processes. Summary of the Invention

[0004] A data processing method, device, equipment and computer storage medium provided by this application can effectively improve the processing efficiency of file data.

[0005] In the first aspect, an embodiment of this application provides a data processing method, which is applied to a file server. The data processing method includes:

[0006] When it is monitored that a first message sent by a unitized service component is included in a first message queue, obtain the first message from the first message queue. The first message includes first metadata information of a target file, and the first metadata information includes first address information, and the first address information is the compressed package address information of the target file;

[0007] Based on the first address information, download the compressed package of the target file from the storage server;

[0008] Decompress the compressed package to obtain the decompressed target file;

[0009] Upload the decompressed target file to the storage server;

[0010] Among them, the compressed package of the target file is obtained by the in-vehicle device after compressing the obtained parking data, and the in-vehicle device uploads the compressed package of the target file to the storage server in advance; the physical distance between the file server and the storage server is less than the physical distance between the storage server and the unitization service component.

[0011] In some possible implementation manners, the first metadata information of the target file further includes the index identification information of the target file in the database. After uploading the decompressed target file to the storage server, the data processing method further includes:

[0012] Receiving second address information sent by the storage server, where the second address information is the address information of the decompressed target file;

[0013] Generating a second message, where the second message includes the second address information and the index identification information of the target file in the database;

[0014] Sending the second message to the first message queue, so that when the unitization service component monitors that the first message queue includes the second message, the first target operation is executed;

[0015] Among them, the first target operation includes:

[0016] Based on the index identification information of the target file in the database, storing the second address information in the database.

[0017] In some possible implementation manners, the first message further includes the identification information of the first message queue. Sending the second message to the first message queue includes:

[0018] Based on the identification information of the first message queue, sending the second message to the first message queue.

[0019] In some possible implementation manners, the physical distance between the file server and the storage server is within a preset range.

[0020] In some possible implementation manners, the first metadata information of the target file further includes the first file attribute information of the target file; uploading the decompressed target file to the storage server includes:

[0021] Based on the first file attribute information of the target file, detecting whether the second file attribute information of the decompressed target file is consistent with the first file attribute information;

[0022] In the case where the second file attribute information is consistent with the first file attribute information, uploading the decompressed target file to the storage server.

[0023] In a second aspect, an embodiment of the present application provides a data processing method, which is applied to a unitization service component. The data processing method includes:

[0024] When it is detected that the database contains the target information, determine the file corresponding to the target information as the target file;

[0025] Generate a first message according to the first metadata information of the target file. The first message includes the first metadata information of the target file, and the first metadata information includes a first address information, where the first address information is the address information of the compressed package of the target file;

[0026] Send the first message to the first message queue, so that when the file server monitors that the first message queue includes the first message, perform a second target operation;

[0027] Among them, the second target operation includes:

[0028] Based on the first address information, download the compressed package of the target file from the storage server;

[0029] Decompress the compressed package to obtain the decompressed target file;

[0030] Upload the decompressed target file to the storage server.

[0031] In some possible implementation manners, the first metadata information of the target file further includes the index identification information of the target file in the database. After sending the first message to the first message queue, the data processing method further includes:

[0032] When it is monitored that the first message queue includes a second message sent by the file server, obtain the second message from the first message queue. The second message includes a second address information and the index identification information of the target file in the database, and the second address information is the address information of the decompressed target file;

[0033] Based on the index identification information of the target file in the database, store the second address information in the database.

[0034] In a third aspect, an embodiment of the present application provides a data processing device, which is applied to a file server. The data processing device includes:

[0035] A first acquisition module, configured to obtain the first message from the first message queue when it is monitored that the first message queue includes the first message sent by the unitized service component. The first message includes the first metadata information of the target file, and the first metadata information includes a first address information, where the first address information is the address information of the compressed package of the target file;

[0036] A download module, configured to download the compressed package of the target file from the storage server based on the first address information;

[0037] A decompression module, configured to decompress a compressed package to obtain a decompressed target file;

[0038] An upload module, configured to upload the decompressed target file to a storage server;

[0039] Wherein, the compressed package of the target file is obtained by the vehicle-mounted device after compressing the acquired parking data; the vehicle-mounted device uploads the compressed package of the target file to the storage server in advance; the physical distance between the file server and the storage server is less than the physical distance between the storage server and the unitized service component.

[0040] Fourthly, an embodiment of the present application provides a data processing device, which is applied to a unitized service component. The data processing device includes:

[0041] A determination module, configured to determine the file corresponding to the target information as the target file when it is detected that the database contains the target information;

[0042] A first generation module, configured to generate a first message according to the first metadata information of the target file. The first message includes the first metadata information of the target file, and the first metadata information includes a first address information, where the first address information is the compressed package address information of the target file;

[0043] A first sending module, configured to send the first message to a first message queue, so that when the file server monitors that the first message queue includes the first message, the file server performs a second target operation;

[0044] Wherein, the compressed package of the target file is obtained by the vehicle-mounted device after compressing the acquired parking data; the vehicle-mounted device uploads the compressed package of the target file to the storage server in advance; the physical distance between the file server and the storage server is less than the physical distance between the storage server and the unitized service component;

[0045] The second target operation includes:

[0046] Downloading the compressed package of the target file from the storage server based on the first address information;

[0047] Decompressing the compressed package to obtain a decompressed target file;

[0048] Uploading the decompressed target file to the storage server.

[0049] Fifthly, an embodiment of the present application provides a data processing device, which includes:

[0050] A processor and a memory storing computer program instructions;

[0051] When the processor executes the computer program instructions, the data processing method provided in any one of the above embodiments of the present application is implemented.

[0052] In a sixth aspect, an embodiment of the present application provides a computer storage medium. Computer program instructions are stored on the computer-readable storage medium. When the computer program instructions are executed by a processor, the data processing method provided in any one of the above embodiments of the present application is implemented.

[0053] In a seventh aspect, an embodiment of the present application provides a computer program product. When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device is caused to execute the data processing method provided in any one of the above embodiments of the present application.

[0054] The data processing method, device, equipment, and computer storage medium in the embodiments of the present application are applied to a file server. By obtaining the first message sent by the unitized service component in the first message queue and based on the first metadata information of the target file in the first message, the compressed package of the target file is downloaded in the storage server. After decompressing the downloaded compressed package, the decompressed target file is uploaded to the storage server, where the physical distance between the file server and the storage server is less than the physical distance between the storage server and the unitized service component. Compared with the prior art, a data processing method, device, equipment, and computer storage medium provided in the embodiments of the present application are applied to a file server. By configuring a file server with a physical distance between the storage server and the unitized service component that meets the preset conditions and combining the message queue, the transmission of long-distance large files can be transformed into long-distance micro-data transmission based on the message queue, and the long-distance file transmission between the storage server and the unitized service component can be transformed into short-distance file transmission between the storage server and the file server. In this way, the processing efficiency of file data is effectively improved, and the time-consuming in the processes of file downloading and uploading is reduced. BRIEF DESCRIPTION OF THE DRAWINGS

[0055] Figure 1 is a schematic diagram of communication between a storage server and a unitized service component at the current stage provided by an embodiment of the present application;

[0056] Figure 2 is a schematic flowchart of a data processing method applied to a file server provided by an embodiment of the present application;

[0057] Figure 3 is a schematic flowchart of a data processing method applied to a unitized service component provided by an embodiment of the present application;

[0058] Figure 4 is a schematic diagram of the system architecture of a data processing method provided by an embodiment of the present application;

[0059] Figure 5 It is a schematic structural diagram of a data processing device applied to a file server provided by an embodiment of the present application;

[0060] Figure 6 It is a schematic structural diagram of a data processing device applied to a unitized service component provided by an embodiment of the present application;

[0061] Figure 7 It is a schematic structural diagram of a data processing device provided by an embodiment of the present application. Detailed implementation manners

[0062] In order to more clearly understand the above objects, features and advantages of the present disclosure, the solutions of the present disclosure will be further described below. It should be noted that, without conflict, the embodiments of the present disclosure and the features in the embodiments may be combined with each other.

[0063] Many specific details are set forth in the following description in order to provide a thorough understanding of the present disclosure, but the present disclosure may be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some embodiments of the present disclosure, rather than all embodiments.

[0064] It should be noted that, in this article, relational terms such as "first" and "second" are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any actual relationship or order between these entities or operations. Moreover, the term "comprising" or any other variant thereof is intended to cover a non-exclusive inclusion, so that a process, method, article or device comprising a series of elements not only includes those elements, but also includes other elements not expressly listed, or also includes elements inherent to such process, method, article or device. Without further limitation, an element defined by the statement "comprising a..." does not exclude the existence of additional identical elements in the process, method, article or device comprising the element.

[0065] A message queue (Message Queue, MQ) is an important component in a distributed system and can be regarded as a container for storing messages during the transmission of messages. At the present stage, message queues are mainly used to solve problems such as application decoupling, asynchronous messages, and traffic peak shaving.

[0066] A file server (file server) is used to provide file services. In the present application, the file server can process related requirements of files and implement file transmission in a network.

[0067] The unitized service component can be set separately in each service unit according to the deployment of different service units. The unitized service component involved in the embodiment of the present application is a service component set in a certain non-specific service unit, which can provide corresponding service functions, such as sending and receiving messages, etc., and can specifically be used to provide corresponding data transmission services for this unit.

[0068] The storage server can be used to store various file data uploaded by other terminals or servers.

[0069] It should be noted that, in the embodiment of the present application, the above file server can communicate with the storage server, and the above file server and the above unitized service component can be pre-communicated with the above message queue.

[0070] Moreover, it should be noted that in the actual application scenario, the number of the above storage servers is often single, and the above service units are often independently deployed in different regions. Therefore, the number of service units is usually multiple.

[0071] Figure 1 It is a schematic diagram of the communication between the storage server and the database server at the present stage provided by an embodiment of the present application.

[0072] As described in the background art part, in Figure 1 when the unitized service component in the unit detects that the database (DB) contains a file to be decompressed, it will download the compressed package of the file to be decompressed from the storage server and decompress it, and upload multiple parking pictures to the storage server after decompression is completed.

[0073] In this way, after the unitized service component receives the storage address of the decompressed file returned by the storage server, it can save the storage address of the file to the database. Based on this, the user side can directly view the parking pictures by calling the file storage address saved in the database subsequently.

[0074] However, in the actual application scenario, since the above storage server and the unitized service component communicate through the public network, the actual physical distance between the above storage server and the unitized service component will increase the service communication delay between the two places, resulting in an increase in the time consumption of file downloading and uploading processes.

[0075] To solve the problems of the prior art, the embodiment of the present application provides a data processing method, device, equipment and computer storage medium. First, the data processing method provided by the embodiment of the present application will be introduced below.

[0076] Figure 2 It shows a schematic flowchart of the data processing method applied to the file server provided by an embodiment of the present application. AsFigure 2 As shown in the figure, the data processing method applied to a file server provided by an embodiment of the present application includes the following steps:

[0077] S210, when it is monitored that the first message queue includes a first message sent by a unitized service component, obtain the first message from the first message queue. The first message includes the first metadata information of the target file, and the first metadata information includes the first address information, where the first address information is the compressed package address information of the target file;

[0078] S220, based on the first address information, download the compressed package of the target file from the storage server;

[0079] S230, decompress the compressed package to obtain the decompressed target file;

[0080] S240, upload the decompressed target file to the storage server.

[0081] Among them, the compressed package of the above target file can be obtained by the vehicle-mounted device after compressing the obtained parking data, and the vehicle-mounted device can upload the compressed package of the target file to the storage server in advance.

[0082] The physical distance between the above file server and the storage server can be less than the physical distance between the storage server and the unitized service component.

[0083] The data processing method of the embodiment of the present application, applied to a file server, can obtain the first message sent by the unitized service component in the first message queue, and based on the first metadata information of the target file in the first message, download the compressed package of the target file in the storage server. After decompressing the downloaded compressed package, upload the decompressed target file to the storage server, where the physical distance between the file server and the storage server is less than the physical distance between the storage server and the unitized service component.

[0084] Compared with the prior art, a data processing method provided by an embodiment of the present application, applied to a file server, can configure a file server with a physical distance meeting a preset condition between the storage server and the unitized service component and combine it with a message queue, so as to transform the transmission of a long-distance large file into long-distance micro-data transmission based on the message queue, and transform the long-distance file transmission between the storage server and the unitized service component into short-distance file transmission between the storage server and the file server. Thus, the processing efficiency of file data is effectively improved, and the time consumption in the processes of file downloading and uploading is reduced.

[0085] The following introduces the specific implementation manners of the above steps 210 to 240.

[0086] In S210, the file server may have established communication with the first message queue in advance, so that it is possible to monitor the messages in the first message queue.

[0087] In specific implementation, when the above file server monitors that the first message queue includes the first message sent by the unitized service component, it can obtain the first message from the first message queue. It should be noted that in the technical field, obtaining the first message from the first message queue can also be referred to as consuming the first message from the first message queue.

[0088] Among them, the first message may include the first metadata information of the target file, and the first metadata information may include the first address information, which may be the compressed package address information of the target file.

[0089] In this embodiment, since the compressed package of the target file is pre-uploaded to the storage server by the vehicle-mounted device, correspondingly, the above first address information, that is, the compressed package address information of the target file, may correspond to the specific storage location information of the target file compressed package in the storage server.

[0090] And, the above target file may specifically include but is not limited to at least one of data such as parking pictures, videos, gifs, and parking environment monitoring reports, and the compressed package of the target file may be a compressed format target file package.

[0091] It should be noted that in practical applications, the above first message queue may specifically be but is not limited to ActiveMQ, RabbitMQ, ZeroMQ, Kafka, MetaMQ, or RocketMQ, etc., and this application does not make specific restrictions on this.

[0092] It can be understood that when different message queue middleware is selected, the data format of the above messages will also change accordingly. For example, when using rabbitmq, correspondingly, the messages in the message queue will be saved in the form of a byte array. And, the above database server and file server need to comply with the message transmission protocol corresponding to the specific message queue, so as to enable communication between the server and the message queue. Specifically, when the above file server monitors that the first message queue includes the first message sent by the unitized service component, it can obtain the first message from the first message queue.

[0093] It should be noted that in this embodiment, the above unitized service component and the above first message queue may be deployed in the same unit to provide corresponding data processing services for the same unit, so that it is possible to ensure the pertinence of the unit service when executing the above data processing method, and avoid processing data across units.

[0094] In S220, specifically, as an example, the above file server may generate a file request corresponding to the compressed package of the target file according to the first address information and send the file request to the storage server.

[0095] Correspondingly, after receiving the file request, the above storage server will return the compressed package of the requested target file to the file server. In this way, the file server can download the compressed package of the target file.

[0096] It should be noted that the above is only an example and does not play a substantial limiting role in the protection scope of this application. In some other embodiments, the file server may also use other methods to implement the download of the compressed package of the target file. For example, the file server may also download the compressed package of the target file from the storage server through the corresponding device interface.

[0097] In S230, specifically, after completing the download of the compressed package of the target file, the file server can decompress the compressed package locally to obtain the decompressed target file.

[0098] As an example, the compressed package of the above target file may be the compressed data of multiple parking pictures pre-uploaded by the vehicle head unit to the storage server. Based on this, after the file server decompresses the compressed package of the target file, multiple parking pictures will actually be obtained.

[0099] It should be noted that in this field, considering the diversity of current file decompression methods, therefore, in the embodiments of this application, the specific decompression means adopted by the above file server when decompressing the compressed package of the target file are not limited.

[0100] In S240, in specific implementation, after completing the decompression of the compressed package of the target file, the file server will upload the obtained decompressed target file to the storage server.

[0101] As an example, multiple parking pictures can be uploaded to the storage server so that the storage server stores the multiple parking pictures.

[0102] Exemplarily, when the decompressed target file is multiple parking pictures, the file server may first send a file storage request corresponding to the multiple parking pictures to the storage server. After the storage server receives the file storage request and confirms it, the file server sends the multiple parking pictures to the storage server so that the storage server stores them after receiving the multiple parking pictures.

[0103] It should be noted that the above is only an example and does not substantially limit the protection scope of this application. In some other embodiments, the file server may also adopt other methods to implement the upload of the decompressed target file. For example, the file server may also upload the above decompressed target file to the storage server through a corresponding device interface.

[0104] In some embodiments, considering the actual needs of subsequent users, it is necessary to save the storage address of the decompressed target file to the database.

[0105] Based on this, the first metadata information of the target file may further include the index identification information of the target file in the database. After uploading the decompressed target file to the storage server, the data processing method may further include:

[0106] Receiving the second address information sent by the storage server, and the second address information may be the address information of the decompressed target file;

[0107] Generating a second message, and the second message may include the second address information and the index identification information of the target file in the database;

[0108] Sending the second message to the first message queue, so that when the unitization service component monitors that the first message queue includes the second message, it performs the first target operation;

[0109] Among them, the first target operation may include:

[0110] Based on the index identification information of the target file in the database, storing the second address information to the database.

[0111] It should be noted that after receiving the decompressed target file uploaded by the file server, the storage server will store the decompressed target file and return the storage address of the above decompressed target file in the storage server, that is, the second address information, to the file server.

[0112] Specifically, the file server may save the index identification information of the target file included in the first metadata information of the first message after obtaining the first message. Thus, after receiving the second address information sent by the storage server, the file server can generate a second message according to the above index identification information and the above second address information.

[0113] Based on this, the generated second message is sent to the first message queue, so that when the unitized service component monitors that the first message queue includes the second message, it performs corresponding operations to save the second address information to the database. In this way, it effectively meets the subsequent user's call requirement for the second address information in the database when requesting to view the target file.

[0114] It should be noted that in some other embodiments, the first metadata information of the above first message may further include other information related to the target file, such as the resource size of the target file, the file format, etc.

[0115] Then, correspondingly, the file server may also generate a second message based on the other information related to the target file, the above index identification information, and the above second address information included in the first metadata information. The generated second message includes the above other information related to the target file, the above index identification information, and the above second address information. Therefore, this application does not make specific limitations here.

[0116] In some embodiments, the first message may further include the identification information of the first message queue. Sending the second message to the first message queue may include:

[0117] Based on the identification information of the first message queue, the second message is sent to the first message queue.

[0118] It should be noted that the above identification information can be used to indicate a unique message queue.

[0119] Exemplarily, the identification information of the above first message queue may be the address information of the first message queue.

[0120] In some other embodiments, considering that the above unitized service component and the above first message queue are usually deployed in the same service unit, therefore, the identification information of the above first message queue may also be information associated with the service unit to which the first message queue belongs, such as the service unit name or service unit number of the service unit to which the first message queue belongs, etc.

[0121] Specifically, when the file server obtains the first message, it will save the identification information of the first message queue included in the first message.

[0122] In this way, after generating the second message, the file server can correspondingly send the second message to the first message queue based on the identification information of the first message queue in the first message.

[0123] For ease of understanding, the process of sending the second message to the first message queue described above can also be regarded as: after the file server obtains the first message in the first message queue and performs corresponding processing operations based on the first message, the generated second message is returned to the first message queue along the original path.

[0124] In some embodiments, in order to further improve data processing efficiency, the physical distance between the file server and the storage server can be within a preset range.

[0125] The above preset range can be set according to actual communication experience. Exemplarily, if historical communication experience indicates that when the physical distance between two servers is within the range of (X Km, Y Km), the communication efficiency such as file transfer between them is relatively high and the time consumption is relatively less, then the above file server can be set at a certain position X Km to Y Km away from the storage server, thereby further improving the communication efficiency.

[0126] It should be noted that the upper limit value of the above preset range is less than the physical distance between the unitized service component and the storage server.

[0127] In some embodiments, since the decompressed target file obtained after the file server decompresses the compressed package of the target file generally remains locally on the file server by default,

[0128] Then, based on this, in order to reduce the data load of the above file server and reduce redundant resources, after uploading the decompressed target file to the storage server, the data processing method may further include:

[0129] Delete the decompressed target file.

[0130] Specifically, the file server can delete the decompressed target file locally at any time after uploading the decompressed target file to the storage server, and this application does not make specific limitations here.

[0131] In some embodiments, considering that the phenomenon of file damage during decompression occurs from time to time, therefore, in order to improve the accuracy of the transmitted file data, the first metadata information of the target file may further include the first file attribute information of the target file; uploading the decompressed target file to the storage server may include:

[0132] Based on the first file attribute information of the target file, detect whether the second file attribute information of the decompressed target file is consistent with the first file attribute information;

[0133] In the case where the second file attribute information is consistent with the first file attribute information, upload the decompressed target file to the storage server.

[0134] It should be noted that the above file attribute information may specifically include, but is not limited to, at least one of file name, file size, file quantity, file format, etc., and this application does not make specific limitations thereto.

[0135] Next, please refer to Figure 3 , Figure 3 which shows a schematic flowchart of a data processing method applied to a file server provided by an embodiment of this application.

[0136] As Figure 3 shown, the data processing method applied to a unitized service component provided by an embodiment of this application includes the following steps:

[0137] S310, when it is detected that the database contains target information, determine the file corresponding to the target information as the target file;

[0138] S320, generate a first message according to the first metadata information of the target file, the first message includes the first metadata information of the target file, and the first metadata information includes first address information, and the first address information is the compressed package address information of the target file;

[0139] S330, send the first message to the first message queue, so that when the file server monitors that the first message queue includes the first message, perform a second target operation;

[0140] Among them, the second target operation includes:

[0141] Based on the first address information, download the compressed package of the target file from the storage server;

[0142] Decompress the compressed package to obtain the decompressed target file;

[0143] Upload the decompressed target file to the storage server.

[0144] In the data processing method of an embodiment of this application, which is applied to a unitized service component, it can determine the target file to be decompressed when it is detected that the database contains target information, then generate a first message based on the first metadata information of the target file, and send the first message to the first message queue. In this way, when the file server monitors that the first message queue includes the first message, it can perform corresponding file download, decompression, and upload operations. A data processing method provided by an embodiment of this application, which is applied to a unitized service component, can effectively improve the processing efficiency of file data and reduce the time consumption in the processes of file download and upload.

[0145] Next, a detailed introduction to the specific implementation manners of the above steps 310 to 330 will be given.

[0146] In S310, in specific implementation, when it is detected that the target information is included in the database, the file corresponding to the target information can be determined as the target file.

[0147] In this embodiment, the above target information can specifically be the decompression flag carried in the first metadata information of the target file.

[0148] Exemplarily, after the in-vehicle device stores the compressed package of a certain file to the storage server, the storage server will return the storage address information of the compressed package to the in-vehicle device. Thus, after receiving the storage address information, the in-vehicle device will upload the relevant file information of the file and the storage address information of the compressed package to the database.

[0149] Correspondingly, after the database receives the relevant file information of the file newly stored in the database and the storage address information of the compressed package, it will generate the first metadata information of the file based on the received data information, and mark the first metadata information of the target file with a decompression flag.

[0150] Thus, when the unitized service component detects that the decompression flag is carried in the first metadata information of a certain file, the file is determined as the target file.

[0151] In some other embodiments, the above target information can also specifically be the metadata information corresponding to the file. Specifically, the above unitized service component can detect whether the metadata information of each file in the above database includes the address information of the decompressed file. Thus, the metadata information that does not include the address information of the decompressed file can be used as the target information, and the file corresponding to the target information is determined as the target file.

[0152] It should be noted that in this embodiment, the above detection operation of the database by the unitized service component can be performed only when the preset conditions are met. For example, the above unitized service component can set a timing task to perform a data scan of the database every preset duration. In some other embodiments, it can also perform the detection operation of the target information in the database when other trigger conditions are met.

[0153] As an example, the unitized service component can also establish a remote communication with other devices in advance. Thus, when receiving a database detection request sent by other devices, it performs the detection operation of the target information in the database, and determines the corresponding target file when it is detected that the target information is included in the database.

[0154] In practical applications, the above database can be at least one of databases such as mysql, oracle, sqlserver, and sqlite.

[0155] In addition, it should be noted that in the embodiments of the present application, the above-mentioned database and the above-mentioned unitized service components can be deployed on the same device. For example, both of them can be deployed on the same server. In some other embodiments, the above-mentioned database and the unitized service components can also be separately deployed on different servers for communication, and the present application does not make specific limitations on this.

[0156] In S320, specifically in implementation, unitization can be to first obtain the first metadata information of the target file through the database, and thus, based on the obtained first metadata information of the target file, generate a first message.

[0157] Among them, the above-mentioned first message can include the first metadata information of the target file, and the first metadata information can include first address information, and the first address information can specifically be the compressed package address information of the target file.

[0158] In S330, specifically in implementation, send the first message to the first message queue, so that when the file server monitors that the first message queue includes the first message, perform a second target operation;

[0159] Among them, the second target operation includes:

[0160] Based on the first address information, download the compressed package of the target file from the storage server;

[0161] Unzip the compressed package to obtain the decompressed target file;

[0162] Upload the decompressed target file to the storage server.

[0163] Specifically, the above-mentioned unitized service component can send the first message to the first message queue after generating the first message.

[0164] In this embodiment, the above-mentioned first message queue and the above-mentioned unitized service component can be deployed in the same service unit.

[0165] In some implementation manners, considering the actual needs of subsequent users, it is necessary to save the storage address of the decompressed target file to the database. Based on this, the first metadata information of the target file can also include the index identification information of the target file in the database. After sending the first message to the first message queue, the data processing method can further include:

[0166] When monitoring that the first message queue includes a second message sent by the file server, obtain the second message from the first message queue. The second message can include second address information and the index identification information of the target file in the database, and the second address information can be the address information of the decompressed target file;

[0167] Based on the index identification information of the target file in the database, store the second address information into the database.

[0168] It should be noted that the index identification information of the above-mentioned target file in the database can be used to indicate the storage location of the above-mentioned second address information in the database.

[0169] Specifically, when the unitized service component monitors that the first message queue includes a second message sent by the file server, it obtains the second message from the first message queue.

[0170] In this way, based on the index identification information in the second message, the second address information can be stored into the database.

[0171] In some embodiments, to ensure the rationality of data processing, the above-mentioned storing the second address information into the database based on the index identification information of the target file in the database may specifically include:

[0172] According to the index identification information of the target file in the database, add the second address information to the first metadata information of the target file to obtain the second metadata information of the target file.

[0173] That is to say, the above-mentioned unitized service component updates the metadata information of the target file in the database.

[0174] Please refer to the following Figure 4 , Figure 4 which is a schematic diagram of the system architecture of the data processing method provided in an embodiment of the present application.

[0175] In Figure 4 , taking the first unit as an example, the first unit includes the unitized service component corresponding to the unit and the first message queue. The unitized service component in the unit can interact and communicate with the first message queue, and the first message queue can interact and communicate with the file server, and the file server interacts and communicates with the storage server.

[0176] To facilitate a better understanding of the above-mentioned data processing method, the present application also provides a specific embodiment of the above-mentioned data processing method. The following specifically introduces the data processing method of the present application in combination with Figure 4 the four parties of the unitized service component, the first message queue, the file server, and the storage server in the first unit in

[0177] Step 1: When the unitized server component in the first unit detects that the database contains target information, determine that the file corresponding to the target information is the target file, and then generate a first message according to the first metadata information of the target file.

[0178] It should be noted that the database can be deployed in the first unit. For example, it can be deployed on the same server as the unitized service component in the above-mentioned first unit, and communication can be established in advance between the database and the unitized service component in the first unit.

[0179] The first message can include the first metadata information of the target file. The first metadata information can include the first address information, and the first address information can be the compressed package address information of the target file.

[0180] Step 2: The unitized service in the first unit sends the first message to the first message queue. When the file server monitors the first message sent by the unitized service component in the first message queue, it obtains the first message.

[0181] Step 3: After the file server obtains the first message, according to the first address information in the first message, it downloads the compressed package of the target file from the storage server. After downloading, it decompresses the compressed package. After the file server obtains the decompressed target file, it uploads the decompressed target file to the storage server.

[0182] Step 4: After the file server uploads the decompressed target file to the storage server, the storage server sends the second address information where the decompressed target file is stored in the storage server to the file server, and the file server receives the second address information.

[0183] Step 5: After the file server receives the second address information, it generates a second message and sends the second message to the first message queue. The second message will include the second address information.

[0184] Step 6: When the unitized service component in the first unit monitors the second message sent by the file server in the first message queue, it obtains the second message. In this way, the unitized service component finally obtains the second address information of the decompressed target file.

[0185] Based on this, the unitized service component can subsequently update the first metadata information of the target file in the database according to the obtained second address information, so that when the user requests to view the target file subsequently, the target file can be browsed and viewed according to the second address information of the decompressed target file stored in the database.

[0186] Based on the data processing method applied to the file server provided in the above embodiment, correspondingly, the present application also provides a data processing device corresponding to the data processing method applied to the file server. The following is through Figure 5 A detailed introduction to the data processing device.

[0187] Figure 5It is a schematic structural diagram of a data processing device applied to a file server provided by an embodiment of the present application. Figure 5 The shown data processing device includes:

[0188] A first acquisition module 510, configured to, when it is monitored that a first message queue includes a first message sent by a unitized service component, acquire the first message from the first message queue, where the first message includes first metadata information of a target file, and the first metadata information includes first address information, and the first address information is the compressed package address information of the target file;

[0189] A download module 520, configured to download a compressed package of the target file from a storage server based on the first address information;

[0190] An extraction module 530, configured to extract the compressed package to obtain the extracted target file;

[0191] An upload module 540, configured to upload the extracted target file to the storage server;

[0192] Wherein, the compressed package of the target file is obtained by the vehicle-mounted device after compressing the acquired parking data; the vehicle-mounted device uploads the compressed package of the target file to the storage server in advance; the physical distance between the file server and the storage server is less than the physical distance between the storage server and the unitized service component.

[0193] The data processing device of the embodiment of the present application, applied to a file server, can, through corresponding functional modules, acquire the first message sent by the unitized service component in the first message queue, and based on the first metadata information of the target file in the first message, download the compressed package of the target file in the storage server, and after decompressing the downloaded compressed package, upload the decompressed target file to the storage server, where the physical distance between the file server and the storage server is less than the physical distance between the storage server and the unitized service component.

[0194] Compared with the prior art, a data processing device provided by an embodiment of the present application, applied to a file server, by configuring a file server with a physical distance satisfying a preset condition between the storage server and the unitized service component and combining with a message queue, can transform the transmission of a long-distance large file into a long-distance micro-data transmission based on the message queue, and transform the long-distance file transmission between the storage server and the unitized service component into a short-distance file transmission between the storage server and the file server. In this way, the processing efficiency of file data is effectively improved, and the time consumption in the processes such as file download and upload is reduced.

[0195] In some embodiments, considering the actual needs of subsequent users, it is necessary to save the storage address of the decompressed target file to a database.

[0196] Based on this, the first metadata information of the target file may further include the index identification information of the target file in the database. After uploading the decompressed target file to the storage server, the data processing device may further include:

[0197] A receiving module, which can be used to receive the second address information sent by the storage server. The second address information may be the address information of the decompressed target file;

[0198] A second generating module, which can be used to generate a second message. The second message may include the second address information and the index identification information of the target file in the database;

[0199] A second sending module, which can be used to send the second message to the first message queue, so that when the unitization service component monitors that the first message queue includes the second message, it performs a first target operation;

[0200] Among them, the first target operation may include:

[0201] Based on the index identification information of the target file in the database, store the second address information in the database.

[0202] In some embodiments, the first message may further include the identification information of the first message queue. The second sending module may specifically be used to send the second message to the first message queue based on the identification information of the first message queue.

[0203] In some embodiments, in order to further improve the data processing efficiency, the physical distance between the file server and the storage server may be within a preset range.

[0204] In some embodiments, considering that file damage often occurs during decompression, therefore, in order to improve the accuracy of the transmitted file data, the first metadata information of the target file may further include the first file attribute information of the target file; the uploading module 540 may specifically include:

[0205] A detection sub-module, which can be used to detect whether the second file attribute information of the decompressed target file is consistent with the first file attribute information based on the first file attribute information of the target file;

[0206] An uploading sub-module, which can be used to upload the decompressed target file to the storage server when the second file attribute information is consistent with the first file attribute information.

[0207] Based on the data processing method applied to the unitization service component provided in the above embodiments, correspondingly, the present application also provides a data processing device corresponding to the data processing method applied to the unitization service component. The following is through Figure 6A detailed introduction to the data processing device is provided.

[0208] Figure 6 It is a schematic structural diagram of a data processing device applied to a unitized service component provided by an embodiment of the present application. Figure 5 The shown data processing device includes:

[0209] A determination module 610, configured to determine a file corresponding to the target information as a target file when it is detected that the database contains the target information;

[0210] A first generation module 620, configured to generate a first message according to the first metadata information of the target file, the first message includes the first metadata information of the target file, and the first metadata information includes a first address information, and the first address information is the compressed package address information of the target file;

[0211] A first sending module 630, configured to send the first message to a first message queue, so that when the file server monitors that the first message queue includes the first message, it performs a second target operation;

[0212] Among them, the compressed package of the target file is obtained by the vehicle computer compressing the obtained parking data; the vehicle computer uploads the compressed package of the target file to the storage server in advance; the physical distance between the file server and the storage server is less than the physical distance between the storage server and the unitized service component;

[0213] The second target operation includes:

[0214] Download the compressed package of the target file from the storage server based on the first address information;

[0215] Decompress the compressed package to obtain the decompressed target file;

[0216] Upload the decompressed target file to the storage server.

[0217] The data processing device of the embodiment of the present application, which is applied to a unitized service component, can, through corresponding functional modules, determine a target file to be decompressed when it is detected that the database contains the target information, then generate a first message based on the first metadata information of the target file, and send the first message to the first message queue. In this way, when the file server monitors that the first message queue includes the first message, it can perform corresponding file download, decompression and upload operations. A data processing method provided by an embodiment of the present application, which is applied to a unitized service component, can effectively improve the processing efficiency of file data and reduce the time-consuming of processes such as file download and upload.

[0218] In some embodiments, considering the actual requirements of subsequent users, it is necessary to save the storage address of the decompressed target file to the database. Based on this, the first metadata information of the target file may further include the index identification information of the target file in the database. After sending the first message to the first message queue, the data processing device may further include:

[0219] A second acquisition module, which can be used to acquire the second message from the first message queue when it is monitored that the first message queue includes the second message sent by the file server. The second message may include the second address information and the index identification information of the target file in the database, and the second address information may be the address information of the decompressed target file;

[0220] A storage module, which can be used to store the second address information to the database based on the index identification information of the target file in the database.

[0221] Figure 7 It is a schematic structural diagram of a data processing device provided by an embodiment of the present application.

[0222] The data processing device may include a processor 701 and a memory 702 storing computer program instructions.

[0223] Specifically, the above-mentioned processor 701 may include a central processing unit (CPU), or an application specific integrated circuit (ASIC), or may be configured as one or more integrated circuits implementing the embodiments of the present application.

[0224] The memory 702 may include a mass storage for data or instructions. By way of example and not limitation, the memory 702 may include a hard disk drive (HDD), a floppy disk drive, a flash memory, an optical disc, a magneto-optical disc, a magnetic tape, or a universal serial bus (USB) drive, or a combination of two or more of these. In a suitable case, the memory 702 may include a removable or non-removable (or fixed) medium. In a suitable case, the memory 702 may be inside or outside the integrated gateway disaster recovery device. In a specific embodiment, the memory 702 is a non-volatile solid state memory.

[0225] The memory may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk storage media device, an optical storage media device, a flash memory device, an electrical, optical, or other physical / tangible memory storage device. Thus, generally, the memory includes one or more tangible (non-transitory) computer-readable storage media (e.g., a memory device) encoded with software including computer-executable instructions, and when the software is executed (e.g., by one or more processors), it is operable to perform the operations described in reference to the method according to an aspect of the present disclosure.

[0226] The processor 701 reads and executes the computer program instructions stored in the memory 702 to implement any one of the data processing methods in the above embodiments.

[0227] In one example, the data processing device may further include a communication interface 703 and a bus 710. Among them, as Figure 7 shown, the processor 701, the memory 702, and the communication interface 703 are connected through the bus 710 and complete communication with each other.

[0228] The communication interface 703 is mainly used to implement communication between various modules, devices, units, and / or devices in the embodiments of the present application.

[0229] The bus 710 includes hardware, software, or both, and couples the components of the data processing device to each other. By way of example and not limitation, the bus may include an Accelerated Graphics Port (AGP) or other graphics bus, an Enhanced Industry Standard Architecture (EISA) bus, a Front-Side Bus (FSB), a HyperTransport (HT) interconnect, an Industry Standard Architecture (ISA) bus, an InfiniBand interconnect, a Low Pin Count (LPC) bus, a memory bus, a MicroChannel Architecture (MCA) bus, a Peripheral Component Interconnect (PCI) bus, a PCI-Express (PCI-X) bus, a Serial Advanced Technology Attachment (SATA) bus, a Video Electronics Standards Association Local (VLB) bus, or other suitable buses or a combination of two or more of these. In a suitable case, the bus 710 may include one or more buses. Although the embodiments of the present application describe and illustrate specific buses, the present application contemplates any suitable bus or interconnect.

[0230] The data processing device executes the data processing method in the embodiments of the present application, thereby implementing Figure 2 and Figure 3 the data processing methods described.

[0231] In addition, in combination with the data processing method in the above embodiments, the embodiments of the present application may provide a computer storage medium to implement. Computer program instructions are stored on the computer storage medium; when the computer program instructions are executed by a processor, any one of the data processing methods in the above embodiments is implemented.

[0232] It should be clear that the present application is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present application is not limited to the specific steps described and shown, and those skilled in the art can make various changes, modifications, and additions, or change the order between steps after understanding the spirit of the present application.

[0233] The functional blocks shown in the above structure block diagrams can be implemented as hardware, software, firmware, or a combination thereof. When implemented in hardware, it can be, for example, an electronic circuit, an application-specific integrated circuit (ASIC), appropriate firmware, a plug-in, a function card, and so on. When implemented in software, the elements of the present application are programs or code segments used to perform the required tasks. The program or code segment can be stored in a machine-readable medium or transmitted via a data signal carried in a carrier wave on a transmission medium or a communication link. A "machine-readable medium" can include any medium capable of storing or transmitting information. Examples of machine-readable media include electronic circuits, semiconductor memory devices, ROM, flash memory, erasable ROM (EROM), floppy disks, CD-ROMs, optical discs, hard disks, fiber optic media, radio frequency (RF) links, and so on. The code segment can be downloaded via a computer network such as the Internet, an intranet, and so on.

[0234] It also needs to be noted that the exemplary embodiments mentioned in the present application describe some methods or systems based on a series of steps or devices. However, the present application is not limited to the order of the above steps, that is, the steps can be executed in the order mentioned in the embodiments, can be different from the order in the embodiments, or several steps can be executed simultaneously.

[0235] As described above, aspects of the present disclosure have been described with reference to the flowcharts and / or block diagrams of methods, apparatuses (systems), and computer program products according to embodiments of the present disclosure. It should be understood that each block in the flowchart and / or block diagram, and the combination of blocks in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device to produce a machine, such that the instructions executed by the processor of the computer or other programmable data processing device enable the implementation of the functions / operations specified in one or more blocks of the flowchart and / or block diagram. Such a processor can be, but is not limited to, a general-purpose processor, a special-purpose processor, a special application processor, or a field programmable logic circuit. It should also be understood that each block in the block diagram and / or flowchart, and the combination of blocks in the block diagram and / or flowchart, can also be implemented by dedicated hardware that performs the specified functions or operations, or can be implemented by a combination of dedicated hardware and computer instructions.

[0236] As described above, the above is only a specific implementation manner of the present application. Those skilled in the art can clearly understand that for the convenience and brevity of description, the specific working processes of the systems, modules, and units described above can refer to the corresponding processes in the foregoing method embodiments, and will not be described herein again. It should be understood that the protection scope of the present application is not limited thereto. Any person skilled in the art can easily think of various equivalent modifications or substitutions within the technical scope disclosed in the present application, and these modifications or substitutions should all be covered by the protection scope of the present application.

Claims

1. A data processing method, characterized in that: Applied to a file server, the method includes: When monitoring a first message queue including a first message sent by a unitized service component, obtaining the first message from the first message queue, wherein the first message includes first metadata information of a target file, the first metadata information includes first address information, and the first address information is compressed package address information of the target file; Downloading the compressed package of the target file from the storage server based on the first address information; Decompressing the compressed package to obtain the decompressed target file; Uploading the decompressed target file to the storage server; The compressed package of the target file is obtained by compressing the acquired parking data by the vehicle computer, and the vehicle computer pre-uploads the compressed package of the target file to the storage server; the physical distance between the file server and the storage server is less than the physical distance between the storage server and the unitized service component; The physical distance between the file server and the storage server is within a preset range, and an upper limit of the preset range is less than the physical distance between the storage server and the unitized service component; the number of the storage server is single; The first metadata information of the target file also includes index identification information of the target file in the database. After uploading the decompressed target file to the storage server, the method further includes: receiving second address information sent by the storage server, where the second address information is address information of the decompressed target file; generating a second message, wherein the second message includes the second address information and index identification information of the target file in the database; Sending the second message to the first message queue, so that the unitized service component performs a first target operation when monitoring the first message queue including the second message; wherein the first target operation includes: storing the second address information in the database based on the index identification information of the target file in the database; The unitized service component is a service component provided in a non-specific service unit, and the user terminal is used to call the second address information stored in the database to view the parking data.

2. The method according to claim 1, wherein The first message also includes identification information of the first message queue, and sending the second message to the first message queue includes: Based on the identification information of the first message queue, the second message is sent to the first message queue.

3. The method according to claim 1, wherein The first metadata information of the target file also includes first file attribute information of the target file; and uploading the decompressed target file to the storage server includes: Based on the first file attribute information of the target file, detecting whether the second file attribute information of the decompressed target file is consistent with the first file attribute information; When the second file attribute information is consistent with the first file attribute information, the decompressed target file is uploaded to the storage server.

4. A data processing method, characterized in that: Applied to a unitized service component, the method includes: In the case of detecting that the database contains target information, determining the file corresponding to the target information as the target file; Generate a first message according to the first metadata information of the target file, wherein the first message includes the first metadata information of the target file, the first metadata information includes first address information, and the first address information is compressed package address information of the target file; Sending the first message to a first message queue, so that the file server performs a second target operation when monitoring that the first message queue includes the first message; The second target operation includes: Downloading the compressed package of the target file from the storage server based on the first address information; Decompressing the compressed package to obtain the decompressed target file; Uploading the decompressed target file to the storage server; wherein the compressed package of the target file is obtained by compressing the acquired parking data by the vehicle computer; the vehicle computer pre-uploads the compressed package of the target file to the storage server; the physical distance between the file server and the storage server is less than the physical distance between the storage server and the unitized service component; The physical distance between the file server and the storage server is within a preset range, and an upper limit of the preset range is less than the physical distance between the storage server and the unitized service component; the number of the storage server is single; The first metadata information of the target file also includes index identification information of the target file in the database. After sending the first message to the first message queue, the method further includes: When monitoring that the first message queue includes a second message sent by the file server, obtaining the second message from the first message queue, wherein the second message includes second address information and index identification information of the target file in the database, and the second address information is the address information of the decompressed target file; Based on the index identification information of the target file in the database, storing the second address information in the database; The unitized service component is a service component provided in a non-specific service unit, and the user terminal is used to call the second address information stored in the database to view the parking data.

5. A data processing device, characterized in that: Applied to a file server, the device comprises: A first acquisition module is configured to, upon monitoring a first message queue including a first message sent by a unitized service component, acquire the first message from the first message queue, wherein the first message includes first metadata information of a target file, the first metadata information including first address information, and the first address information is compressed package address information of the target file; A download module, configured to download the compressed package of the target file from a storage server based on the first address information; A decompression module, used for decompressing the compressed package to obtain the decompressed target file; An uploading module, configured to upload the decompressed target file to the storage server; The compressed package of the target file is obtained by compressing the acquired parking data by the vehicle computer; the vehicle computer pre-uploads the compressed package of the target file to the storage server; the physical distance between the file server and the storage server is less than the physical distance between the storage server and the unitized service component; The physical distance between the file server and the storage server is within a preset range, and an upper limit of the preset range is less than the physical distance between the storage server and the unitized service component; the number of the storage server is single; The first metadata information of the target file also includes index identification information of the target file in the database, and after uploading the decompressed target file to the storage server, the method further includes: receiving second address information sent by the storage server, where the second address information is address information of the decompressed target file; generating a second message, wherein the second message includes the second address information and index identification information of the target file in the database; Sending the second message to the first message queue, so that the unitized service component performs a first target operation when monitoring the first message queue including the second message; wherein the first target operation includes: storing the second address information in the database based on the index identification information of the target file in the database; The unitized service component is a service component provided in a non-specific service unit, and the user terminal is used to call the second address information stored in the database to view the parking data.

6. A data processing device, characterized in that: Applied to a unitized service component, the device comprises: a determination module, configured to, when detecting that the database contains target information, determine the file corresponding to the target information as the target file; A first generating module, configured to generate a first message according to first metadata information of the target file, wherein the first message includes the first metadata information of the target file, the first metadata information includes first address information, and the first address information is compressed package address information of the target file; A first sending module is configured to send the first message to a first message queue, so that the file server performs a second target operation when monitoring that the first message queue includes the first message; The second target operation includes: Downloading the compressed package of the target file from the storage server based on the first address information; Decompressing the compressed package to obtain the decompressed target file; Uploading the decompressed target file to the storage server; wherein the compressed package of the target file is obtained by compressing the acquired parking data by the vehicle computer; the vehicle computer pre-uploads the compressed package of the target file to the storage server; the physical distance between the file server and the storage server is less than the physical distance between the storage server and the unitized service component; The physical distance between the file server and the storage server is within a preset range, and an upper limit of the preset range is less than the physical distance between the storage server and the unitized service component; the number of the storage server is single; The first metadata information of the target file also includes index identification information of the target file in the database. After sending the first message to the first message queue, the method further includes: When monitoring that the first message queue includes a second message sent by the file server, obtaining the second message from the first message queue, wherein the second message includes second address information and index identification information of the target file in the database, and the second address information is the address information of the decompressed target file; Based on the index identification information of the target file in the database, storing the second address information in the database; The unitized service component is a service component provided in a non-specific service unit, and the user terminal is used to call the second address information stored in the database to view the parking data.

7. A data processing device, characterized in that: The device includes: a processor and a memory storing computer program instructions; When the processor executes the computer program instructions, the data processing method according to any one of claims 1 to 4 is implemented.

8. A computer-readable storage medium, characterized in that The computer-readable storage medium stores computer program instructions, and when the computer program instructions are executed by a processor, the data processing method according to any one of claims 1 to 4 is implemented.

9. A computer program product, characterized in that When the instructions in the computer program product are executed by a processor of an electronic device, the electronic device executes the data processing method according to any one of claims 1 to 4.

Images